Right now the goal at Oculus Story Studio is move the Post-its.
The true finish line is locking their latest animated virtual reality movie Henry, but to do that they have quite a few issues to fix. There's a conference room wall in their San Francisco office covered in dozens of sticky paper squares of all shades, each representing a problem someone has to fix before the project is done. It’s like something out of a scene from Silicon Valley, only instead of tweaking a compression algorithm it's optimizing what is essentially a cartoon.
Then again, most cartoons aren’t made using videogame engines. And right now, that reliance on game-making software—specifically Unreal Engine 4—is creating a lot of new Post-its.
It's two weeks before Henry's July 28 premiere, and the creative brain-trust of Story Studio is huddled in a dimly lit room (they do meditation in here in the mornings). Ramiro Lopez Dau, the director, is wearing the latest prototype of Oculus' headset and watching the current version of Henry—something he's seen dozens of times. His supervising technical director, Max Planck, is at a computer console, his shoes off; he's watching the feeds going into both of Lopez Dau's eyes, while the handful of other employees in the room watch Planck's display and point out glitches. A ladybug shows up in the wrong color? That’s a Post-it. One of Henry's eyes looks wonky? That’s another.
"These will all become Post-it notes," Story Studio's creative director Saschka Unseld says to me in a hushed voice so as not disrupt his team's flow. "Ryan [Thomas], our main production coordinator, is constantly typing."
Indeed, Thomas will spend most of the next hour looking up from his laptop only to gather enough info for the next note. What the Story Studio team is doing, if it had a filmmaking analog, would be looking at dailies—except only one person can watch it at a time. Also, "dailies" is a bit of a misnomer. They used to do this once every couple of weeks, then weekly—there was too much to fix in between, and it took hours to render—but now, two weeks out, they do it every afternoon. Fix a problem, lose a Post-it. Watch it again, gain two more. "It makes me weirdly paranoid that things suddenly break, because I don’t know how much stuff will be broken tomorrow," Unseld says ruefully. "Sometimes it’s rough to get a sense of these things are done, we move on to fix these things."
But breaking things and fixing them is the whole point of Story Studio. Last fall, a few months after Facebook acquired Oculus, the studio was founded with the mission to determine what was possible for VR filmmaking, and then to share that knowledge with the world. "Part of the mission is 'inspire and educate,'" producer Edward Saatchi says. And a huge hurdle to doing that has been figuring out how to make movies using both traditional animation software like Maya and videogame tools, while also making Hollywood filmmakers comfortable with the media as "an art-form, rather than cinema in a videogame," as Saatchi puts it.
They’ve done it—Story Studio had its coming-out party at Sundance in January, showcasing an exploratory short called Lost—but they had to break Unreal in the process. "A couple of months ago we switched," Unseld says, gesturing toward the small team of animators working on Henry. “We disconnected ourselves from the out-of-the-box Unreal Engine and opened up the code so we can just play with the innards of it.”
Epic Games, the developer behind the Unreal Engine, has always kept the code for the software open-source; The company encourages people to play with it, but as Epic’s founder Tim Sweeney notes, Story Studio is "pushing the engine really hard." To get what they want out of Unreal 4, visual effects supervisor Inigo Quilez has written code to make things like fur and shadows behave more easily and naturally. It's a concern game developers rarely worried about outside of non-playable "cinematic sequences"—most players race by shadows before they really register.
Unreal 4 does feature an animation tool that helps developers build cinematic experiences, and by the end of the year they hope to incorporate something called Sequencer that will feel more familiar to filmmakers used to working in ProTools. But what Story Studio is doing is still beyond that—and Epic is happy to have them guide the way. "I think the thing people are going to be really impressed with when they watch Henry, and I've only seen little bits and pieces, but it really does look like a classic animated movie," says Epic CTO Kim Libreri, who made his bones doing VFX for The Matrix movies before moving on to Lucasfilm and eventually into gaming. "They’re doing a lot of tricks to produce that look, but it shows that with perseverance and the right mindset to scale your techniques to work for your art, you can get a pretty good result."
It wasn’t just that Story Studio needed Unreal to do something it wasn’t made for. They also wanted to do things that had never been done in movies, period: they want Henry to look you in the eye, no matter where you are in his virtual world.
At its core, Henry is a simple story. The titular hero is hedgehog who loves hugs. (This, clearly, is problematic.) While celebrating his birthday by himself, his balloon animals come to life and try to befriend him. (This, clearly, is really problematic.) There are emotional ups and downs, and when Henry experiences a certain feeling, he looks at the viewer to share his sadness or excitement.
This is an incredibly rare thing. It’s also unnerving; the first time I tried an early demo, I immediately felt like a voyeur, as if I was spying on this poor hedgehog during his sad solo birthday. But in VR, the whole point is that you’re meant to feel as though you’re physically there with Henry—and as Unseld pointed out to me when I took off the goggles, it’s weird if Henry doesn’t recognize someone sitting on his living room floor. "It's just like 'Why don't you look at me? I'm right here!'" Unseld laughs. But when he does it carries an emotional connection, an empathy, that even those abused action figures in Toy Story 3 didn’t quite muster. (Unseld will later lament that this connection makes comedy twice as hard because Buster Keaton-esque physical comedy just feels "mean.")
That first demo was in early June. At the time, Story Studio was still in the prototype phase and didn’t even know if the eye contact function would work. "Now that you've seen this, you see how putting in interactivity is going to be really confusing, it's going to be really weird," Saatchi said to me afterward.
Ultimately, the team did figure it out. Once they mastered the right tech and animation flow of Henry looking at the spectator (animators are obsessed with the idea of “in-betweening”—making sure a character moves at the perfect speed to relay his or her mood), they built a button that would allow them to trigger the action. Then, day after day, they watched it and hit that button until they figured out all the right times for Henry to share his moment with you. And in those moments, says Lopez Dau, "he is absolutely alive."
Obviously, VR filmmaking wasn’t even a concept five years ago; as for whether it will still be one in another five years hinges in large part on what Story Studio is doing. The division's mandate is to create as much animated VR content as it can in the hopes of divining what styles and genres work for the medium; Henry is their stab at comedy, and next up is Bullfighter, which will deal with tension. (At this point, very few people even have developer headsets; Oculus’ consumer units aren’t set to be released until early next year.) There are others at the vanguard like VRSE's Chris Milk and Felix & Paul, but when it comes to figuring out what the hell VR filmmaking is, Story Studio is in uncharted territory—especially with regards to animation.
"With Story Studio, one of the greatest things [they're doing] is helping storytellers be able to get across a message," Libreri says. "VR is a different medium from film, very different from game-making. They're establishing the rules, they're doing work that I think 10 years from now when we look back and can see what VR has turned into, we'll go 'Wow, Story Studio really did shine the light on the way forward.'"
Shining that light is still a work in progress, however; illumination and texture are still flummoxing the Story Studio crew, even as they enter the last few days of production on their film. After seeing the wall of Post-its, I ask Unseld how much has changed in the month or so since we last spoke. "Everything and nothing," he says, and to my untrained eye, this feels true. The project was supposed to be "CG locked" a week ago, but when Max Planck came in this morning there were 700 adjustments that had been checked in to the system. "It's as if 50 people were editing an article at the same time," Unseld says, "we have to just stop."
For now, work continues. For instance, that aforementioned ladybug. Initially it was supposed to be an IFTTT-style Easter egg that would float in and out of the experience if viewers noticed it the first time it flew in, but they eventually realized it worked better as a White Rabbit to guide people's exploration. With positional tracking, VR allows you to look under tables and around corners, but new users need some prompting.
Like Unseld, Planck and Lopez Dau are Pixar vets; the three of them are well aware that the animation powerhouse had no operating manual to speak of when it built its arsenal of animation tricks and techniques. "The first Toy Story was nearly breaking the tools with overload," Unseld says. To grow the way Pixar did, Story Studio is going to need more engineers to make this process easier.
Until then, there are Post-its. Considering the things left to fix, and how often those fixes introduce even more issues, it’s easy to wonder whether Henry will ever feel done. I ask Unseld how many times he would have to watch a flawless experience before finally feeling convinced that it was ready. "There was a saying at Pixar," he says. "'A movie is never done—it just gets released.'"
