Love it or hate it, virtual reality is having a moment. Kamal Sinclair, Director of the New Frontier Lab Programs at Sundance Institute, which supports the creation of new media projects, said as much in an address to members of the press at this year’s Sundance VR Palace. 

“For us,” Sinclair said, “the VR program started with a love of technology and to see where technology could take storytelling. It quickly has now gone to a place where it's about experiences that you can't quite convey in a newspaper article or in a movie, even. More than that, the philosophy behind it is a love and inquiry into humanism and into a philosophy of: this is our next frontier of existence, our next frontier of consciousness.”

Sinclair's sentiments convey the potential of the evolving medium, but also how daunting such a new landscape can be. Fortunately, many independent creators are breaking this ground for the rest of us and helping to come up with best practices. We spoke to several filmmakers whose projects were featured at this year’s Sundance and Tribeca VR showcases to learn about how they approached the work and get advice for anyone who is looking to step into this wide open territory.

'The Life of Us'Credit: Within

Life of Us

A two-person VR experience that propels you through the entire history of evolution.

Jonny Ahdout:

One of the exciting things about VR is that all of the tools that we grew to love in film have a completely different effect in virtual reality, and it's almost as if all the little tips and tricks that we knew have all these new powers that we get to rediscover again. 

There's no rule book, no rubric, and very few experiences out there that you can reference. You know, we have hundreds of years of filmmaking behind us that we can study academically before we go and make our own films. With VR, it's an experiential medium, and you're feeding an experience directly into someone's eyes and ears in the same way that they experience the real world.

You have to go into it with an open mind, and you have to trust your instincts and your gut. "Does this experience feel right? Is it giving me what I want?” You can't necessarily cut all the time in VR, so ask, "Why do you feel like you need to cut?" Then, what's the version of a cut that works in this medium, to get you the effect that you're looking for?

A still from Mindshow by Gil Baron, Adam Levin and Jonnie Ross, an official selection of the New Frontier VR Experiences program at the 2017 Sundance Film Festival.Credit: Sundance Institute

Mindshow

Software that allows you to create your own animated VR shows with your body and voice.

Gil Baron:

You can’t just take a film and up-res it to VR. It’s not VR without being built for that. And something that works in traditional film language—a crash zoom or a quick cut to a closeup—is very jarring when it’s in your personal space. So what we’ve learned is to be gentle and break things, but understand that as you do, there’s an implicit contract with the viewer that you need to be respectful of because VR is personal. 

A still from Extravaganza by Ethan Shaftel, an official selection of Tribeca Immersive at the 2017 Tribeca Film Festival.

Extravaganza

A mix of 3D animation and live-action footage that makes you a literal puppet in a satirical and in-your-face menagerie.

Ethan Shaftel:

The tools I'm using are tools I'm familiar with through linear, non-immersive films. I’m using Cinema 4D and After Effects and film actors, but you definitely leave a lot behind.

What traditional film does is a set of brain hack technologies that have been developed for a hundred years of how to deliver first-person experience vicariously to the audience. The thing is, many of those techniques or technologies that we've developed no longer apply. In VR, the one fundamental component that cannot be ignored is location; it’s point of view, but in a very concrete sense. Not the abstract sense in a film, where it’s a soft technique, versus literally being in someone's body. In VR, it's a very overwhelmingly noticeable, “Whoa! I have a body.” 

Closeups also mean something very different in VR. What a closeup does in a traditional film is indicate that we're almost inside the head of the subject that we're filming. What a closeup does in VR is you're right in front of the face of the person you're filming. Empathy is secondary to immersion. "Wow, there's a giant face."

A still from Melting Ice by Danfung Dennis, an official selection of the New Frontier VR Experience program at the 2017 Sundance Film Festival.Credit: Danfung Dennis

Melting Ice

An up-close exploration of Greenland’s collapsing ice sheet that reveals the dramatic effects of climate change.

Danfung Dennis:

The fundamentals of storytelling stay the same—strong characters, emotion—but it's a very different medium than film. It's highly spatial. Placing the camera essentially becomes the composition. You no longer have a frame, but you have this 3D space. And so putting the camera into place is really the same as making that frame for an image, except you're putting out a spherical image. 

VR invokes a sense of presence—that you're actually there. You need to figure out how to use that presence and how to transport someone into a place and give them a direct experience of that story. And they become the storyteller, so I think there's a little bit of letting go from a traditional directing standpoint and delivering the raw experience and then letting the viewer have their own unique, individual experience through it. Formulate the story to tell someone later. 

A still from Chocolate by Tyler Hurd, an official selection of the New Frontier VR Experiences program at the 2017 Sundance Film Festival.Credit: Sundance Institute

Chocolate

VR music video for Giraffage’s “Chocolate,” in which you play a robot in a world of cats.

Tyler Hurd:

Obviously, you can’t control where people are looking in VR, but you can still control a composition spherically. I think a lot about foreground, midground, and background and making something interesting in each place and trying to compose a spherical frame. I think that’s the biggest crossover between VR and film. Other than that, it’s completely different.

A still from Auto by Steven Schardt, an official selection of Tribeca Immersive at the 2017 Tribeca Film Festival.

Auto

A sci-fi night-in-the-life of an Ethiopian immigrant “safety driver” employed by the self-driving taxi services of the near future.

Steven Schardt:

VR is quite difficult at first. The learning curve is steep because there are no established conventions and you do have to teach yourself the various technologies required to go from acquisition to delivery and from production to post-production.

I began this project with little VR experience. The whole crew had extensive filmmaking experience but none had shot a VR piece and, in fact, half of them hadn't seen VR before embarking on it. But through the course of about a year-and-a-half of online tutorials taught by anyone from teenagers in Indonesia to 53-year-old VFX specialists in the UK, I was able to learn how to stitch with Autopano Video, to composite with After Effects, SynthEyes, do modeling and compositing with Cinema 4D and OctaneRender. There’s an extensive list of software packages [to learn]. I think you really have to want to do a piece in VR.

A still from Treehugger: Wawona, an official selection of Tribeca Immersive at the 2017 Tribeca Film Festival.Credit: Marshmallow Laser Feast

Treehugger Chapter 01: WAWONA 

An interactive, multi-sensory installation that takes you inside a giant sequoia tree. (Winner of the 2017 Tribeca Storyscapes Award)

Ersin Han Ersin:

In the beginning of a project, we always ask, "Can you do that in any medium other than virtual reality?" If you can, don't do it. It's as simple as that. Once we’re past that part, the second question is, "What makes it unique for that particular medium?"

For the case of Treehugger, you are sitting in a forest and seeing the perception of four different animals and insects. It's based on scientific facts, but, obviously, we are speculating. You are putting on those goggles as if you are swapping out your perception with something else. You can't do that in the real world. Your tactile senses are completely open. You are smelling the scent of the tree, which means we needed to design a scent-scape. You are hearing the recorded bio-signals, the heartbeat of the sequoia tree. These things could not be done in any other medium.

A still from The People's House by Felix & Paul Studios, an official selection of Tribeca Immersive at the 2017 Tribeca Film Festival.

The People’s House

An intimate tour of the White House’s public and private rooms with Barack and Michelle Obama.

Félix Lajeunesse:

In The People's House, you transcend space and time through the whole experience. Honestly, I do not believe that this is something that you could achieve as well with cinema, because you have a sense of presence. It just adds this layer of emotional connection that I believe does not exist in cinema, at least not to that extent. For us, that is a very strong feature of cinematic virtual reality that we always try to emphasize. 

The whole notion of the emotional connection to a person or to a character is far different in our perspective in virtual reality than in cinema. For instance, in this piece, there is a moment where President Obama talks to you and says, "The first time I came here in this room...." He talks about the Oval Office. He says, "I was surprised by the amount of light," and then he turns and he starts talking about the light that he sees in the room. Every single time that I watch this piece, I think, "Is it disrespectful for me to turn and look at the light?"

Every single time, I get tricked into starting to behave like a human being inside of that situation. If I really was in that room with that person, how would I behave? My brain reacts the same way. This is part of the phenomenon of presence. 

Paul Raphaël:

The first question we ask ourselves is, "What is the nature of your presence in the piece?" Careful thought is put into who the viewer is at any given moment and how that impacts the tone and the nature of the shot.

It's a question I would recommend anyone ask themselves. The answer doesn't necessarily have to be a character. In fact, in most of our documentary work, the answer is the viewer is the person who is watching the experience. But that is still something to keep in mind as you sculpt the piece: there is not just a 360 camera there, there is actually a person that you're putting in the scene. That has an impact on everything else that happens in the scene. It's not an invisible presence. I should say it's not an immaterial presence. It does impact. It's like an ecosystem which the viewer is a part of.

More on VR

For longer discussions with VR creators, check out our podcasts on the topic:

• VR is Not Filmmaking—Here's Why [PODCAST]
• How Filmmakers Can Conquer VR's Biggest Challenges [PODCAST]
• Where VR Has Never Gone Before: Unprecedented 40-Minute Sundance Short ‘Miyubi’ [PODCAST]

Additional reporting by Oakley Anderson-Moore.