Kirkley’s film Orange Sunshine uses interviews and Super 8 recreations to detail the rise and fall of the Brotherhood of Eternal Love, a group of surfer-hippies who hoped to bring enlightenment to the masses via LSD in the early 60's. Eventually, they became the world’s largest distributors of hash and psychedelics.
The tale behind the 10-year-making of the film is interesting in its own right, but what really intrigues us is that Kirkley decided to push things further and venture into VR, an unusual move for a relatively low-budget documentary. The Beginning, a VR experience that accompanies the film, takes us — via motion trackers and an HTC Vive headset — into a spiritual circle of people tripping acid in the desert. Although the visuals are computer-generated, it feels real by design; the entire piece was shot with actors, and each scene was painstakingly recreated in CG. Furthermore, the motion tracking provides for a unique experience: when you move your head or body, your digital character mirrors you in the virtual world.
Kirkley and his co-creators, Adam Amaral and Glenn Snyder from the company Master of Shapes, faced many creative and technical hurdles. No Film School sat down with them the day after the film’s premiere at SXSW 2016 (also the day before The Beginning was to launch in the festival’s VR/AR showcase). After trying a special sneak peek of the experience, we got into the nitty-gritty of its creation, from shooting considerations to creating 3D depth.
"In game engine, we have almost infinite possibility. And when anything is possible, you can forget that this is a film and it needs to feel cinematic."
NFS: William, why did you decide to add a VR component to your project?
Kirkley: A couple years ago I saw Chris Milk's VR documentary Clouds Over Sidra. It's a story of being in a Syrian refugee camp, narrated by a thirteen-year-old girl. I felt like I was in her home and at school with her. After the short film, I was bawling. It was incredible; so powerful and so immersive. I had never experienced a connection like that before in a film. It really took me somewhere so much further than I could have ever imagined. From that point on, I knew I wanted to do something with VR.
Clouds Over Sidra, by Chris Milk
When I was working on Orange Sunshine, I thought, "How cool would it be if we could take audiences further into the Brotherhood experience?" We zeroed in on the moment when the Brotherhood formed. I had this rough idea of having audiences go with them out in the desert, in Joshua Tree, underneath the night sky, to have an LSD experience just like the Brotherhood did on their first time.
There's not a whole lot of VR associated with indie documentaries. We will be doing some kind of grassroots tour where we travel around with the film, and I thought this would be the perfect companion piece for that. It'd be a great way to motivate people to actually come to the theater and see the film and have the VR experience.
"The best visual effects are the ones that no one ever sees, where a director is on set creating something beautifully with their DP and then you accentuate that."
NFS: How did you approach the creation of the VR piece?
Snyder: Adam and I both come from a visual effects background. We both feel that the best visual effects are the ones that no one ever sees, where a director is on set creating something beautifully with their DP and then you accentuate that and fill it up a little more. That's kind of the approach we took to this. Let's not make something absurd; let's make something that feels very comfortable and natural, and then we'll add little tweaks that make you feel like, "Wait am I…? What was that?”
NFS: What was the process of shooting the piece in live-action before you turned it into CG?
Kirkley: We started with filming our actors on a green screen. Then we went and shot plates at both locations that you see in the piece: out at Joshua Tree at nighttime and inside the house. Then these guys [Amaral and Snyder] took all of that and rebuilt it.
Credit: Rudi Barth
Amaral: We [paid] massive attention to the detail of the original locations. Will scouted really authentic spots, like a mid-century home that was authentic to the era. That was our reference; we didn't change anything about it.
Kirkley: We rented props and dressed it all up a if it were a narrative set. Shooting for this was still approached traditionally. It's still all the traditional styles, but there's just a lot more work on top.
NFS: How did you have to think differently about shooting the material than when you're directing a traditional film?
Kirkley: You can't think in terms of framing. You have to throw framing out the door. You're living inside a world, so you have to think about any direction that you could see. It's no longer about creating a perfect frame, which I've always been so used to doing. It's much more exciting and challenging thinking about that immersive experience where you can have so many more interesting storytelling things happening around you, especially using audio. I love the spatial sound that we did. From a directing standpoint, it was about staying on top of the tonality and capturing the feeling.
In directing the actors, it was pretty similar. It's almost a little bit more of a theatrical approach. It's a little bit more "bigger is better" than on film because our picture is so big.
Amaral: I think we've definitely got a mood. Like Will said, staying on top of it tonally is tough because, in game engine, we have almost infinite possibility. You can kind of get lost in infinite possibility. We needed someone to reel us back in and be like, "Guys, we're trying to go for this warm tone, this '60s vibe." When anything is possible, you can forget that this is a film and it needs to feel cinematic.
"The Beginning"
Snyder: Another difference for the shoot is that you can't use a sound stage anymore. Having a sound stage is kind of unrealistic because [in VR] you can turn around and see your whole crew sitting there. Will came in and paid attention to all the details. There are records on the floor that match the era; nothing in the room is out of place. When you look around, you're not like, "Aw crap, there's a brand new Schwinn, this entire thing is broken." We got to take those references and just try to match them one to one. They're just things you don't think of when you're coming into a normal filming approach.
A big difference in creating the story itself is timing, because we're used to this progressive buildup of stories. We know that we're supposed to have a protagonist, then there's supposed to be a tipping point in the beginning — we build up, we come back down, we have a crescendo at the end. That doesn't work in VR because there is so much extra stuff happening. In a film, you can do it that way because you're sitting down and it's just kind of pacing through and you're kind of mindlessly enjoying it. When you're in VR and it feels like you're a person there inside the scene, you're just like, "What the hell is going on?"
As a creator, you need bring people in, give them a moment, and then plateau for a second. Let them take it in, let them enjoy it and feel out what's happening, and then move into the next moment. The story becomes this ebb and flow as opposed to a constant buildup.
NFS: Once the footage was shot, how did you go about building the actual experience?
Snyder: We used the stereo footage that was shot flat, with extreme convergence on our characters. We brought it into a game engine called Unreal, where we then used visual effects and game techniques to recreate every single object in the scene as photoreal as we possibly could.
For example, Adam learned incredible skills of what's called “pre-baked” lighting, where we baked all the lighting to upgrade from a normal video game quality to become as filmic as possible. We tried to go as rich and crisp in the colors as we could, but since it's on a game platform, it has to be in real-time so we had to learn a lot of techniques for how to make that happen.
[youtube https://youtu.be/Z3EIpAcGWZA?list=PLZlv_N0_O1gZaB0IgQEnO9WOXYRx3Puvo What is Virtual Reality? By the Unreal Game Engine team expand=1 site_id=26256498]
Then we brought in things like the image generator Google Deep Dream, which they call “computers dreaming.” You give it images and you get something back out that looks absurd and unexpected. We spent a couple weeks just creating random effects that we felt were very true to the experience of taking acid.
Amaral: The live-action background plates were shot for reference, but we actually ended up making everything fully CG. That was a labor of love, but the benefit of that approach is perfect 3D depth. It's a challenge right now, in the infancy of stitching and 360 video, to get perfect depth, especially for close objects. Convergence under three feet is really challenging, so doing it all CG is kind of like the ultimate cheat because you get the best of both worlds: amazing depth and exact objects and control.
Snyder: We had two months of just trying every stereo[scopic] technique we knew. One of the early experiments we tried was... you can take all of the imagery you get out of a 360 image and get what's called a “depth map” of how far away things are. If you do it well enough, you can stretch that image out and have what was just a flat image turn into geometry all around you because you know how far each pixel is from the next pixel. We started with that and it actually looked really cool, but then things started to break. As I started to add more immersion, it [became] harder and harder to keep this gag going.
At one point, Adam was like, "Dude, I have to recreate it," and I was just like, "You're not going to want to do that," and then he recreated it and we looked over the lighting afterwards, looked at the two versions and we're like, "Thank god we did that. It looks so much better now."
"You don't realize how much of your immersive experience is built on what your ears are feeling."
Amaral: Sound is something to note because I think that it's often overlooked in VR filmmaking. The visual is absolutely important, but I would say it's about 50 percent and audio is another 50 percent. We're visual guys and we didn't realize how important sound was. Especially spatialized audio, which you'll experience here. When you hear a sound from the left, it comes from the left, and when you turn to face forward, it'll actually be frontal now. That subtle amount of accurate sound combined with accurate visual just completes the immersion. Your brain is almost tricked there; it thinks it’s real.
Credit: Rudi Barth
Snyder: You don't realize how much of your immersive experience is built on what your ears are feeling. If you're creating a VR film, but you're not using a game engine or a real-time engine, the audio is going to be locked to wherever you're looking. If I'm looking at you when you're talking to me, but then I turn to look at Will, it's going to sound like Will's talking to me instead of your voice staying where your voice is. It's weird when you rotate with your head and you don't feel the room staying where it is. With spatialized audio, our audio engineer brought the whole omnidirectional side to things.
NFS: What did you learn about collaborating between film people and tech people?
Amaral: Will speaks director language and we speak technical language, so it’s been nice to merge those two. The one thing with VR that I sometimes felt bad about is that there are technical limitations. It was always a compromise of trying to get Will’s vision in, and also get it to run at a smooth frame rate or just get it to work.
Kirkley: In terms of directing, this is much more collaborative than film, and I really appreciate that. Even though these guys are incredibly technical, like engineers, they're also very creative. They came up with so many fantastic ideas and ways we could push some of our simple ideas and build on those. It brought so much to the table creatively.
"If you have rules in your head and you're very attached to them, you just need to burn them down and start over. It's going to be fun, though — that's the exciting part. Don't be scared of it. Enjoy it."
Snyder: A lot of times when you work with a director they're just very much like, "This is what I'm seeing. If you can't do what I'm seeing, I'm going to go talk to someone else." It gets really stressful. You're trying to make something, but the technical limitations are something you can't really override.
NFS: Any more advice for filmmakers considering a VR project?
Snyder: One of the biggest things I would stress to any type of filmmaker is that this breaks all the rules. Throw out all the nomenclature you have. Dutch angles don't exist anymore; your over-the-shoulder is out the window. Blocking a scene doesn’t work the same way. It's exciting because now you’re just like, "Everything I learned means nothing and I get to just figure this out." If you have rules in your head and you're very attached to them, you just need to burn them down and start over. It's going to be fun, though — that's the exciting part. Don't be scared of it. Enjoy it.
For more, see our complete coverage of the 2016 SXSW Film Festival. Listen to our podcasts from SXSW (or subscribe in iTunes):
No Film School's coverage of the 2016 SXSW Film Festival is sponsored by SongFreedom.
