These Filmmakers Test the Limits of Unreal Engine Without Breaking the Bank

Finding Unreal Inspiration

We decided to blend a scene made in Unreal into a live-action film with real characters and real drama, to see how seamlessly it could be integrated. How life-like its effects can be. What new possibilities it could open up for three friends with no budget but plenty of big ideas?

Since we lacked the budget and time (and 80,000 liters of blue paint) necessary to create the next Avatar, we decided to create a short in which the Unreal segment was an emotional extension of the drama in the main portion of the film.

We committed ourselves to the film being science-fiction because sci-fi is one of our big collective passions here at our production company Step Three. Then, we decided on a subject matter to play around with: social media influencers.

Have you been on Instagram lately? It’s a whirlpool of self-proclaimed wellness gurus and misinformation. We figured that’d be an interesting place to begin–not least because the question of what’s real versus what is digital fabrication echoed what we wanted to achieve with Unreal.

That’s how we came up with Something For Your Mind. It centers on a podcaster who begins to question the ethics of her mental health and well-being content. This soul-searching leads her to experiment with a mind-altering device.

Taking inspiration from both blockbusters such as The Matrix and lesser-known works of fiction such as Robert Longo’s Johnny Mnemonic and Vincent Ward’s What Dreams May Come True, we wrote a script that would visualize a character’s journey into their own mind using this device.

That, it turns out, was the easy part.

Laying the Groundwork

It was our intention from the start that our actress would provide 100% of the performance for the character within Unreal. We were determined not to simply add a computer-generated sequence to the project, as that wouldn't align with our vision.

Our lead actress, Phoebe Marshall, visited our studio in South East London, where we used an application Polycam to scan her face. This scan served as the basis for recreating her likeness in Unreal Engine. This digital version of her character was meant to be a surreal, otherworldly representation of herself–not a simple copy and paste.

That’s where we hit our first roadblock. Unreal Engine offered a limited selection of hair types and clothing options. None of the available hair templates closely matched Phoebe's natural hair, so we decided to incorporate a short scene into the story to justify a mid-film change in Phoebe’s hairstyle.

Clothing presented another challenge.

As with the hair, we first picked what we liked in Unreal, then we meticulously selected real-world attire that closely resembled what the character would wear in the animation, ensuring authenticity to the story's setting.

Create the Location

Next, we needed to conceive the visual world within the character’s mind. This had to be established before filming began so that Phoebe could interact with it convincingly during her performance. We sought to create imagery that would be instantly recognizable to the viewer and an extension of the live-action component.

Since our story revolved around Phoebe's character, Mel, venturing into her own mind, we aimed to reinterpret the imagery of an actual brain. The image of a tree came up as an idea, with its shape reminiscent of a brain and its branching structure resembling neurons. We envisioned the tree with sections that would glow, mirroring the way synapses operate in our brains. Additionally, we wanted this tree to exist within a vast, empty, water-filled environment, as water held significance in her surroundings: there’s a fish tank in her apartment and the film ends on a nearby bridge, overlooking a river.

Both the tree and the water elements are introduced in the film before we encounter them in the Unreal Engine sequence. Everything Mel encountered within her mind was a reflection of her environment in real life.

As we began working with motion capture and Unreal, some of the technical constraints of our workflow became immediately clear.

Facial capture wasn't yet perfected. Character and world design had their limitations, and our character motion capture wasn't exactly flawless. In other shorts we had seen, these limitations were often masked with techniques like "shaky cam"​ or clever camera angles. We wondered how much of that to emulate in our film.​

On the first day of filming, we shot the opening four-minute sequence of the film in the morning and spent the afternoon working on MoCap. In one of our previous short films, we used a full-body suit for this, but during his meticulous research time, Paweł discovered Move.ai–an app allowing the user to combine several iPhones placed in different parts of the room to capture motion and create tracking points for post-production.

The app was still in Beta at the time and just like with any technology, we encountered some early software problems, but nothing that would stop us from capturing what we needed.

We placed four iPhones in a circle around the actress and used a scratch-built helmet contraption with a fifth iPhone attached to it, pointing its camera directly at Phoebe’s face to be captured with a free app Live Link Face. If anyone walked into the studio we’d hired at that time off the street, well, let’s just say it would have looked very weird.

In less than two hours we were done, capturing all of the necessary data we needed for a nearly three-minute long sequence.

Once in post, Paweł spent two months ensuring that every movement of every limb and every facial expression captured on the day matched exactly what Phoebe’s performance had been and worked within the space. As before, we looked for honesty in performance.

Once the character was animated to a usable level, we started setting up cameras and lighting the space.

Being avid gamers ourselves, we were set on the idea that this scene would not look like a computer game, but rather have all the usual cinematic tropes that are used in film today–such as motivated backlight, depth of field, and a play of light and shadows we see in great works of cinematography.

To match the main camera, we set up all our shots with a full-frame sensor equivalent to the Sony FX9 and aimed to use the same focal lengths that the main camera used throughout filming, primarily the 35mm, 50mm, and 80mm of Leica-R. We created these restrictions to ensure that the field of view of cameras matches what we shot in real life.

What became apparent very early on was that this was not going to be a case of setting up a few lights, finding the right camera angles, and rolling with it.

Instead, Unreal Engine cinematography must be thought about in the same way as real-life cinematography: For every shot that made sense for the story progression, we not only had to figure out how to move the camera and light the character but, more importantly, we had to work on the set design around the character. In this case, the set design meant that the water, tree, and the background of the sky (known as Skybox in this project) must look good and consistent throughout the scene. As such, we had to take an approach of single-camera drama.

In the early versions of the animation, the shots lacked depth and looked kind of bland.

Working in a 3-Deminsional Space

Once Mik learned how to move the entire world around the character and adjust the props and background to the shot, things really started to make sense.

Suddenly, we were working with what could be described as a 3-dimensional space, with depth of field and the background playing a major role in making the scene look cinematic. We now had the ability to put stars in the sky in the exact place that their lighting would reflect on the water and create the right level of bokeh (the effect of a soft out-of-focus background).

Animation took a long time to figure out and put together for Paweł, but lighting and camerawork weren’t quick either.

In total, Mik spent nearly 30 days working about eight to ten hours a day on finding the best shots and lighting this sequence, from the day of the first YouTube tutorial he watched to the export of the final shot that matched the edit.

Then, due to our lack of processing power, we spent a further two weeks exporting high-resolution master files ready to be used in the grade with DaVinci Resolve.

The great thing we had going was that most of the days, two or three of us would sit together in the same room. Then every day or two we would watch each other’s work and give each other feedback on what shots worked and matched with each other.

We went deep into details such as playing with the exact positioning of tree branches to allow the right shadows to fall onto the character’s face at the right time. Consistency was the key element to ensure the sequence felt right.

At the end of the day, every short film is a showcase of the filmmaker’s ideas and skills. What Unreal allowed us to do was showcase further levels of imagination than what’s normally achievable on a self-funded short.

In Unreal, we were able to create a sequence that involved a large infinity pool and a sky dome with stars you could control. We were able to move the camera, in the same way, a high-end production would be able to move it with a drone-like aerial or a crane shot moving up from the bottom to the top of the tree.

Even underwater cinematography became possible for us.

It’s as if we were to put a large infinity pool on the floor of the newly opened Las Vegas Sphere, take control of the Sphere’s gigantic LED screen, and make a film–except we only had a computer and our imagination to work with.

Final Thoughts

Something For Your Mind taught us that, as this technology advances, its impact will almost certainly create pathways into the industry for all sorts of filmmakers who otherwise might not have been able to actualize their vision.

Maybe an Oscar nominee in a year or two’s time will be a kid in their parents’ basement after all.