Unreal Game Engine Could Change VFX Forever

The Unreal game engine renders visual effects in real-time, and its technology could change the face of post-production.

Two cars race each other on winding, scenic highway, one slowly overtaking the other. Standard car commercial? Not quite. It's being edited in real-time—and only one of the cars is actually there.

When we recently marveled at the announcement of the Blackbird, a "mule" car from VFX company The Mill that can act as a stand-in for any virtual vehicle, it seemed like a futuristic invention that might not have real-life implications in the near future. Little did we know that the company would hit the ground running. Utilizing Epic’s Unreal Engine, which computer game developers use to render visual assets in real-time, The Mill shot a short film that pushes the boundaries of both technologies, merging real-time VFX and live-action storytelling.

Using this new system, called The Mill Cyclops, the short film's director was able to make complex VFX edits in real-time. By simply pressing a button, the director could swap out the entire body of the car and alter its physical properties, transforming it from a red 1950s Chevy to a blue Volkswagen. It is, in effect, a real-time generated movie.

It combines CGI elements with the real world in real-time, creating cinematic augmented reality.

"We created a virtual production toolkit to visualize what you see in the film—a virtual car," Boo Wong, global director of emerging technology at The Mill, told FastCoDesign. "But that can be extended to any character, prop, etc. From a visual effects point of view, that’s super exciting."

By way of example, watchThe Human Race, in which the 2017 Chevrolet Camaro ZL1 competes with the Chevrolet FNR autonomous concept car.

Video is no longer available: vimeo.com/206280957

How it works

How do you combine CGI elements with the real world in real-time, creating cinematic augmented reality? 

It all starts with what's on set. The Blackbird has data capture equipment on board, including tracking markers, depth-sensing LIDAR (the technology used in self-driving cars), appendages that protrude from the roof, and 4K 360-degree RED cameras, all of which help VFX artists map the movement and orientation of the car. The artists then superimpose a CGI "skin," or photorealistic image, of a car onto the curves mapped from set. The software analyzes the footage, recognizes the position of the sun, and creates a realistic lighting system for the CGI car's shiny surface.

The fact that everything can be rendered live on set is a milestone achievement. 

On set, the director views a live video feed on a preview monitor, where the Blackbird is re-skinned in real time with the final pixels that will appear in the commercial, rendered at 24 frames per second, 60 times per second.

The fact that everything can be rendered live on set is a milestone achievement. Previously, VFX artists had to manipulate lighting in the post-production process. Now, Cyclops allows this to happen in real-time. If the DP moves a light, the virtual elements—a car, character, or otherwise—respond to that light change.

What does this mean for the movies?

In the short-term, this technology is game-changing for the commercial industry. But with time, it could upend the Hollywood VFX industry. It is, effectively, our first portal into a marriage between cinema and augmented reality.

As Variety notes, studios would be wise to utilize this technology for on-set visualizations as long as traditional, after-the-fact rendering is still the industry convention. But once Epic releases Hollywood-friendly support for Cyclops later this year, filmmakers should expect a sea change in the VFX process.     

Your Comment


Nice! That is a game changer!

March 3, 2017 at 3:59PM

Camera operator

pun intended? :)

March 5, 2017 at 1:12AM


Came here to say that the AI car looks fake af, then I realized the red one was CG too. Impressive.

March 3, 2017 at 4:34PM


Oh wow. That was exactly my thought as well- then realised the concept car is MEANT to look fake. That behind the scenes video is pretty mind-blowing.

A shiny car is a moderately easy thing to model as they go, but from here it's just a matter of GPU power to scale up to doing the full virtual characters with hair and skin on screen in real-time, too.

That is pretty awesome.

March 4, 2017 at 7:42AM

Hywel Phillips
Director / Cinematographer

I think it might be gamebreaking engine. It looks truly amazing!

March 4, 2017 at 12:35PM


That's great post. Hack coonking fever game by clicking the following link http://bit.ly/2n0vJAZ

March 4, 2017 at 3:42PM, Edited March 4, 3:42PM



A nice article, thank you.

Just a 2 cents,
I will share some of my feelings. For my film (which is nowhere finished) I am using CG and I had internal battles as to how go about doing that. This technology is truly incredible and would save me a ton of time and money (btw I love Unreal games, UE4 (Unreal Engine 4) is breathtaking))... but then not.

This technology is not free for commercial purposes (same thing with other Realtime video game rendering engines of today like Frostbite, Fox, Unity, Cry engines)), the company that created the engine asks for a cut of the commercial profits of the TV ad with CG, integrated VFX CG to live film, full CGI animation film or PS4/XBX1/PC videogame all made with their realtime video game engine. That is they grant you a commercial license to use their rendering engine for commercial purposes (making money from selling your TV ad, film or video game made with their engine). The commercial purposes costs vary, but all are prohibitive for indie filmmaker on a shoestring budget below 5000 bucks (the commercial costs can be 3000$ for one of these engines, and some ask a % from the profits you made from your creation instead/or both (which can amount to far more than 3000$ paid to them to use their engine). It's all a question of money, yet, again, and again.

I wanted to use GPU Accelerated Realtime Rendering Videogame engines, like Unreal Engine 4, since it is photorealistic depiction of Reality, same quality as ultra dogslow long rendering CG with CPUs (non-realtime rendering engines used in CG for VFX and fully CGI animation films like Pixar's Renderman Engine rendered CGI animation films) . I ended up using Pixar's Renderman technology in 3delight Renderman homologue (DNA Research company, 3delight was used to render for Jurassic World film's CG dinosaurs like the raptors and the Indominus Rex dino, and most likely, in the next Jurassic World 2, they will use it again for the CG dinos because of its stellar rendering quality (as it use Pixar's Renderman technology algorithms). It's slow, very slow, but the result is Hollywood like big buget Movie quality CG, not cruddy CG by a crap 3d rendering engine that makes the renders look as if they were rendered in 1983 ('crap Retro CGI', think Lawnmower man or original Tron films CGI : CGI at its very infancy and not photorealistic at all).

It is still incredible and opens doors for uncomplex shots. For detailed shots the video game rendering engines cannot accomplish what you need, although as somehave said, it's only a question of computational power: add more GPU graphic cards or higher transistor count per GPU card and you can render ultra intensive and memory consuming tasks such as billion-polys mesh surface micro-tesselation Displacement, camera bokeh depth of field, photographic global illumination and spectral based lighting, high sampling motion blur, and so on and so on, and you realize quick why engines fall apart when you throw them the entire kitchen (the kind of visual quality and fidelity needed for a motion picture): they are not Film Production Tested and Ready.
But only Video game Ready and Tested. That's why they are seldomly used in the cinema industry, but that is slowly changing with GPU card prices dropping; hopefully the Realtime Video game rendering engines, like UE4, will drop their commercial license costs; right now, it's just unaffordable for indies but only bigger budget productions plus there are limitations to these engines depending on your rendering hardware ($$$$)..

Just a two cents.

March 9, 2017 at 9:56PM