Find out in this interview!
Imagine stepping onto a film set that lets you capture a golden hour that lasts the entire day, change the weather, move photoreal background mountains, see a blockbuster-quality VFX creature interact with actors, or scout for locations around the world, all without leaving the room.
Now imagine doing this entirely in real-time.
That’s what happens when you use LED virtual production technology. And the good news is it’s no longer just a figment of our collective imaginations or even just a demo at trade shows.
Thanks to its successful use on projects like The Mandalorian, this technology is real, on its way to going mainstream, and already changing Hollywood.
But what goes on under the hood to bring this cutting-edge tech to life, and what should filmmakers expect when shooting on an LED virtual production stage?
We speak with one of the tech pioneers at disguise to see how it all comes together. Learn more below!
Credit: disguise
What Is LED-Based Virtual Production?
At its most basic level, LED virtual production is what happens when a combination of technologies lets filmmakers replace their green screens with walls made up of LED panels. With the help of a game engine, these LED walls display real-time backdrops and visual effects, all directly on set.
“This creates a far more immersive experience for the entire film crew compared to using green screens,” said Ed Plowman, CTO of disguise.
He’s spent the past three years developing an LED-friendly virtual production workflow that has helped companies like Framestore, Lux Machina, and High Res deliver films including Solo: A Star Wars Story and live-action commercials for Nissan and Hyundai.
“Using an LED virtual production stage means filmmakers can see—and capture—all the final pixels of a shot in-camera on set,” Plowman said. “All without waiting for post.”
According to Plowman, other benefits of using LED stages include:
- Post-production teams will no longer have to spend time keying from green screens.
- Post-production teams will automatically have correct reflections and contact lighting on actors or physical set pieces, as well as no more green spill.
- If a director wants to change the angle of the light so that the shadow on the actor goes in a different direction, the director no longer has to move any real lights. They just have to move the sun in the virtual LED environment.
- Actors no longer have to visualise the final film with only a green screen for reference. Their eyelines will be more accurate when they look at Computer Generated (CG) elements.
- Virtual cues can be placed for anyone on set using LED floors.
- Directors and cinematographers will be able to see, modify and sign-off on background locations and visual effects in pre-production (even doing virtual location scouts) so there are fewer re-shoots or iterations needed in post.
- There’s no need for travel. Any world can be shot from one location.
- Large-scale set extensions can be created to seamlessly match with the LED backdrop in post, just in case the LED wall doesn’t fill up the whole camera view.
- Dynamic augmented reality elements can be seamlessly added for LED assets that need to move in front of your actors.
Credit: Framestore, All of it Now, XR Stage
LED virtual production is revolutionizing what can be done on set—but it’s also redefining our production pipelines.
Work that is typically associated with post-production needs to be moved into pre-production, in order to build imagery for the LED screens. This requires a massive cultural shift. Instead of "fixing it in post," visuals need to be ironed out before even walking onto the first day of shooting.
For Plowman, however, this simply goes back to the way moving image pipelines were originally built.
For example, Disney’s multiplane camera rig, used in 1937 for Snow White and the Seven Dwarves, involved a camera facing down towards multiple layers of glass panels that had backdrop art painted onto them.
These panels would then be manually moved past the camera one-by-one at various speeds and distances in order to create a realistic sense of parallax, or depth. Forest sequences would feel more real as the trees would move more quickly past the camera than the moon, for example.
“In early films, all these kinds of effects would happen in-camera, on-set through tricks of perspective,” Plowman said. “That is exactly what we’re doing too—moving the creation of effects back onto film sets. Only now, we’re applying technology to them.
Credit: Orca Studios
Thanks to real-time tracking data, today’s advanced virtual production tech will also understand the spatial setup of the stage and where the camera is in relation to the screen.
“That means your visuals will constantly recalibrate according to your camera’s real-time position, so they’ll have the right perspective warping to look accurate in-camera for the final shot,” Plowman said. “In other words, there’s no need to worry about the panels looking like a simple flat video screen backdrop. It’ll always feel like real, three-dimensional space that you can zoom, pan, tilt, and dolly through.”
And that’s just if you're just pushing images straight to the LED wall, then capturing them in-camera.
“You can also add augmented reality elements to the scene as extra planes that sit in front of your actors,” Plowman said. “You can make a 3D dragon that flies through the scene, seamlessly going from the LED backdrop to flying in front of the actor, then back into the LED again. And you can use the LEDs from the dragon to light that actor, or even get the dragon to do some basic interaction with the actor.”
Credit: disguise
Breakdown of an LED Virtual Production Setup
There are multiple technologies that can make up a state-of-the-art LED virtual production setup. Some of the basic elements include:
LED panels. These can be as bright as 6000 nits. In the early days of LED virtual production, many stage layouts looked like an open-faced cube with an LED floor and walls.
“Turns out that's the worst possible configuration you can have,” Plowman said. “In the corners, you've got two sets of lights fighting each other, so if you observe it from the wrong angle, you see a seam.”
Modern LED stages—or volumes—now typically have a Mandalorian-style curved screen. These wraparound layouts are called a "cave" and can come in huge sizes.
“I’ve seen stages large enough to drive a car between the two sides of the curved LED screen,” Plowman said. “Luckily, the car was on a track so wouldn’t risk damaging the LEDs!”
The quality of the LED panel typically depends on what is called the pixel pitch—which describes the density of the pixels (LED clusters) on an LED display that is correlated with resolution.
The real-world camera, which will have a tracking device on top of it like Ncam, Stype RedSpy, or Mo-Sys StarTracker.
Generative or real-time content from a game engine like Unreal. This content can be completely CG, or a real location that has been scanned with LiDAR or photogrammetry. This real-time content will then feed into the LED screens.
Content mapping software and hardware, like disguise’s rx and vx hardware units and r18 software. These will help to render the visuals out of the game engine and accurately map them onto the stage with minimum latency. The system ensures the LED visuals always adapt to where the camera is in three-dimensional space. Content can also be fed through an ACES pipeline.
Credit: disguise
Tips for Shooting on an LED Virtual Production Stage
As with any disruptive technology, workflows have to change in order to shoot on an LED virtual production stage.
“There are cultural barriers to overcome between people on set,” said Plowman. “There’ll typically be one person looking after the traditional lighting and another person who looks after the media server and LEDs, and then there’s the cinematographer. If the cinematographer asks to set up a stage in a certain way, and then the person in charge of the media server/LEDs tells them you can’t because it won’t work with the LED, that’s a problem."
The most important thing to remember, according to Plowman, is that shooting on an LED virtual production stage isn’t like turning up to shoot something physical—and it’s not an all-in-one solution for every single type of shot.
The best way to know what capabilities it can bring is to take a day ahead of time to test out what a virtual production volume can do, before starting a project.
“There are plenty of sequences that should still be captured on location,” Plowman said, “but there are sequences—like those that would involve a green screen—where using LEDs can completely transform your workflow for the better. The only thing to remember is that for the LED shots, you need to do the work upfront. You will have less post-processing work to do, so you have to come prepared to do a lot of pre-visualization work ahead of time.
“If you do that correctly, on the day, you have more creative freedom than you would if you were trying to fix everything in post. You’ll be able to see the final shot there and then, with both CG and real worlds combined together. When we talk about movie magic, this is what I’d imagine. That immediacy and impact of seeing the full picture on set rather than waiting weeks to see it on screen.”
What's next? Check out more about virtual productions
Learn how to prepare for working in the volume as a cinematographer. Check out options for LED backdrops (and here's another one). Learn more about how this technology was used on The Mandalorian.
