December 14, 2015

Incredible Software from Disney Research Seamlessly Blends Faces from Different Takes

Did you like the performance of an actor in one take versus another, but the camera move only works for one or the other?

Or maybe you've decided that a performance should change more gradually in a given take. All of that is possible with new software from Disney Research and the University of Surrey called FaceDirector, which is aimed at letting you choose which facial performance you'd like in a given scene from multiple takes:

For those crying foul about manipulating faces or the integrity of the scene, these things are already being done all the time. Another common technique is actually physically removing the entire actor rather than just changing the face (which requires more work). This is something David Fincher likes to do quite a bit, especially when the camera move is particularly difficult or he wants to use different takes of a wide shot at the same time.

It's worth noting if you didn't realize already that this is still in the research phase, so don't expect to be able to purchase a copy of FaceDirector tomorrow. Here's more on their technique, which avoids the 3D reconstruction that's typically employed today:

We present a method to continuously blend between multiple facial performances of an actor, which can contain different facial expressions or emotional states. As an example, given sad and angry video takes of a scene, our method empowers a movie director to specify arbitrary weighted combinations and smooth transitions between the two takes in post-production.

Our contributions include (1) a robust nonlinear audio-visual synchronization technique that exploits complementary properties of audio and visual cues to automatically determine robust, dense spatio-temporal correspondences between takes, and (2) a seamless facial blending approach that provides the director full control to interpolate timing, facial expression, and local appearance, in order to generate novel performances after filming. In contrast to most previous works, our approach operates entirely in image space, avoiding the need of 3D facial reconstruction. We demonstrate that our method can synthesize visually believable performances with applications in emotion transition, performance correction, and timing control.

There are tons of potential uses for this software, and while it did seem to work when the camera was moving, I expect the best looking result to come from a scene where the actor is in a similar spot with similar lighting. Though this could be considered part of the "fix it in post" method, it's very possible that the two great reactions you'd like to use are just in different takes. To me, that's not using post as a crutch, but as a way to utilize every piece of material you shot on set. When there's no money to go back and reshoot, if tools like these can be made affordable, they could be a huge help. 

This also isn't Disney Research's first rodeo with this sort of thing, they've been developing all sorts of amazing technologies, including automatic redubbing and new techniques for creating HDR video:

For more on FaceDirector, check out the in-depth paper here.     

Your Comment

13 Comments

See I kind of a agree, but at the same time, in editing we already do this by cutting away and cutting back to a different anyway. So... Like I agree it's disturbing for sure, and it feels like we remove "something" from the actors, like them basically. But like i said, we do this already, we have already been doing this and just hiding the cut with reactions and inserts and cut aways. So its interesting and I definitely am not against it completely, but I'm not for it as a goto post tool... Because it arguably would be easier to get the actor to do it right on set, and cheaper. But I suppose this could help remove the need for spending money on pick ups for pointless reaction shots for a scene that you already shot.
Since Disney is the one backing this, I can only assume they've been playing with it in their latest movies, their super hero movies.

But I felt there was an emphasises on "video" rather than "celluloid" is it doable with film or no (for whatever reason) ?

December 14, 2015 at 11:16AM

0
Reply
David Carmichael
Shooter, editor
107

In an interview recently, Walter Murch talked about the increased ability to manipulate the frame in post—the sanctity of the frame when shot vs. the ability for the editor to recompose shots, etc. He mentioned that the terminology is even changing, from "shooting" to "acquisition", which could be based on the idea that instead of shooting a frame of a movie, you're acquiring stuff to really make the movie later.

https://vimeo.com/143188593, around 4:00 onward.

This research from Disney certainly would fall into that ethos.

December 14, 2015 at 12:16PM

1
Reply
avatar
David S.
2716

This is the sort of shit that gets David Fincher rock hard!

December 14, 2015 at 12:21PM

6
Reply
Filmmaker Dude
Film stuff
182

I have so many problems with this. But it is kind of amazing that they can do this to this extent. But notice how controlled each of the environments are. we are always looking at the actor straight on, full in the face. with very little changing about the angle. Even in the handheld example.

December 14, 2015 at 12:48PM

6
Reply
avatar
Michael Markham
Actor/Filmmaker
1053

I feel like that critique is more about how the technology is right now vs its potential.

December 14, 2015 at 2:23PM

10
Reply
avatar
Sean Voysey
Creative Director
330

No, I have a real problem with the potential. You are stealing the artistry from the actors. Trust your actors. Hire good actors. They will make magic happen. I almost see this as a violation of what the actor does and what they bring to the table. And whenever I've seen directors use different takes in 2 shots, something is lost. Fincher has done it better than anyone, but his films have begun to feel dead inside and I think it's that level of control that is part of the problem. You loose the immediacy of the actors performance, you loose their humanity, and therefore I think you loose the ability to really affect people deeply.

December 14, 2015 at 11:55PM

0
Reply
avatar
Michael Markham
Actor/Filmmaker
1053

Well sure, if your film isn't abstract. This technology would be great if you're trying to represent an alien species, or maybe an imaginary sequence where a person's emotions seem to be changing at an impossible rate. Or maybe you want to represent a glitch in time, where out of no where everyone's emotions but the main character's switches gears. You could act that naturally, but a scene like that actually calls for the artificial.

December 15, 2015 at 11:24AM, Edited December 15, 11:25AM

6
Reply
avatar
Danny Bittman
Director / Writer / Media Composer
168

Fincher will absolutely love this.

December 14, 2015 at 6:58PM

6
Reply
avatar
Terma Louis
Photographer / Cinematographer / Editor
1695

auto-tune for actors

December 14, 2015 at 10:00PM, Edited December 14, 10:00PM

10
Reply
Ricky
Director/DP
288

I can already hear a major news TV station editor asking: can you add a little more fear in his face.

December 15, 2015 at 4:56PM

0
Reply
avatar
Vladimir Miketa
Cinematographer & Editor
337

It would be sweet to have this as a quick and simple solution for removing unwanted blinks!

December 16, 2015 at 12:34PM, Edited December 16, 12:34PM

0
Reply

Pair that with already existing voice-altering software, Voila! Bogus footage for the news, courtroom becomes one step more achievable.

December 16, 2015 at 2:12PM

5
Reply
avatar
Emeka Akwuobi
Filmaker/Editor and other stuff
194

Crazy tech and it is awesome that we have gotten this far. But I feel it takes too much away from the actors and removes part of the art form.

December 17, 2015 at 11:49AM

6
Reply
avatar
Zack
86

I mean this is kind of neat, though the morphing of the faces is very visible.

December 17, 2015 at 12:05PM

0
Reply
Kyle Dockum
Videographer and Editor
1062