Last week we discussed the power of virtual dubbing, where AI-powered tools are used to simulate the mouth movements of different languages to create a more seamless dub. But it seems that wasn't the only company pushing this tech forward.
Now Adapt Entertainment, a company based in Isreal, is using machine learning to create seamless voiceovers and dialogue replacements for foreign language releases.
“Our goal is to broaden the global reach of content so audiences can experience enhanced storytelling that maintains the integrity of the original project without dubbing or subtitles,” said Darryl Marks, founder of Adapt Entertainment.
But how does this tech work, and what are some of the more nefarious implications it could have for film and television?
Beyond Voiceover
Fall, our previous use case, utilized virtual dubbing tech to replace expletives within the film, even though the company behind that specific software initially created the suite for supporting foreign language releases.
Adapt Entertainment’s AI-generated tools are capable of not only inserting new language or replacing dialogue but adjust the actor’s facial and mouth features to project the image of the actor actually saying the new dialogue. While we want to say we can't tell the difference, the technology still has a few more steps to tread before we can call it seamless. However, the process is still incredibly beneficial for studios looking to replace dialogue with other languages or replace dialogue altogether.
Adapt recently used machine learning to completely replace the Polish and German dialogue in the Holocaust film The Champion and convert it into English. The film, which was released in 2020, had yet to make an impact with a wider English language audience, and Marks was so committed to changing that result that he bought the domestic rights to the drama just to see if he could change that.
How Virtual Dubbing Works
The process is pretty interesting. Adapt brings in the original actors and has them repeat their performances in different languages while being recorded in front of multiple cameras and angles. The company's neuro-rendering engine, known as Plato, then uses the new camera angles as a training set to alter the mouth and facial movements to better sync with the new dialogue.
“The computer studies the face of the actor in the film, and also footage filmed when the new dialogue is rerecorded,” Mike Seymour, VFX supervisor on The Champion and Adapt’s technical adviser, explains. “Then, using advances in AI and machine learning, the entire film has the actor’s face replaced by speaking in English. It is all visually built from actors recording dialogue in a sound studio.”
The Champion director Maciej Barczewski calls the neuro-rendering technology a total “game-changer,” stating that he wanted to create the film for the English language market but couldn’t because he didn’t have the money.
“Tech technology could change the whole paradigm of international filmmaking. As a director, I love the fact that the audience is no longer distracted by subtitles anymore,” Barczewski praised. “It often happens that you lose some of the messages of the film. That will happen no more here.”
Dubbing 'The Champion'Credit: Adapt Entertainment
Not a Deep Fake?
The process digitally replaces the actor’s lip movements along with the dialogue replacement. Still, it isn’t a so-called “deep fake” so much as syncing of the film actor’s new performance in other languages. Although, I'm sure we could find a few folks that would argue otherwise.
Adapt uses the original actors to lay down the foreign language tracks with reshoots to capture the facial performance, so we could essentially call it dialogue and facial remapping if we want to get technical. The benefits are clear right from the get-go.
“It’s great because it leaves the authenticity of having the director there, the original cast there, and not taking anybody out of context,” Marks said to Yahoo! Entertainment. “What we’re trying to do is promote this content in the original format in the best light without having to do anything weird.”
And it works. But Marks admits that the same process can be done without the actors. He simply chooses to avoid the underlying stigma that the term “deep fake” implies, creating a false performance by replacing it. Marks believes it isn’t replacing but “enhanced” with other language performances.
This is where the more nefarious nature of this tech comes to light. When is an actor's performance still their performance? If you can make an actor say what you want, would you even need to hire them for the full length of the film? While we're just reaching these "what ifs?" we as creatives should all tread carefully as we move into the AI future.
Speaking of...
High Praise For The Future
Marks has proven that an entire film can be overdubbed using AI with success, and he believes that the future is wide open for an affordable way to not only release films in countries with their language tracks but older TV shows and other projects for less than the cost of doing it during principal photography. Rather than remaking a lesser-known foreign film with an A-list actor for a larger market, these incredible films can be experiences as they were originally meant to be.
“Maybe there’s no more reason to have remakes,” Marks says. “What was wrong with the original? What was wrong with Another Round with Mads Mikkelsen? Mads was great! Let’s just keep it with Mads and put it in English!”
Marks believes he can find great films in smaller markets like Africa or Asia and market them in larger markets through this process. “All these films that never get picked up because it’s a great story, but who cares about a story from Ethiopia or a story from Vietnam,” Marks adds, “but maybe now, with the technology, I can sell it to a distributor.... broaden the audience for these films and not be deterred by subtitles or dubbing.”
Adapt was recently honored by Fast Company’s Annual List of The Next Big Things in Tech, which honored over 80 different breakthroughs in technology shaping the future of industries like film and entertainment. Its goal is to have an impact on the future of how international content is consumed.
Audiences can enjoy a film without being taken out of the moment with a dub that seems out of place or unnatural. The days of just editing out the scene because of bad language might be gone as well.
As the Plato engine matures, it’s clear that AI continues to entrench itself in a beachhead as a premiere tool to open up even more exciting possibilities for getting films where they need to go and doing it in a manner that is seamless and natural. Way beyond any notion of an AI-driven deep fake. However, we all must tread carefully.
What do you think about the way AI tech is being used in film? What issues are you worried about? Let us know in the comments!
Source: Yahoo Entertainment
