Last season of The Mandalorian, deep fake technology was employed to overlay the young face of Mark Hamill over a body double. The 70-year-old Hamill was also used for closeups, and his image and voice were then de-aged to show him in all his Return of the Jedi glory.
While the technique worked to propel the season finale story forward and introduce the Jedi to the Mandalorian timeline, many fans complained about how it looked.
Referencing the phrase “uncanny valley,” which Wikipedia defines as “the relationship between the human-like appearance of a robotic object and the emotional response it evokes,” fans bombarded the internet with critiques on how the character looked fake.
Credit: befores&afters
Moreover, a YouTuber who goes by the name Shamook took the same scenes and recreated a young Mark Hamill using Deep Fake technology to make it more lifelike and reduced that robotic vibe that fans often complain about.
We covered how fans preferred Shamook’s version of the scene, and Lucasfilm seemed to agree. So they hired him.
Luke Skywalker Returns
“[Industrial Light and Magic is] always on the lookout for talented artists and have in fact hired the artist that goes by the online persona ‘Shamook,'” a Lucasfilm representative said in a statement. “Over the past several years ILM has been investing in both machine learning and A.I. as a means to produce compelling visual effects work and it’s been terrific to see momentum building in this space as the technology advances.”
The dividends of that investment can be seen in episode 6 of The Book of Boba Fett, and it’s really good.
Perhaps too good for many fans, who all too often exist to complain. To be fair, though, Mark Hamill was in a near-fatal car accident in the late 1970s, which scarred the actor’s face. Lucasfilm had to write a scene into The Empire Strikes Back to explain Skywalker’s new facial flaw. Forty-two years later the virtual Skywalker’s scar has miraculously healed. All thanks to digital deep fakery.
Deep-What?
Deep Faking uses artificial intelligence to replace a face by comparing the subject to images of another found online. The more images it has to compare, the more lifelike an overlay can be.
With images chronicling more than 40 years of Mark Hamill to draw from, the AI that Shamook used was able to craft a very impressive image of Skywalker, without the Uncanny Valley. At least to our eyes.
Mark Hamill on 'The Mandalorian'Credit: Lucasfilm
The deep fake algorithm likely over-relied on pre-accident images of Hamill to create Skywalker, because not only was there no scar, but Hamill looked younger than the time frame the character of Skywalker is in for the Boba Fett episode.
Putting digital plastic surgery aside, the uncanny empty stare of the CGI Luke was replaced by a more alive, perhaps soulful Jedi. Coupled with the de-aged voice of the real Hamill, it was as if Disney had gone back in time and shot a few scenes with the younger Hamill.
Shamook did a masterful job, and now we all are able to enjoy Luke’s performance, rather than nitpick it as being computer-generated.
The Voice of Mark Hamill
Another gripe that some fans had was about the technique that was used to create Luke’s dialogue. A deep fake audio, if you will.
Created by the startup Respeecher, this technique follows a similar technique as Shamook used with the visual image. A computer studied audio clips from Hamill’s early career, and then replicated the tone and timbre of his voice to create original dialogue for Luke to say in the episode.
The process is known as “speech synthesis,” and because it’s still in its infancy, the technique lacks a lot of the soul behind the tone of a human voice.
The filmmakers do say, however, that the quality of the audio from the Mandalorian Luke has improved with this Boba Fett version. Many didn’t know that it was computer-generated, but those with so-called “golden ears” suspected that it wasn’t a human voice.
Advances in Technology
As both technologies mature, they will only get better. Even to the point where original characters will be created who don’t exist or are too old to play the character.
There’s already talk of an upcoming episode of a Star Wars streaming series planning to include a younger Han Solo, and perhaps Princess Leia. As many know, Carrie Fisher, who played the iconic princess passed away about five years ago.
Princess Leia in 'Rogue One'Credit: Lucasfilm
When a Leia cameo was required for Rogue One: A Star WarsStory, the uncanny valley made it obvious that a young Leia was CGI. Now, with the permission of her family, Lucasfilm has the opportunity to continue Leia’s story, as well as Han’s. And who knows? Maybe an older Obi-Wan Kenobi played by Alec Guinness will show up? Or Lando Calrissian? Or maybe even Grand Moff Tarkin will return again?
Whether that’s a good idea or not we’ll leave for a different time. But today, it seems like in a galaxy far, far away, this technology appears to have no limit.
