Italian
The future of film dubbing: How AI is revolutionizing the industry
The discomfort of watching dubbed foreign content
At present, watching a piece of foreign language content that’s been dubbed in English can be an exercise in discomfort. Thanks to the proliferation of local-language film and TV — brought to us mostly by streaming platforms — the experience of viewing a dubbed Danish crime procedural or a Spanish boarding school soap opera are nightmarish trips to the Uncanny Valley. That’s the term widely used for technology that does not suspend our disbelief and, in fact, takes human resemblance and tosses it into a blender of dystopian nightmares.
Enter Flawless, an AI-driven filmmaking studio that wants you to stream buzzy shows and still be able to sleep at night (sans visions of mismatching mouth movements and brutal scene cuts). Founded in 2018 by multi-hyphenate director Scott Mann and Nick Lynes, Flawless’ proprietary tech TrueSync maps over the faces of actors and delivers translations that are the most impressive we’ve seen in the wild west of AI. The company previewed a sizzle reel of its work at this year’s Cannes Film Festival, which Variety can exclusively share here.
The birth of Flawless and its mission
Designed to protect artistic copyrights and comply with Hollywood labor guilds, Flawless raises big questions about the value of international content and its breakthrough potential in U.S. markets. Variety can also report that Flawless is officially in the film distribution game. The company has entered a joint venture with XYZ Films and Tea Shop Productions to acquire foreign film rights. The partnership will convert the projects to English language with what Flawless promises to be “perfectly lip-synced visuals” and release them in relevant territories. The first title for release is Victor Dannell’s award-winning sci-fi romp “UFO Sweden,” which will release under the domestic title “Watch the Skies.” A full slate of projects is expected in the coming months.
Scott Mann spoke with Variety at length about AI fear and loathing in show business, building an ethical code, and the brain mechanics of the Uncanny Valley.
The journey from filmmaker to AI innovator
How did you get started in this space?
This all began in 2018. My background is directing film, writing, and producing. I happened upon what was what turned out to be the first paper that introduced generative AI to the world. It was called “Deep Video Portrait” and came out of the SIGGRAPH Conference, which is basically the Oscars for science. It blew my mind.
I’d always worked in the VFX realm, and it was clear the impact this tech could potentially have on our industry. I reached out to the authors of that paper, which led to the founding of the company with those folks and [my partner] Nick Lynes. Christian Theobalt, Pablo Garrido, and Hyeongwoo Kim founded our science department and have grown it to the leading computer-driven generative science department in the world. That’s our secret sauce.
The optimistic view of AI in Hollywood
There’s a lot of fear and anxiety around AI in Hollywood at the moment. What’s interesting about your company is that you’re optimistic.
We founded this thing from a filmmaking point of view. I love art, and essential to our mission is to see this technology as having the ability to make tools. There’s a different view that this aims to eradicate the creatives and steal art. To Napster the hell out of it, to ride roughshod and disrupt for the sake of disruption. But we believe in the expression of human stories, and film is the most advanced form of that. The tech is amplifying that and adding more layers of creativity. There are benefits. Then you have the economy of this whole business, which relies on certain protections and an understanding of how it works.
The importance of education and understanding
There’s a big lack of education around AI in this town. How does that impact your business?
This is a global issue. It’s not just Hollywood. Education and understanding are the key. There are certain companies — the flawed ones, let’s call it — that would rather not educate because it reveals a kind of underhand player. You need to have an honest conversation about how and where it’s disrupting, and where the positivity is. We’ve prioritized understanding over the past few years with the studios and the labor guilds – including SAG-AFTRA – and said, “Look, this is how this should be used.” The next [AMPTP contract] negotiation is something everyone should be starting to talk about now. There’s this transformational thing entering our industry and we should be prepared.
The industry’s response to AI
Good point. Is the industry paying attention to this in a meaningful way?
I would confidently say there’s a sea change in attitudes out there. Before last year’s strikes, there was a cautious route to greenlighting the use of AI in projects. People were having early conversations and seemed to understand them, and it came from a place of trying to improve the process. But there is a fear about crossing over. The industry is in such bad shape in places right now that many don’t want to rush into the wrong solution.
Building an ethical code
How did Flawless build its ethical code?
It emerged from problems I experienced personally as a filmmaker. I’d always wanted to fix film dubbing. I did a movie where I’d taken such care, a film called “Heist” with Robert De Niro. It was very intimately made and nuanced, and a joy to deliver what we did. I saw the foreign language version that had been dubbed. The script got completely rewritten and the mouth movements were completely different. I realized, “God, no wonder things don’t travel well.”
I got a bug to fix this problem and tried a bunch of VFX like head scanning and other nonsense. It wasn’t practical, and it didn’t cross the Uncanny Valley. This is a long way of saying you can’t do this ethically if you are removing and destroying whole areas that contribute to a film. We’ve learned that ethics and legal rights go hand in hand.
The Uncanny Valley and its impact
Speaking of the Uncanny Valley, it’s so brutal watching lip-dubbing in its current form. Especially the way streaming has escalated it.
As human beings, we study each other in great detail. If we see something is out of the ordinary, our brains are programmed to go on hyper alert. Throughout the evolution of filmmaking, our brains have gotten fooled and retrained. Your subconscious is doing the work to check for reality. If something doesn’t marry well – particularly sound and vision — you’re immediately putting a huge barrier to any kind of immersion of empathy and believability in what you’re watching. Think of how much expression is conveyed through the face and the gestures and the performance of an actor, without even hearing the sound. There’s so much in that. All those reasons make [present lip dubbing] a problem. But it’s good to have a clear problem to solve.
The potential for international films
There’s also a pretty game-changing element here for international films. It could open a new star market and eliminate a lot of U.S. language remakes of celebrated films.
Showing our materials to filmmakers, especially over the past year, they realize the potential from going to a local stage to a global stage. It’s a huge opportunity to get your work out and it’s been invigorating. They are so excited about showing their work to a wider audience, and especially in America.
The future of Flawless
Has any studio or corporation tried to acquire you outright?
Yes, when we first dropped down here in Hollywood. We had a very short promo reel in 2021 and some tried to broker a price right out of the box. Nick and I agreed that what we want is to serve everyone, not just the interests of one studio. We want to be Switzerland, because it’s very limiting otherwise. But that’s helped us raise money and grow the company. It’s a good problem to have.
You also want to get into the distribution game.
We’ve always looked at ourselves as a studio, to some degree. We want to be future-focused. You need to do the whole thing properly from pre-production, production to release. All these stages are impacted by the technology. My outlook is that we should be getting films to 10x the audience. That, and making amazing films for less cost with these tools. If you look at our first, “UFO Sweden,” it’s obviously been translated to English but also gave the filmmakers freedom to add and change things they couldn’t afford the first time around. You can choose to be like an algorithmic component and try to do one job, but I think the real power comes to making the movies and distributing. The whole shebang. And in terms of post-production, we can do more. Look at Pixar, they get to iterate their films up and to the moment of release.
Acquisitions for the JV with Flawless will be led by Ryan Black, XYZ’s James Emanuel Shapiro, and James Harris of Tea Shop.
Reflecting on the future
As we stand on the brink of a new era in filmmaking, the potential for AI to revolutionize the industry is both exciting and daunting. The work of Flawless and its TrueSync technology offers a glimpse into a future where language barriers are no longer a hindrance to enjoying global cinema. The ethical considerations and the need for education and understanding are paramount as we navigate this new landscape. The journey of Scott Mann and his team at Flawless is a testament to the power of innovation and the enduring importance of storytelling. The future of film dubbing is here, and it promises to be a thrilling ride.
