Exploring the Hybrid AI and Animation Workflow Behind This Standout Short From Tribeca

NFS: What was the original inspiration for the film? When did the process of bringing it to life start?

Connie He: The original spark was actually pretty simple: I couldn't sleep.

During the pandemic, I was spending all day in my apartment and became obsessed with the sounds coming from upstairs. Every night there seemed to be a new mystery: dragging furniture, loud thumps, things falling over, and my imagination would immediately start constructing stories about what was happening.

What started as a personal frustration eventually became something I wanted to explore creatively. At the same time, I'd been thinking about how visual style can communicate emotion, so those two ideas slowly merged into the film. Once we began developing it, the project became as much about experimenting with visual storytelling as it was about telling Ada's story.

NFS: What was the process like for working with the researchers at Google DeepMind?

CH: It was a blast working with researchers at Google DeepMind. So many of the researchers who joined the team were musicians, painters, or photographers in their own lives, and they were all passionate about collaborating with artists on creative projects.

One research engineer, Erika Lu, said, “Filmmaking and research share a lot in common. They both start with a goal, and while the path to the goal may not be clear in the beginning, you get to explore different approaches to get to the final result.” That really resonated with me.

I appreciate that so many of my teammates are great listeners and collaborators. They help set the tone for the team and approach the creative process in a respectful way.

NFS: What AI video generation models were used? Can you briefly outline the process for prompting and reviewing?

CH: Our film took a slightly different approach than a typical text-to-video or image-to-video workflow. Crucially, we didn’t rely on text prompting. Having tight control over animation performance, timing, and rhythm was extremely important to the story, so we decided to explore a video-to-video workflow. In a video-to-video workflow, artists design the input video to guide the generation of the output video. That allowed us to explore both 2D animation and 3D animation and find the right balance to bring them together. Each artist could also work in their comfort zone, using whichever tool felt most natural and creative for them.

Our R&D team built a bespoke pipeline to bring the painterly visual styles and the animation together. We implemented controlnets and mask editing tools for both video and image models, giving artists the freedom to iterate based on their creative vision.

We reviewed shots in dailies every day, discussing our goals and different approaches. Sometimes the solution came from an artist doing a paintover, other times we needed to develop new tools to achieve the look we are going for. In the end, it became a very organic, iterative process.

NFS: What was the editing process like for working with video generations as well as audio and sound design?

CH: Because the film is built around Ada's emotional experience, every creative choice, whether visual or audio, had to support what she was feeling in that moment. The editing process became a lot about balancing those elements and making sure they were all working together, which is something our editor, Sarah Affleck, handled beautifully.

Sound was especially important because so much of the story is driven by things we never actually see. The audience is hearing the same noises Ada hears and trying to interpret them alongside her. In a lot of ways, the sound design is what powers the entire narrative.

NFS: Overall, did this hybrid approach of traditional animation and custom video generation save time? Or did it help boost creativity?

CH: For us, the goal was never to save time; it was to explore the possibilities of what this technology could do in animation, and open doors of possibility that were closed before.

We were trying to create a very specific painterly look that reflected Ada's emotional state throughout the film. Using new tools to explore that visually allowed us to experiment in ways that would have been difficult before.

NFS: How would you like to see AI be used in film and video? How could this technology be most practically and ethically helpful?

CH: I think the most interesting applications are the ones that expand creative possibilities.

Every major technological shift in filmmaking has eventually become another tool that artists can use to tell stories, and I think that's the most productive way to think about these tools as well.

For me, the goal isn't automation, it's creative empowerment. The most exciting future is one where artists, filmmakers, and researchers work together to build tools that help people express ideas that were previously too difficult, expensive, or technically challenging to realize.