Why Netflix Is Using Machine Learning To Train AI To Do Match Cuts

To be fair, there are a lot of exciting—and arguably good things—going on in the world of AI right now when it comes to filmmaking and video production. We’ve covered lots of cool AI tools and features designed to make the life of a video professional easier and more streamlined.

However, as we’ve also covered, there might simply be certain elements of filmmaking that AI will (probably) never truly replace.

One of those areas of filmmaking expertise might have to do with video editing itself, and the many nuanced and creative decisions which a video editor chooses to make to turn a bunch of footage into a cohesive, and emotive, story.

Yet, despite this Netflix has recently let it slip that they’re hard at work feeding hundreds of thousands of hours of footage from their streaming shows and movies through a machine learning algorithm to train a new AI to craft match cuts.

Let’s explore everything we know about this news, as well as provide some helpful links and resources to any non-AI-bots reading this article who might want to hone their own human skills in case they want to compete with these future Netflix-fed AI algorithms.

What Is a Match Cut?

Let’s start with the basics: what is a match cut?

In theory and practice, a match cut is nothing more than a simple editing technique that cuts between two shots that share something in common. You’ve seen these match cuts in action in everything from Kubrick to Spielberg, in television commercials, and in your friend's wedding film. It is a true staple of video editing.

Here’s a great video that provides some further context and examples:

As you can see, while the match cut can be a general term, to be more precise about match cuts you might want to focus on one of the three different types of match cuts out there.

The Different Types of Match Cuts

What are the three different types of match cuts you ask? Those are usually divided into graphics, audio, and movement. Here’s a little more info about each:

• Graphic Match Cut: With the graphic match cut, you focus on one particular thing in the frame and change everything else between the shots. That could be a specific object, shape, or even a color.

• Audio Match Cut: With the audio match cut both shots are completely different visually, but what connects them is a similar sound or a line of dialog.

• Movement Match Cut: For the movement match cut, the same camera motion, or action is carried between the shots.

While each is unique to its own properties, you’ll often see match cuts that combine these different types. But the overall goal of a match cut is to subtly (or sometimes overtly) make a thematic connection between two scenes that otherwise might not appear to be connected or relevant to each other at all.

What Netflix Is Doing With the Machine Learning?

Now, let’s look at what Netflix is doing with machine learning and match cuts. In a video featuring different creative technologists and machine learning engineers, we have these Netflix developers take us through their recent work of trying to train an AI program to learn how to pull off match cuts.

Netflix certainly does a good job of explaining how they’re going about this process by implementing some pretty sophisticated techniques like instance segmentation and optical flow—both complex machine learning systems which will undoubtedly help them break down and quickly algorithmic thousands of hours of footage.

The original goal for this project appears to be simply to help Netflix editors cut together trailers more quickly and easily.

However, as the team describes in the video, eventually this technology could be trained and used for cutting together new shows and content based on these algorithmic suggestions.

Why It Probably Won’t Work for Them

There’s a lot to unpack here, and, for its part, the internet (and Twitter in particular) has done its job of lambasting this video and its promises of machine learning match cuts. The biggest issue which many are pointing out is simply that a match cut is not a finite task. Match cuts come in all shapes and sizes and are largely dependent on emotion and creativity beyond just simply framing or movement.

Yes, there are concrete elements to match cuts that an AI could certainly recognize, which, putting Netflix’s work in the best light, is what the team is trying to do. However, I’d personally argue that it's a bit of a faulty premise that Netflix video editors are wasting tons of time searching for similar shots to find match cuts for when they’re editing trailers for new Netflix shows.

Ultimately, only time will tell as to how helpful this work might be or even what Netflix might eventually choose to do with this new machine-learned information. But if our gut tells us anything here, it’s that this machine learning AI match cut program probably (hopefully) won’t work for them.

How do you feel though? Any thoughts on this new AI match cut program from Netflix?

Let us know your thoughts in the comments below.