How Adobe’s Generative Fill Will Change Your Approach to AI Video Editing

Like many who work in the film and video industry, I have plenty of friends who also work in design or creative services roles. And trust me, while AI is wreaking havoc on the professional video world, it’s absolutely demolishing the creative design profession as well. (After all, as we’ve established, there are indeed some elements of video editing which AI probably won’t ever truly replace.)

Still, with the news of a new Generative Fill tool coming to Photoshop, we can’t help but wonder not just what new generative AI tools Adobe (and other NLE manufacturers) plan to release to apps like Premiere Pro and After Effects here soon. And that’s really the takeaway from this news. It’s not a question of if anymore — it’s simply when.

Let’s take a deeper dive into this new Generative Fill tool for Photoshop and explore how it might be a good leader for what other generative AI tools we can expect to shape the future of video editing as well.

Generative Fill for Photoshop

Launched as a new beta tool in Photoshop, Generative Fill is designed to help creators “jump-start their creativity” by helping them make stunning updates to their images with simple text prompts. Just as you’ve seen with MidJourney or other generative AI apps like Runway ML (which is already a great case study for generative AI video editing), this new tool is just one of the first coming to Adobe’s new suite of AI-powered tools and features.

Generative Fill, in particular, is meant to add, extend, or even remove content from your images in a non-destructive way and by simply using text prompts. The results, when done correctly and for the right project, can indeed be quite impressive as Adobe’s AI attempts to match photo-realism with additions that can be rendered in seconds.

However, Adobe is being a bitwise here in this rollout. After announcing Adobe Firefly at NAB, there’s been a lot of speculation about what AI-powered tools the company will plan to weave into their apps first. And while Generative Fill is indeed quite impressive, it’s really only meant to be supplemental or additive to your projects; it’s not meant to replace design or photo work altogether — just yet.

Generative AI Tools to Come

Also, as we begin to dive into this new Generative Fill tool a bit more, it’s interesting to note what else this tool can do besides just the headline-catching elements. This new tool is actually a bit more powerful than its marketing is letting on. Along with the ability to generate object backgrounds on a text prompt, it’s also capable of generating objects (and possibly even people) while extending images and even removing objects (which, to be fair, Adobe has offered for some time).

Here’s everything Adobe promises Generative Fill will be able to do today.

• Generate objects: Select an area in your image, then describe what you’d like to add/replace through a text prompt. 
• Generate backgrounds: Select the background behind your subject, then generate a new scene from a text prompt. 
• Extend images: Extend the canvas of your image, then make a selection of the empty region. Generating without a prompt will create a harmonious extension of your scene. Generating with a prompt will add content to your image while extending the rest of your scene.
• Remove objects: Select the object you want to remove, then generate without a prompt to let the Generative AI technology make it disappear. 
• And more… Generative Fill is incredibly versatile. Experiment with off-the-wall ideas, ideate around different concepts and produce dozens of variations in a snap.

The real question now is: what will Generative Fill AI be able to do with video editing tomorrow?

More generative AI is 100% coming to Premiere Pro soon too.Credit: Adobe

Generative Fill for Video

I have no doubt in my mind that if Adobe can roll out this Generative Fill tool for Photoshop today, a similar tool could be unveiled for Premiere Pro tomorrow. The limits aren’t necessarily on the AI side, but of course, rather on the processing side. This rollout of Generative Fill as a beta with Photoshop is most probably a great way for Adobe to judge interest and computing power before it ramps up generative AI tools into other programs.

However, as we’ve mentioned, there have been AI tools and other smart features in Premiere Pro and After Effects for years. And while some are indeed cool and useful for certain situations, I don’t think Object Removal and Content-Aware Fill tools in AE have broken the internet or the industry. Instead, they’ve been fun to try out and use occasionally, but most editing pipelines and workflows haven’t used them too much.

Even if Generative Fill were to be released for video editing tomorrow, I can’t think of too many examples of it being useful on a macro-scale. And, even if it were, the amount of micro-management you’d need to do frame-by-frame, even perhaps for, say, turning a vertical iPhone shot video into a 16x9 composition, would be possible — but quite time-consuming regardless.

Still, we can also probably rest assured that Adobe (and everyone else in this space) is doing its diligence and research to find the best ways to roll out more generative AI into their video editing products and services. And if you think this fad might pass by, you will be sadly mistaken.

What do you think of this new Generative Fill tool for Photoshop? Do you see yourself using something similar if it were to come to Premiere Pro? Let us know your thoughts in the comments below.