August 17, 2014

Disney Engineers Invented an Algorithm That Automatically Edits Multi-Cam Footage

Disney Automatic EditingWe all know that editing is an incredibly complex craft, one that requires not only an immaculate sense of timing, but also an in-depth knowledge of narrative structure. The edit is, after all, the final re-writing of the script. With that said, editing can also be, well, a pain in the ass, with hours on end spent making minuscule changes. But what if an edit, or at least a competent rough cut, could be done with an algorithm designed to choose the best shots and string them together with continuity? Well, a group of engineers with Disney Research have done just that, and they've put together a brief video explanation of how it all works.

Before we get to the video, it should be noted that the primary focus of this algorithm seems to be social situations where multiple people are filming the same content with the camera phones that have become ubiquitous in modern society. However, with the way that the algorithm is designed (with a focus of the rules of editing and cinematography), this technique might very well find practical application in the world of narrative and documentary films and television.

While this technique is absolutely fascinating for a number of reasons, it's obviously never going to replace an experienced editor. With that said, depending on how advanced and accurate the algorithm becomes, it could very well become an invaluable tool to assist editors with things like footage organization and rough cutting. Just imagine dumping your footage at the end of the day, leaving the algorithm to do its thing overnight, then coming into work the next morning to find a coherent and workable rough cut of the previous day's material waiting for you.

If you'd like to learn more about the science behind this algorithm, head on over to the Disney Research press release and download the full PDF.

Link: Automatic Editing of Footage from Multiple Social Cameras -- Disney Research

[via Videomaker]

Your Comment

27 Comments

Blimey!

August 17, 2014 at 5:54PM, Edited September 4, 11:56AM

1
Reply

That is impressive. i would use it. That overnight rough cut suggestion is exciting..

August 17, 2014 at 6:33PM, Edited September 4, 11:56AM

0
Reply
Donald Denis

"Just imagine dumping your footage at the end of the day, leaving the algorithm to do its thing overnight, then coming into work the next morning to find a coherent and workable rough cut of the previous day’s material waiting for you."

and maybe, just maybe, you loose the chance of, by loosing yourself in the footage, let your intuition came with possible creative and different solutions too... some of the great moments with montage in cinema came from this possibility. Or as William Burroughs would say, when you get lost, you find the way! :)

But thats OK, since the religion of technology (as David Noble, former M.I.T. history of science professor calls it) is all about we surrendering our rights and dutties to machines. :) We fucked the ecosystem we live in, so lets give the responsabilities to our mighty creation! :)

After all, who needs intuition (that thing only david carson, the surfer designer, seems to care about! :D) when we can have precision and velocity (as Paul Virilio defines it: a military take over of our imaginary worlds) :)

August 17, 2014 at 8:13PM, Edited September 4, 11:56AM

5
Reply
guto novo

Right because that is what happened with every other bit of filmmaking that was improved by technology. It could be a great tool and just like every other tool is not a replacement for talent and story. The cream will continue to rise to the top.

August 17, 2014 at 9:52PM, Edited September 4, 11:56AM

7
Reply

"Right because that is what happened with every other bit of filmmaking that was improved by technology."

this is a mantra we love to keep saying but filmmaking as a storytelling platform is the same since ever. Filmmaking is what? a 2D screen projecting a sequence of images that create a poetic meaning via juxtaposition. Like PAsolini used to say, cinema is first a poetic medium and later a narrative one. So that is the same since the beginning of this medium. It works like a dream, with dream logic most of the time (unlike realistic novels for example). So what´s improving?

What a new technology can do to make cinema better than it was in already in the hands of Ozu or Tarkovski or Bergman or Bresson?

Basic addiction have already been made: sound and color.

All that came later are little bits of fine tunning for those basic addictions. They made cinema better than it was already in the 50s?

Sure they made the spectacle part of cinema more powerful. But even with all of that, do you think cinema, as an art form, needs all of that to create "better" art than those created by the old masters?

Like painting... some new addictions were made, from brushes to different type of paints (like acrylics), but a David Hockney or Francis Bacon painting, even with new technology, still are under the principles of the old masters from italy. You can make cool painting with an Ipad, procreate and a adonit stylus too, it is a different technological tool, you don´t have the texture of real painting (the movement of the brush strokes bumping in the canvas), but the principles of painting are all there, the same principles that makes painting an art form.

Cinema as art is simple: a 2D canvas with movement and poetic logic onto it plus storytelling.

Stereoscopic, surround sound, etc, are great stuff for the entertainment part of it. Sure. But as a storytelling medium, all you need is this, a camera, a sound recording and a projector or 2D screen. If you change that, it´s not cinema anymore, it´s something else.

So to create a mood, to express a vision, to tell a story to grab the audience soul, you don´t need more than that. Check out Cassavetes movies! probably among the best studies of human crazy behaviors under the influence :D of love, lust, desire, etc, ever made. It gets you in the gut. It don´t need more than that, and it works today the same way it worked 40, 30 years ago.

New technology would make those movies better? I don´t think so.

Somebody in another comment said it well, the masters of storytelling today are making movies that are slower, and also without the use of fancy new technologies.

The blockbuster fun movies for creating cash for big studios, those need the new technological stuff to make it sound new, young, fresh, but those movies -they can be fun - they have the power of a Cassavetes or Tarkoviski movie in create a resonance with our inner self? to create a dialog with the things that make us to be what we are? That´s what art is about.

And for the art, cinema is complete for a long time, like painting is complete as a technological platform for a much longer time!

I´m not against virtual reality, videogames, and stuff, but it´s not cinema, it´s another thing, not better or worse, but different, and it can carry storytelling principles, since life is a storytelling process, so we apply storytelling in almost everything.

" It could be a great tool and just like every other tool is not a replacement for talent and story."

sure it can be. But again, it would change the process. And the process is part of every art form. Chance the process, and you will see a change in the quality of the art... for the better or the worse.

" The cream will continue to rise to the top."
not always. Shane Carruth, he is a filmmaker I´ve learn to love, and deserve the top higher than the top he gets! :D the same could be said about Cassavetes in his time too.

Again, those phrases are mantras, we can believe them blindly or we could be critics of them too and see a little deeper :)

and maybe this fever for technologies to save cinema is doing the opposite. :)

"If cinephilia is dead, then movies are dead too . . . no matter how many movies, even very good ones, go on being made. If cinema can be resurrected, it will only be through the birth of a new kind of cine-love. " - susan sontag
http://www.nytimes.com/books/00/03/12/specials/sontag-cinema.html

pardon my bad english here. and if this tool is what you need to let your intuition - that thing that feed artists since ever - go for it! :)

August 17, 2014 at 10:54PM, Edited September 4, 11:56AM

0
Reply
guto novo

New technology doesn't make movies better* directly, it makes movies easier to make. Which allows more storytellers to create a higher quality product, more people to access that product and more people to experience an untold story.

I for example suffer from fibromyalgia, sarccoidosis and am a paraplegic. I would not have had the physical ability to make a film 25 years ago, however, today I can and make a moderate living with my camera. You just may have enjoyed one of my movies, that was only possible to bring to you with modern technology.

So, can I see your work? I have a functioning Kinetoscope here, or is that to far into the machine cult for you, perhaps you could send me your latest flickbook?

*Depending on your definition of better, personally, I feel that sound for example improved the movie experience, I think colour is a brilliant tool that can add a lot to a movie etc etc.

August 18, 2014 at 5:01AM, Edited September 4, 11:56AM

0
Reply
William Jones

I'm shocked that they would even want to try to develop something like that.

I'm calling B#LLS$&T on this one. They are neglecting to tell us the 3D tracking side of this and how much time they loose doing that. I doubt Boujou could have handled the basketball footage let alone anything that is meant to be fast enough to save time.

Congratulations Disney Corporation on another attempt to kill the art to improve your bottom line. You should never have killed cell animation.

August 17, 2014 at 8:25PM, Edited September 4, 11:56AM

1
Reply

Remember there was already a company that tried to do that but it was useless, it was Avid in media composer, this option was ridiculous...

August 17, 2014 at 9:08PM, Edited September 4, 11:56AM

5
Reply
JDG

One thing that a computer algorithm will never be able to do is find the human element through editing two shots back to back creating a new thought or feeling. Art is thought and emotion not just something that is technically correct.

August 17, 2014 at 9:09PM, Edited September 4, 11:56AM

3
Reply
Bill Fowler

Again, this kind of technology will never replace an experienced editor. It does, however, have the potential to be a useful time-saving tool for editors in the future, one that might help them focus more on the art and emotion of what they're doing.

August 17, 2014 at 10:02PM, Edited September 4, 11:56AM

0
Reply
avatar
Rob Hardy
Founder of Filmmaker Freedom
4489

I actually think it may have more applications in the realm of corporate videos and non-professional consumer footage. Down the road, and in a bad-case scenario, this could lead to a lot of the more available video editing jobs being lost.
In terms of its use in editing narrative scripted work, though, I see almost none, at least professionally. Unless the director, editor, and writer all have no idea what they want out of their movie, the software stands to either disorient the emotional intentionality of the movie's scenes or pollute the editing staff's existing ideas with its own sequence of content. (Admittedly, the latter is less of an issue as long as those affected by it are also okay with it.) Besides, only the most inefficient productions would have so much extra footage of their scenes that they could just pile it in to such a program and press "enter".

(Side thought: For this to be more effective, it should be combined with audio analytics as well...)

August 19, 2014 at 9:56AM, Edited September 4, 11:56AM

1
Reply
Mr Blah

I don't fully understand those talking about human inspiration in the editing room; I thought films had a shot list and illustrated storyboard visualised in pre-production anyway. Bearing that in mind, could someone please give me an example of any type of serendipity moment in the editing room.

August 17, 2014 at 11:35PM, Edited September 4, 11:56AM

5
Reply
Saied

Search for articles about the intuition process in the montage of. Bonnie and Clyde. The disjunctive technique was born accidentally in the editing room. Btw, that movie is also a master class in the art side of editing... And keep in mind not all movies are shot the way you described. For example, wing of desire from wim wender, not even a final screenplay was available during shooting... When approaching peter falk to work in the movie, there was just an idea of what the movie would be, the movie, like a sculpture, was made during the making of it. And it is one of the most poetic movies in history. :)

This tool may be great for technical process, like news, journalism, indeed, but as a way to speed up the process in art, it may change part of the process of discovering what is inside the artist/director/editor thru what is apparently "irrelevant" in the acquired footage. and that would kill the magic of intuition. :) if you get interested in it, search too for the process used by David Carson, the surfer designer, in his workshops, may be useful for you to develop more the intuition that usually is born in the process of getting lost in the material to reach a point of insight. :)

But Bonnie and Clyde is a fine example that the process of montage in a film, as art, is poetic and artistic, and also that art can be fun and crazy too! :)

August 18, 2014 at 12:43AM, Edited September 4, 11:56AM

0
Reply
Guto novo

Saied - a shot list doesn't tell you how long each shot should last nor how the montage should be spliced together. So, maybe you have a sequence lasting 20 seconds but one editor will drop it to 6 and others still will cut it into three 3-second pieces in between other shots (like an actor reaction or B-roll, etc).
.
As a side note - on YouTube, there should be a Deep Purple concert shot by half a dozen folks on their smartphones that was then edited into a multi-cam footage after the concert. The audio is still crap - compared to something taken off a mixing board - but the video portion looked almost professional with the shakiest parts and inaccurate live zooms left out on the cutting room floor. (as it were)

August 18, 2014 at 12:58AM, Edited September 4, 11:56AM

0
Reply
DLD

I didn't know that of Wings of Desire which is one of my favorite films. Alekan's cinematography is wonderful. I also recommend any Tarkovsky or Bresson for understanding time in cinema. They are both masters at it albeit with very different editing styles.

August 19, 2014 at 8:39AM, Edited September 4, 11:56AM

6
Reply
Ghostdog

Saied. If you are truly interested read First cut, conversations with film editors, It will open your eyes to the aspects of an editor that you seem to question. Wonderful insights into how some of the most wonderful scenes in film history were created due to the editor not having enough footage or the right footage and having to think outside of the box to create something even remotely watchable. Thats just an extreme example. It happens in every scene on every detail.

August 18, 2014 at 10:16AM, Edited September 4, 11:56AM

0
Reply
seth

Thanks Seth (and Guto novo, DLD), I think I'm a little clearer on it now, and will probably get the book you recommended.

August 18, 2014 at 3:34PM, Edited September 4, 11:56AM

0
Reply
Saied

As a news photojournalist (cameraman) who was called in to work this morning to shoot and edit reversals of a reporter conducting a Skype interview (I know), I applaud anything that has the potential to allow us to brush past the sausage-making and spend more time on the craft side- I never hear anyone lamenting not calibrating their cameras to scopes before a shoot (as i typed that sentence i felt hysterically old)

August 17, 2014 at 11:46PM, Edited September 4, 11:56AM

8
Reply
Robert Bikel

This video kind of sums it up: Humans Need Not Apply
http://kottke.org/14/08/humans-need-not-apply

August 17, 2014 at 11:56PM, Edited September 4, 11:56AM

5
Reply
Wayne

Indeed, it's what is happening, but we are I denial , as usual.

The thing is,human biology does not need to evolve faster, it has all we need inside it already.

Check out the hammeroff and penrose theory for how the brain works, it had recently its first confirmation with enarmonics ultrasounds in neuronal tubules - which is a problem for the wet dream of transhumanists. That is why Ray Kurzweil gets irritated when the first conformation of their model happened! :D

Quantum biology was a field major biologists used to make fun about, like epicene tics was also in recent past - since we should be defined only by our genes! :D - but now it's a serious field, i mean, macro world can show quantum effects, from plants to birds, as far we are learning.

What this means? The potential hidden in our biology is still a mystery.

We gotta the weird placebo effect, for instance.

And the idea that we can download our minds into machines, as Ray kurzweil propose in his singularity is also unlike, as Miguel nicodelis says, the mind is not 100% computable! plus there is the researches by the heart math institute showing how the huge electromagnetic field by the heart is as much important as the brain's field to regulate the mind, and also the hypothesis by neurosurgeon rodolfo llinas in his book "the I of the vortex", suggesting that the Self is not inside the brain, but in the dialog between the brain, hypothalamus, and the heart in 40hz - weird
Y the same model Chinese daoists proposed 3 thousand years ago! And it's good to remember that the frequency of the heart can by simulated via fractals, but it's yet not fractal, since fractal, as all in math, is symmetric ( the commutative ring principle in the root of almost all in math) and the heart is, as all in nature, asymmetrical, the mathematician Steven strogatz remember us in a simple video in his site about this, the map/math is symatrical, and our biology as nature too are asymmetrical. Math to represent asymmetrical process (divergence), has to converge it first, so creating a deep disharmony in the process as explained by Italian math professor luiggi borzacchini.

We love to repeat the mantra that technology is neutral! But if math is basically symmetrical and nature is mostly asymmetrical and in the process of modeling it we create, as borzacchini shows, a deep disharmony, how can the technologies we develop from mathematical models by neutral? ;) we don't need to go that far, just checking the start of the biosphere the oceans, of our bodies, we can see its. to that neutral at all! We indeed live in a deep disharmony, inner and outer!
youtube.com/watch?v=QV9dhGv_tTs

That's all to say that we are living in a trance state, where we learned to disbelieve the potential of our own biology and to worship machines instead! Wild and crazy! A whole civilization in denial of it's own potential and worshiping it's own destruction and devolution in return! :D

Wayne, if you got the curiosity, I would suggest for you a book called "left in the dark". By tony wright, a new, with lots of scientific papers linked to sport the ideas in the book, but a new interpretation of evolution as a process of devolution. and I decided to practice the changes in my diet and living based on the book, and indeed, I was degenerating and destroying my brain before!! :D

Another tip is the good and old fashioned Chinese Daoist Psico-physiological alchemy! :D try to practice that with discipline and around 3 to 4 hours a day and you will start to bring out the quantum part of your biology into reality! There are two techniques that are serious, I've practice both, one is dr. Yan xin qigong, and the other is spring forest qigong... Practice level 1 for 3 month and then go for level 2 for more 3 month, two hours a day and start to get a taste of the potential of your own biology! :) Like this dude in a doc shot with a GH2! :D
v imeo.com/ondemand/mrelektro/62013952

Human beings are full of crazy potential, I decided to take charge of my body and mind complex and I don't. Regret! We are a lot more than what our culture and education limit us to believe we are... The problem is, it takes a lot of work to change the devolution we got into! :D

But we are indeed in deep problem caused by our denial of the states of things!
As this vid you posted shows!

August 18, 2014 at 1:31AM, Edited September 4, 11:56AM

1
Reply
Guto novo

I should use the iPad only to read comics and use procreate and sketchbook! My ugly English become uglier in it! :D

August 18, 2014 at 1:38AM, Edited September 4, 11:56AM

0
Reply
Guto novo

No one cares, Mr. Shiva.

August 18, 2014 at 11:44AM, Edited September 4, 11:56AM

0
Reply
Natt

This whole fear of technology taking over jobs is silly. The whole point of technology is to make products and services better and cheaper. The point of that is to sell more products and services to people. If nobody has jobs then they won't be buying anything so these robots will be useless. If it gets to the point where robots do most of the work then we will all be free to do non-essential functions like art without worrying about making a living. So the ultimate result of super-technology will be nearly complete freedom to do whatever we want without having to worry about how to survive.

Look at what the Internet has done for society. It has reduced some industries yes, but it has opened up many more ways to replace those sources of income. Look at how many people have started online companies with a few dollars and have the freedom that a traditional job reduces. Look at how many big gatekeepers have been replaced and industries opened up to average people. We need to learn to adapt with the technology.

August 29, 2014 at 1:50AM, Edited September 4, 11:56AM

0
Reply
Dandy

Hey Mr Michaelangelo Sculptor, we made a machine that cuts out that pain-in-the-ass boring stuff at the beginning and tells you what you're going to make.
Then you just finish it off. Good huh?

August 18, 2014 at 3:12AM, Edited September 4, 11:56AM

10
Reply
Jimi

-Hey Mr event videographer we made a machine that rough cuts the long form version while you concentrate on the creative highlights. Then you just finish it off.
-I'm no Michaelangelo and would appreciate the help, thanks.

It's all in the context; some people will find it useful while others would despise it. Hell, throw a 4k for that zoom use in a 1080p timeline and make the algorithm cut to the beat. It's ok if somebody doesn't want to use this Disney gimmick, 4k cameras or even a NLE, but they should get ready for a tough competition.

August 18, 2014 at 3:46AM, Edited September 4, 11:56AM

2
Reply
IvI

See app Vyclone. www.vyclone.com - been around for several years.

August 21, 2014 at 3:46PM, Edited September 4, 11:56AM

2
Reply

Well, hell...if I'm doing my work from the start a s a writer-director-creator of visually driven narratives, guess what? I've already got a good 20 percent of my specific choices (edits) down in the writing stage anyway.

And I didn't say -- shooting or production script.
I said -- WRITING.

I've got director-friends that just shoot coverage. Don't like storyboards; love winging it. Have convinced ( or conned ) the producer, money people, investor...that they need all this artistic freedom to find their vision ( always on someone else's coin). And of course, they want and need final cut...so they can sort through all theit shots, footage...a good 30 percent eding up on the floor -- money down the drain.

Never their own.

Got no problem experimenting with this technologyin the rough cut stage, like someone else mentioned. because at the end of it all...we can't be precious anyway witheverything shot; likewise with everything written.

August 21, 2014 at 5:29PM, Edited September 4, 11:56AM

8
Reply
MARK11