October 19, 2012

Photoshop for 4K Video? It's Coming and It Will Blow Your Mind

Photographic retouching is nothing new, and it's as common as the digital technologies used to take the photos themselves. Retouching a video, however, is a bit trickier, and it usually requires more work to make a specific effect blend in with the rest of the image. Foton, Inc, a company from Japan, is working on technology that is achieving simply mind-blowing results with video that look as good as any Photoshopped image. Click through to check out these impressive clips.

This is what Foton had to say about the first video:

We developed special movie retouching techniques that won't damage a detail of material, and don't use "blur skill" for it that is why you feel more real. And also we work on 4K data without compressing 4K into HD. This model already had the natural beauty. However,we could improve it more to perform retouching,free transform, and color correction in the movie.

PetaPixel said this about the video:

It appears that the software is advanced enough to track with subjects’ faces as they moved around the frame, allowing for the edits to be consistent throughout the entire video without any blurring (look at how clear the skin is!). Although the new technique does away with frame-by-frame editing, Foton says it still spent 3-4 days creating the short proof-of-concept clip above.

This sort of manipulation is commonly done on movies and TV shows, but it usually takes a tremendous amount of time and work to do something like this on this sort of scale, and make it blend in perfectly. It's not often that you need to actually do this much to a video image, but if you could, and it could be done rather quickly and efficiently, why wouldn't you?

Here are a few more of their videos:

The other videos don't seem to be embeddable at the moment, but you can check out the rest on Foton's Vimeo page. You might be thinking that this isn't all impressive or that they do this all the time on movies, but most effects are not done at 4K, as the cost is just too great. It also seems that the technology they have can be made to work with lower resolutions, but I'm sure they have a custom tracker that works a lot better when there is more information and more detail.

The real test for this sort of thing will be to see it with a lot of motion. It's likely the tracker will have a difficult time if the scene changes drastically, but since we don't have any examples of that, we can't be sure. Can you do something like this in After Effects? Probably, but the amount of detail that is retained after retouching is impressive, and the retouches are blended seamlessly in a pretty impressive way.

What do you guys think? If the technology needs the extra pixels, does this make the case for 4K video? Have you done any retouching like this in your own work?

Links:

[via Gizmodo & PetaPixel & Fstoppers]

Your Comment

45 Comments

I removed a zit from an actress once...4 second clip...and used AE to track the edit. It was a pain in the butt, but a fairly simply procedure. What these guys are doing is LEAGUES ahead of that! While some people have complained that the models are moving slowly, not at lot of tracking needed by the software, etc..this is the beginning.

Think of Photoshop 10 years ago.

Pretty sure Adobe will find away to do this as well, or they'll just buy these guys ;)

October 19, 2012

0
Reply
sean

This just blew my mind. Absolutely amazing! I do vfx for a living. This is just jaw droppingly amazing stuff. I am assuming their software automates much of the tedium out of the process for these types of vfx shots.

October 19, 2012

-1
Reply
Jaw dropping

The digital make-up and retouching is astounding, though it does depress me a bit to see an expressive naturally aged face such as the woman's above being "youngified". The obsession with youth in media is a shallow one at best. But ranting aside, this will certainly be an awesome tool for storytelling across vast timespans, like Tron Legacy, allowing actors to either play their older or younger selves.

October 19, 2012

-1
Reply

The ability to have one actor play multiple ages is exactly what I thought of... seems like a less jarring solution than what they did in Tron.

October 19, 2012

-2
Reply

this is so, SO amazing!!

October 19, 2012

-2
Reply
Antonio Pantoja

This will do to the video industry what Photoshop did to the photo industry! Absolutely mind blowing!

October 19, 2012

0
Reply
TehRandax

I can see actresses in the future requiring digital retouching in their contracts ;)

October 19, 2012

-2
Reply
Marcus

Extremely impressive. It's on a different level entirely, and without question the next level of retouching. And how they can do it up to 4K with those results is mind blowing. Package it as a plug in, and it's destined to become the Photoshop of motion editing.

I have no doubt that Adobe's paying attention in a big way.

October 19, 2012

0
Reply
Manahi

Wonder what the render time is on something like this between real time puppet warp, 3d motion tracked dodging and burning, and 3d motion tracked healing brush. Would be useful for a movie where an older or younger self would want to be portrayed...

October 19, 2012

1
Reply
Antony Alvarez

Now they can stop blurring out only portions of Diane Sawyer face on ABC World News and just fix the damn thing...I felt pity for those guys who had that touch up job.

October 19, 2012

-1
Reply

It's a lot harder than this in AE. Really excited for this, not just for retouching but for make up as well.

October 19, 2012

-1
Reply

This is disgusting.

Retouching software should be binned if its trying to change what is natural to the subject. Stop making people feel inadequate and deal with the subjects.

If you have to correct or take out "imperfections" then don't use that person, use someone else.

The sheer subtext involved in programmes like this is f***ing ridiculous. This isn't mind blowing, this is horrifying.

October 19, 2012

-2
Reply
David

Slow down there. Yes, retouching can be done unethically. But the very use of it is no more unethical than using makeup. Frankly, most talent appreciates being seen in the best light, so to speak. This is especially true with music videos or other productions that highlight the talent.

There are a lot of good articles and debates out there about retouching in the photo industry that apply here as well. You should take a look at some of those.

October 19, 2012

0
Reply
David S.

I disagree that digital retouching and makeup are on a par ethically. Energy drinks are allowed in professional cycling but blood transfusions are not. Are there any mainstream magazines where the cover girl's face looks even passably real?

October 19, 2012

0
Reply
Pippy

Again, like the guy I replied to, you are jumping to the extreme. It's not inherently wrong. When it is done excessively, yeah, I agree that it becomes disgusting and unethical. But slight touch-ups and the stuff we see in magazine ads are two completely different things.

Just because the misuse of a tool is unethical does not make the tool itself unethical. Retouching has always been done in visual media, even since before photography. I'm not sure what your experience has been, but in my experience the talent usually wants slight blemishes taken out (if possible). Like I said, this is especially true if highlighting the person on camera is the point of the video (the artist in music videos, brides in wedding videos, etc.) If asked, they almost always want pimples cleaned up, etc. Taking out a pimple isn't the same as making someone impossible thin or making someone look ten years younger.

I really didn't understand your comparison to energy drinks and blood transfusions.

October 19, 2012

1
Reply

I agree with David. There is a line but no matter who you are, you don't want to be looking at white heads in 4K. You just don't. Personally I think temporary stuff is fine to remove (ie. pimples, excessive darkness under eyes etc) but permanent stuff should be left as is (wrinkles, age spots etc).

There is something wrong with changing the person but there is nothing wrong with making them look like they would on their best day. At that point it's no different than make up.

October 19, 2012

-1
Reply

Agreed. We all perform digital modification of some form or another. This is just another form. Sure it can be abused, but I think it's a valid tool that can and should be used. I'm sure the same technology can add imperfections, which is often needed in filmmaking.

October 20, 2012

1
Reply

whooa

October 19, 2012

-1
Reply

I don't know what everyone is so excited about. This is so old news: http://www.youtube.com/watch?v=PvSUfThAHas

October 19, 2012

1
Reply

"The new world of Cinema"

When by default we start getting things right in post instead of merely "correcting".

Not that I'm complaining, quite fascinating.

October 19, 2012

-1
Reply
Raphael Wood

This could definitely be useful. So far people seem to be thinking of using it just to make people look younger, more attractive...or older. I can see this being used to easily distort people into supernatural demons, one eyed mutants, or big eared inbreds with one side of their nose drooping down.

As far as digital representations of people go, sure, it'll get all William Gibsony to the point where movie stars are purely digital and never age...and these small steps are on the way there...but in the meantime, we should have fun with it.

October 19, 2012

1
Reply
Brett Roberts

Cool! Now we can make people look faker and faker!

October 19, 2012

-1
Reply
john jeffreys

Love how the anti-technology trolls always show up in these threads.

Visual Autotune is 100% awesome. Other than special effects and digital make-up, these techniques can SAVE the image from camera and compression artifacts. So-called beautification software is really just an advanced psychovisual optimization for an encoded image. Just as a wagon wheel turns backwards in the Westerns, you're stuck with any facial expression, body movement or lighting effect the camera's native frame rate can capture at the moment. Just ask any animator how important keyframes are for storytelling.

October 19, 2012

1
Reply
Mark

you know you never had to worry so much about skin imperfections so much until HD came along, film had it's own way of dealing with that! ;)

October 19, 2012

2
Reply
Ross

Actually we do this since 10 years for makeup products and you still buy them. What's new here ? Hmm like ever just internet fame. It's not informative, it's old stuff reviewed with marketing 4K on it. Come on guys.

October 19, 2012

0
Reply
Super

Wow unreal..

October 19, 2012

0
Reply

Imperfections are a part of character... performance... setting... the "natural costume". I don't think this technology solves any issues. Unless you'd like your work to look like Surrogates.

October 19, 2012

0
Reply
Travis Jones

Just so you know - two major brands already use this software in their ads.

I feel sorry for any producer whose 'older talent' see that vid above.

This is only the beginning. The race is on between 3G and postfix for blemish free photo-realistic actors.
That is a HUGE 'want' in advertising.
CGI will win in the end, but still years away. At the end of the day, ALL film-making is animation of a kind.

Also, a lot of total rubbish being written above. My favourite thing has been people pointing to The Master as unblemished 'old-style' filmmaking. If I legally could I'd post the list of effect shots in that film.

October 19, 2012

-1
Reply
marklondon

I enjoyed The Master, visually at least...and I can't believe anyone would say that? What are they smoking? In one scene alone, it was hilariously obvious that someone had lazily used the same female body double's body parts digitally grafted onto two different characters in the same shot!

October 20, 2012

0
Reply
trackofalljades

wow

October 20, 2012

0
Reply

Yeah had seen this. Looks great, but the big test will be it's ease/speed of use and as mentioned by Joe, how it will track. Not a lot of movement in the clips, and if it doesn't track well, it will be of limited use.

October 20, 2012

1
Reply
Lliam

this is truly amazing

October 20, 2012

0
Reply

It is revolutionizing, time saving and enhance storytelling as it unfolds unique images. The imagination will push the technology. Artists will not complain . It reminds me Art Department and Elastic Reality users blow away by advanced motion FX software that now comes as plugins for editing suites.

October 20, 2012

0
Reply
Agni Ortiz

Oh, on a lighter note, just imagine how much it could "enhanced" the porn film industry! ; )

October 20, 2012

0
Reply
Agni Ortiz

ok... everybody seems to think that is some new software that these people have invented. This is almost surely not the case and nowhere have these people claimed to have written any. They talk about new "techniques" not software. In one of the videos you can even see them using photoshop. After photoshopping, all of those retouches can be applied to the moving footage using something like mocha pro with object removal plates (which even take into account some lighting changes while applying the photoshopped clean plates to various areas of the face. Sure, this is time consuming, but these guys never said that what the did was quick and easy. The more impressive video is the eyelashes one, which seems to involve some 3d work or something. In any case, I'll bet anyone dollars to donuts that there's no new software out there.

October 20, 2012

0
Reply
siltal

Exactly! I work in the VFX industry and I can tell you this is not new. This is most surely just a lot of Planer tracking in Mocha or mostly likely Nuke. In fact, here is a tutorial on how to do it in Mocha:

http://www.cgmotionbox.com/2011/10/retouching-with-mocha-pro-remove-modu...

And the fact that it's 4K doesn't have as big an impact as you are making it out to be. And you are wrong about the film industry not doing this in 4K. VFX is almost always done in 4K. You just need a really fast machine, a s**ton of RAM and patients. But keep in mind that because of planer tracking, this is no longer a frame by frame painting process. You basically create a bunch of retouched "patches" that then get tracked in 3D to the persons face. Warping is easy. We all know about that. So the process here is really not that difficult. Notice to that the subjects aren't moving around a lot. When you do this kind of stuff for a living you basically look at shots like this: How many times she turns around or drastically changes perspective is how many times you are going to have to repaint and retouch the subject. This is basically just like if you had a retouch job and had to work on about 10 or so RAW images. As new things come into view, the old things go out of view.

Probably the most difficult part was the eyelash enhancement. That looked like actual 3D geometry that had to be tracked to the movement of the eye lids. Not really that big of a deal but it requires you to jump into some 3D software and create the geometry, rig it so it;s deformable, track the original footage in 3D, animate the lashes, and render it and then comp it back in.

By the way, if you are interested in hiring me to do this kind of work just let me know. It's kind of expensive though. ;-) galenbeals-at-gmail-dot-com

October 21, 2012

-2
Reply
Galen

How to buy/ download the software as advertised in this article?

October 21, 2012

0
Reply

October 21, 2012

-1
Reply
Galen

Not impressed because all of this (tracking, painting, restoration, blemish removal, repair brush, patch cloning) can ALREADY be done with Silhouette software http://www.silhouettefx.com for only $995.

October 23, 2012

0
Reply
Razor

Muy bueno, gracias por la información.

October 25, 2012

0
Reply
oscar

Yada, yada yada. May be nice for the fashion industry, But also it makes me sad. Let's just move a head and make 3D models instead.

October 25, 2012

-1
Reply
Emil

Really useful for CU shot and Fashion shot. Love it!

October 25, 2012

-2
Reply

Of course you can do this in After Effects. PS is probably borrowing AE code to do this. The AE interface is more intuitive though.

October 26, 2012

2
Reply
John

This is going to be welcome edition to a lot of studios, most of which already have multiple seats of PS. The thing is the types of cloning/corrective tools PS has and the workflow for manipulating that kind of thing is pretty straight forward for stills. Obviously I would have to see what the trackers interface is. The large boon on this is the fact that it will ease the transition of a lot of seasoned retouchers into cine grading which automatically means a lot of amateurs will get waylayed on their road to ripping people/companies off. For me there are a couple of specific things I would love to do in Photoshop using PS methods before hitting DaVinci Resolve or a Baselight.

November 12, 2012

-1
Reply
Butch

The huge benefit PS has is that some things, once they're tracked, can be put on autopilot simply because of the "content aware" options that Adobe PS owns the stable of.

November 12, 2012

0
Reply
Butch