December 27, 2012

Toshiba 'Out-Lytros' Lytro, Designing a Camera Sensor Capable of Video Refocusing

Lytro is a company implementing a novel and fascinating idea -- obvious enough for anyone to appreciate it's wow-factor, but original enough that visual creatives can be impressed with its technology -- you can set your focal distance after taking your shots. Now, you can also even shift your perspective a bit, another near-magical innovation for digital photography. That said, right now Lytro cameras have several major limitations: they are stand-alone cameras, which may be inconvenient for shooters used to novelty photography on a tiny multi-use device -- plus, there isn't great direct mobile integration with social media. Interestingly, Toshiba has just announced that it's developing its own lightfield-type sensor, specifically for tablet and smartphone applications -- and, it's expected to allow focus shift for your mobile video as well.

There's plenty to be said for a 'Lytro-type-thing's' desirability to be built into a smartphone, as opposed to a separate non-tweeting non-texting non-internet-surfing device unto itself -- that may pretty much sum it all up there, actually. Beyond that, the other benefits Lytro provides for its own lightfields -- such as the ability to unlock new qualities of your images via later software updates, are things conceivably Toshiba could follow suit on in collaborating with mobile manufacturers. With all the added benefits of mobile integration -- think of the Instagram-type of appeal you'd think a lightfield shot would have -- Lytro itself will probably be hurt by the potential success of Toshiba's push here. With thanks to Engadget for the find, here's some of the report from the Asahi Shimbun in Japan (image courtesy Takashi Kamiguri):

The cube-shaped module is about 1 centimeter per side and contains a dense array of 500,000 lenses, each 0.03 millimeter in diameter, in front of an image sensor measuring 5 mm by 7 mm. The same mechanism works similar as the way the compound eye structure functions in insects.

Each lens captures a slightly different image from one another, and the camera produces a large, complete picture by using original software to combine the 500,000 tiny images.

The principle of which should sound similar to anyone familiar with Lytro, except for the physical dimensions.

http://www.youtube.com/watch?v=L8Nc316hJnQ

Lightfield lens array and sensor integration into mainstream mobile devices worldwide could be the next step in the evolution of this type of photography alone -- but with the addition of video to the picture (literally), as the Asahi Shimbun also states, Toshiba's surely looking to pull ahead.

The module-equipped camera can also be used to take videos, and allows the users to retain the image of a figure in the foreground while replacing the background.

U.S. camera manufacturer Lytro Inc. is marketing cameras that have similar focusing functions, but the devices are about the size of a human palm.

Unless Lytro can mobilize a new device -- or even tap their already existing cameras for video, something that doesn't seem likely -- very quickly, they may be left behind in the burgeoning lightfield photography tech-genre they themselves pioneered. Or, perhaps, they come out of left field with more options of even greater novelty value than anything Toshiba has on the drawing board -- we'll have to wait and see, as Toshiba's designs won't see massively see the light of day until late next year at the earliest.

The most interesting aspect of all this, though, is the ability to refocus in motion pictures -- and while I can't even really imagine what this will be like (are we talking in real time, and how many times can I set or reset the focus of a moving shot?), it will surely be of interest to many of us, especially since RED seems keenly after a similar option.

In what ways do you think refocusing video abilities will affect what we do, if at all, even if just on mobile devices?

Link: Toshiba putting focus on taking misfocusing out of photos -- Asahi Shimbun

[via Engadget]

Your Comment

9 Comments

Never understood Lytro as a company. With so much competition from regular cameras (nikon, canon, etc) to smart phones capable of taking decent photos, where did they expect to gain customers come from?

Not sure if there technology is patentable, but the only way I would assume for them to thrive would be to license it out to the other manufacturers.

December 27, 2012

0
Reply
cappy

strike out 'come', *their

December 27, 2012

-1
Reply
cappy

According to Wikipedia - a Gabriel Lippmann first suggested the basis of a lightfield camera in 1908 and then in 1992 Adelson and Wang proposed the design of a microlens array - so yea, Lytro can't really patent their camera, even though they are the first to create a consumer model.
The market base is kinda huge if you think about it - though digital photography made the art accessible to practically everyone, reducing even the necessity to focus would make it accessible to absolutely everyone. That said, Toshiba will probably come out on top in this game primarily due to the ease of access to the technology their mobile sensor would provide.
Most people at this point use their phones for pretty much everything, so if that was equipped with a lightfield camera sensor at no real added cost, then that would change the game entirely in their favour - at least in terms of market penetration. Just imagine the iPhone 6 being released with this camera - done deal, game over :)
I'm still quite interested to see what RED manage to cook up - I've had the suspicion for around a year and a bit now that we will see the whole fixed-lens, consumer-price Scarlet equipped with maybe a 3k plenoptic sensor some time next year :)

December 27, 2012

0
Reply
Andreas Kopriva

That ad is hilarious. Lytro will get you laid apparently.

December 27, 2012

1
Reply
Stu Mannion

To me, the end goal is not just about refocusing or being able to shoot 3D with one lens.

The end goal is to pair this with a display technology that gives us not just stereo but true 3D. The good old "hologram" if you will.

Think about the reverse of this: if you showed a capture from this 500-lens thing on a matching display with 500 tiny lenses, you would see a different image as you walked around it. Kinda lenticular-like. Granted it'd be low-res. But if you do it at high enough resolution you are in business.

Another example: if you had a screen made of tiny pico-projectors and each pico-projector shows the appropriate part of the lightfield to you, we may indeed approach something akin to a holographic effect.

I'm not nuts - Paul Debevec (you know, the guy who pioneered HDR, which was once a similarly magical technology) even has a paper on it:
http://gl.ict.usc.edu/Research/LFD/

I for one am really interested in where this is all going.

At the very least, capturing the full lightfield of a shot will have very interesting implications for VFX. Once we get the tools we need to deal with the data, that is!

December 27, 2012

-1
Reply

Great comment! The implications of this technology are definitely really exciting -- both for mainstream motion pictures (like the VFX you hinted at) as well as for the other side of things, more experimental work. I would also be really interested to see what kind of bizarre effects could be achieved by hacking something like a Lytro, perhaps messing with its de-lightfield-ing process to 'break' images on purpose, but still allow interactivity and shifting in some ways -- now I probably sound more like the crazy one, but it's all very intriguing -- thanks to everyone else for the comments as well!

December 27, 2012

0
Reply
avatar
Dave Kendricken
Writer
Freelancer

I thnik the idea of multi lens projection/hologram is Brilliant!

That is '3D' I think everyone will love.

It could also give video content a type of spacial interactivity it have never had before.

It will likely take alot of work & alot of time to get it into a consumer package, but i really hope something like that is in our future.

December 28, 2012

0
Reply
Steven Falconer

great concept

December 30, 2012

0
Reply
Natt

I work for a non-profit organization where I do a lot of editing videos together for short-term missions trips. We always send a camera along with the team, but we don't have the budget to send along a cinematographer on each trip, so I often am working with footage that's out of focus, too shaky, etc. If I had the ability to go in and change the focus point of a shot in post, that would be absolutely amazing. The only limitation I'm thinking would be file sizes. I'm not sure how much disk space a Lytro image takes up, but saving 24 or 30 of them a second has gotta take up a lot of space.

Even just for my own work, I'd love to have the option of pulling focus in post. I'm really low budget, so I've done a few jobs where I wanted that shallow depth-of-field look and to achieve it I've manually focused a handheld T2i with a 50mm 1.4 lens set at 1.8, so as you can imagine not all of the shots turned out perfect. It takes a lot of practice, and even with all that practice you still miss shots.

January 7, 2013

1
Reply