The most eye-opening demonstration at the NVIDIA GPU Technology Conference last week was from Adobe’s David Salesin (Sr. Principal Scientist) and Todor Georgiev (Sr Research Scientist), who showed their Plenoptic Lens along with software for processing the resulting images.
There was a gasp of amazement from the audience when we saw what the process is capable of. We saw an image refocused after the event.
For anyone who has ever taken an out of focus picture – which I guess is everyone – the immediate reaction is to want one NOW. Another appealing idea is to take an image that has several items of interest, but at different depths, and shift the focus from one to another.
So how does it work? It starts with the plenoptic lens, which lets you “capture multiple views of the scene from slightly different viewpoints,” said Salesin:
If you have a high resolution sensor then each one of those images can be fairly high resolution. The neat thing is that with software, with computation, you can put this together into one large high-resolution image.
In a sense you are capturing a whole 4D lightfield. You’ve got two dimensions of the spatial position of the light ray, and also two dimensions of the orientation of the light ray.
With that 4D image, you can then after the fact use computation to take the place of optics. With computation you have a lot more flexibility. You can change the vantage point, the viewpoint a little bit, and you can also change the focus.
To resolve that, to take these individual little pieces of an image and put them together into one large image from any arbitrary view with any arbitrary focus, it turns out that texture mapping hardware is exactly what you need to do that. Using GPU chips we’ve been able to get speedups over the CPU of about 500 times.
Note that the image ends up being constructed in software. It is not just a matter of overlaying the small images in a certain way.
There is a good reason NVIDIA showed this at its conference. Suddenly we all want little cameras with GPUs powerful enough to do this on the fly.
I guess this demo is likely to show up again at the Adobe MAX conference next month.
There’s another report on this with diagrams here.
The flaw I can see with this is we get barely enough light to make a good image using current lenses, unless you are using DSLR. Now if we suddenly need 19 lenses in that same amount of space, the quality is going to suck big time.
Just look at the size of the lens in the first pic, that is how big it needs to be to capture DSLR quality imagery with that technology. Now try squeezing all those into the space of a point-and-shoot lens and you will have cheap mobile phone camera quality.
Nobody is going to much care that you can alter the focus if the resulting image is too dark and grainy. I suppose the fact you have 19 different versions of that one shot will allow a certain amount of software fixing, but will it really make up the difference? The quality of the example shots above aren’t even that good, especially considering how small they are.
Its an interesting technology no doubt, but even hard core DSLR nuts are going to struggle to carry around a lens like the one above. Imagine taking wedding photos with that thing, ouch, you would get a slipped disc after the first job.
This isn’t all that new — I remember reading Salesin’s plenoptic lens paper back in summer of 2003.
Alex is absolutely right. It’s a very neat party trick, but not remotely practical until we get order-of-magnitude improvements in sensors.
That’s why they did the demo using a Contax medium-format camera with a digital back. Those things cost $4000 and have 40 megapixels of resolution.
Divide 40 megapixels by 17 and you still get 2 megapixels — decent enough to show on a screen. Divide a regular 10-megapixel DLSR image by 17 and your photo is barely webcam resolution.