synthetic aperture confocal imaging
I first learned of Synthetic Aperture Photography at SIGGRAPH 2004 at Levoy’s paper session “Synthetic Aperture Confocal Imaging”. The idea is to integrate an array of discrete apertures into a larger, simulated aperture. The paper was actually about macroscopic confocal imaging using a bank of projectors to simulate an illumination system with a very large aperture. Since the illumination gets the wide aperture, off-focal-plane objects are dim and don’t reduce the contrast of the final image as much as for imaging performed with a simulated aperture on the camera. I didn’t have the luxury of illuminating the neighborhood with a bank of projectors, so you can see that my synthetic aperture shots have reduced contrast where the background is partially occluded.
To get this demo done in a day, a made a few simplifying assumptions. The camera model is linear, the camera translates at a constant rate along a line, and the synthetic focus is parallel to the imaging plane. Assuming all that, there is almost nothing left to do: the result is a sum of shifted images, scaled by the number of contributors. The shift is a function of the distance to the focal plane. (Google needs to make some simplifying assumptions with their video processing… it took longer for a few frames of video to be prepared for posting than for me to do the entire project.)
I transferred the video from my cell phone to Julie’s MacBook over Bluetooth and coded up the shift-and-sum in Mathematica. Autocorrelation gave some hints as to good focal planes. I made a variation of the shift-and-sum to allow the focal plane to be “programmed”. Here is N. Broad Street in Milford, CT, imaged at street level. The street is actually lined with cars that occlude parts of the source video. The image will be sharper when I add a function to tilt the focal plane.
Hwang et al. proposed a variation on synthetic aperture photography using a microlens array to produce multiple apertures with a single, stationary camera. The SIGGRAPH paper also considered autostereoscopic displays for generating the synthetic aperture light field, which amounts to the same technique.