Joshua Napoli

I first learned of Synthetic Aperture Photography at SIGGRAPH 2004 at Levoy’s paper session “Synthetic Aperture Confocal Imaging”. The idea is to integrate an array of discrete apertures into a larger, simulated aperture. The paper was actually about macroscopic confocal imaging using a bank of projectors to simulate an illumination system with a very large aperture. Since the illumination gets the wide aperture, off-focal-plane objects are dim and don’t reduce the contrast of the final image as much as for imaging performed with a simulated aperture on the camera. I didn’t have the luxury of illuminating the neighborhood with a bank of projectors, so you can see that my synthetic aperture shots have reduced contrast where the background is partially occluded.

To get this demo done in a day, a made a few simplifying assumptions. The camera model is linear, the camera translates at a constant rate along a line, and the synthetic focus is parallel to the imaging plane. Assuming all that, there is almost nothing left to do: the result is a sum of shifted images, scaled by the number of contributors. The shift is a function of the distance to the focal plane. (Google needs to make some simplifying assumptions with their video processing… it took longer for a few frames of video to be prepared for posting than for me to do the entire project.)

I transferred the video from my cell phone to Julie’s MacBook over Bluetooth and coded up the shift-and-sum in Mathematica. Autocorrelation gave some hints as to good focal planes. I made a variation of the shift-and-sum to allow the focal plane to be “programmed”. Here is N. Broad Street in Milford, CT, imaged at street level. The street is actually lined with cars that occlude parts of the source video. The image will be sharper when I add a function to tilt the focal plane.

Hwang et al. proposed a variation on synthetic aperture photography using a microlens array to produce multiple apertures with a single, stationary camera. The SIGGRAPH paper also considered autostereoscopic displays for generating the synthetic aperture light field, which amounts to the same technique.

On Sunday, I made a synthetic aperture demo to remove foreground objects from a street side video stream. I had Julie drive at a constant speed down North Broad Street in Milford, CT as I captured a video with my cell phone. The image quality of the refocused image could be improved by setting the focal plane more carefully. I made a video to show the comparison of the captured against refocused, but Blogger.com is choking on it right now. I’ll show some individual frames instead. Here are a couple of examples of foreground occluders being removed:

The strong, static system of C++ seems to provide almost everything that the weak and dynamic type systems of Perl and Python do, but also adds an extra, automatic error checking step. It’s a bit perplexing as to why C++ seems to fail to provide overall advantages in development efficiency or extensibility. The failure can be attributed to extra development cost to maintain parallel type calculations throughout the software and also to the failure of type checking to provide a sufficiently broad proof of the program’s self-consistency.

In order to compile, the types must be consistent. Unfortunately, as C++ currently stands, the majority of this consistency checking falls in the hobgoblin category. The software developer must redundantly calculate the result type of almost every expression. This is a useless exercise that adds no information to the program (the compiler independently calculates the types anyway). In some cases the return type of a function is considered an implementation detail that should not be exposed to the application design (for example, boost::bind). Here it is impossible (or at least detrimental) to reflect the return type. This makes a noticeable reduction in the expressive power of such libraries. C++ 0x may help with auto and valtype keywords. Since the “initializer expression” may not be readily available for class member variables, these keywords will add some asymmetry between state variables and scoped variables. One should probably think of scoped variables as descriptive names for subexpressions of a function rather than simple values.

The compiler’s verification that the types in the software are consistent should give an indication that the software is to some degree correct. Evidently, that is not the case as C++ software needs essentially the same testing regimen that weaker languages do. Two aspects of the language contribute to the disappointment in C++ software robustness. First, the standard library offers no assistance. For example, there is no “type qualifier” mechanism for doing things like dimensional analysis or taint analysis. Without library support, it is difficult to make use of the type system for verifying any non-trivial degree of consistency within the program. The second factor is the asymmetry between types and values. Although most test cases could be evaluated statically, there is no way to induce the compiler to make the evaluation and assert an error on failed test cases. Without this integration, software development organizations must invest in a substantial infrastructure to organize, compile, and execute specialized testing projects.

At a higher level, the case of program validation with C++ could be vastly improved if it had some facility for introspection and pattern replacement on its abstract syntax tree. It’s hard to understand how any modern language fails to provide this bit of introspection. One would like to inject new rules into old software: for example, to deal with instances of ‘new’ where the result is not immediately assigned to a smart pointer (replace the raw pointers to shared pointers or issue errors). Presumably you will learn more interesting and powerful practices in the future that would be beneficial to apply to the code that is being written today.

Integration of the abstract parse tree with the code editor opens up some interesting possibilities. For example, pattern replacement operations in the previous paragraph could be represented within the editor. At SciFoo Camp, Charles Simonyi and Ted Kaehler both described work on development environments that would have the ability to render different views of some underlying syntax tree. UML has similar ideas, but in UML the focus is on integrating different levels of abstract design rather than elucidating the implicit relationships within an actual piece of software.

Skippers Dock
60 Water Street
Stonington, CT

The restaurant has outdoor seating on a dock overlooking the picturesque bay. The food did not live up to the standard set by the view, and we expected more based on the price. Our appetizer, drinks and dessert cost $80, but none of the dishes showed any evidence of creativity and most could have benefited from more skillful preparation. For waterfront dining, S&P Oyster in Mystic is a better choice. It had better food and lower prices.

Up River Café
37 Main Street
Westerly, RI

My wife and I are fans of Up River Café. The restaurant is built over part of the river in downtown Westerly, Rhode Island. There is a patio that overlooks the river, for al fresco dining. We always enjoy the patio, although there are a few mosquitoes after sunset. The two of us spend about $130 for a dinner with a glass of wine and a desert. Dinner has always been good and the river ambiance makes for a nice evening.

Most recently, I had a swordfish dish with polenta and a red pepper coulis, topped with fennel. Other restraints (such as Legal Seafoods) will serve instant “polenta” partially disguised, but at Up River the polenta was made properly, preserving the milled corn taste to contrast with and slightly sweeten the fish. The sauce and vegetable topping were perfect. My wife had striped bass with artichoke hearts, purple fingerling potatoes and red pepper coulis. The swordfish dinner was more interesting and flavorful than the striped bass.

My wife liked the prickly pear sangria, although the portion was quite small, since the wine glass was mostly full of ice.

S&P Oyster
Mystic, CT
206-911-1343

In the summertime, I sometimes annoy my wife by going to extraordinary lengths to find waterfront al fresco dining. Even in Mystic, CT, a place intimately tied to the sea, there are surprisingly few options for a nice dinner on the water. This scarcity is an advantage to the owners of S&P Oyster, a nice restaurant situated at the end of the famous drawbridge. On Labor Day weekend, the place was packed, of course. My wife and I waited an hour and a half for a seat on the outdoor patio. The view of the river was nice. The operation of the bridge and the constant stream of humanity crossing it provided some entertainment. Dinner was reasonably priced, although unexceptional. For $70, we each had an appetizer and dinner, and we split a desert. The salmon was tasteless. You may want to skip the scallop and proscuitto dish on the “lighter fare” menu. The proscuitto in the dish is too salty.