Towards Motion-Aware Light Field Video for Dynamic Scenes
ICCV 2013. Sydney, Australia.
Simulated scene with moving butterfly and beetle and static grass. Notice the low spatial resolution refocusing of Lytro, as well as artifacts on moving objects for Liang et. al.A standard video camera can either focus in front or back, but cannot achieve digital refocusing. Our approach allows capturing high spatial resolution LF for dynamic scenes.
Current Light Field (LF) cameras offer fixed resolution in space, time and angle which is decided a-priori and is independent of the scene. These cameras either trade-off spatial resolution to capture single-shot LF or tradeoff temporal resolution by assuming a static scene to capture high spatial resolution LF. Thus, capturing high spatial resolution LF video for dynamic scenes remains an open and challenging problem. We present the concept, design and implementation of a LF video camera that allows capturing high resolution LF video. The spatial, angular and temporal resolution are not fixed a-priori and we exploit the scene-specific redundancy in space, time and angle. Our reconstruction is motion-aware and offers a continuum of resolution trade-off with increasing motion in the scene. The key idea is (a) to design efficient multiplexing matrices that allow resolution tradeoffs, (b) use dictionary learning and sparse representations for robust reconstruction, and (c) perform local motion-aware adaptive reconstruction. We perform extensive analysis and characterize the performance of our motion-aware reconstruction algorithm. We show realistic simulations using a graphics simulator as well as real results using a LCoS based programmable camera. We demonstrate novel results such as high resolution digital refocusing for dynamic moving objects.
(a) Optical ray diagram of our setup. (b) Actual working prototype. (c) All 25 coded aperture masks used in our approach.
Some Simulated Light Fields
S. Tambe, A. Veeraraghavan, Amit Agrawal. Towards Motion-Aware Light Field Video for Dynamic Scenes. IEEE International Conference on Computer Vision, ICCV 2013