Improving the spatial, angular, and temporal resolution in light field imaging
Date
2013
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
A light field captures a dense set of rays as scene descriptions in place of geometry. Recent advances on computational imaging have enabled novel and efficient light field acquisition devices. For example, the new Lytro and Raytrix cameras are able to capture light fields in a single shot. However, the effective spatial resolution is reduced by the number of microlenses. For example, in Lytro, the resulting image at a desired focal plane has a resolution of 1080x1080, which is too low for photographic uses or in computer vision tasks. The acquired light field also has a low angular resolution, usually less than 10x10 for each spatial sample. This results in aliasing artifacts when synthesizing dynamic Depth-of-Field (DoF). Finally, the data size of each captured light field can easily reach 20 MB, prohibiting live streaming and processing at interactive frame rates. ☐ In this dissertation, I develop a new class of image processing algorithms and camera designs that can significantly improve the spatial, angular, and temporal resolution in light field imaging. ☐ Spatial Resolution: We develop a simple but effective technique by maneuvering the demosaicing process. We first show that traditional solutions that demosaic each individual microlense image and then blend them for DoF synthesis is suboptimal. We instead propose to demosaic the synthesized view at the rendering stage by first mapping the rays onto the refocusing plane and then conduct resampling. Our approach can significantly improve the spatial resolution while reducing the aliasing artifacts. ☐ Angular Resolution: We introduce a light field triangulation scheme to improve the angular resolution. Our triangulation technique aims to fill in the ray space with continuous and non-overlapping simplices anchored at sampled points (rays). Such a triangulation provides a piecewise-linear interpolant useful for angular super-resolution. We develop a novel triangulation algorithm that uses the depths and structures of 3D lines as constraints for producing high quality triangulations. For robust depth estimation, we further present two light field stereo matching algorithms that greatly outperform the state-of-the-art. ☐ Spatial-Angular Resolution: We further present a unified framework to simultaneously enhance the spatial and angular resolutions by stitching multiple light fields. We first estimate the warping function between two light fields and then stitch them by finding an optimal cut through the overlapping region. We further accelerate the graph-cut algorithm via a coarse-to-fine scheme. We demonstrate various stitching applications to improve the field-of-view as well as translational and rotational parallaxes of the light fields. ☐ Temporal Resolution: Finally, we construct a hybrid-resolution stereo camera system for acquiring and rendering dynamic light fields. Our system couples a high-res/low-res camera pair to replace the bulky camera array system. From the input stereo pair, we recover a low-resolution disparity map and upsample it via fast cross bilateral filters. We subsequently use the recovered high-resolution disparity map and its corresponding video frame to synthesize a light field using GPU-based disparity warping. Our system can produce racking and tracking focus effects at a resolution of 640x480 at 15 fps.
Description
Keywords
Triangulation algorithm, Resolution, Light Field Imaging