Generating Images from Light Fields Utilizing Virtual Viewpoints

Abstract

Systems and methods for the synthesis of light field images from virtual viewpoints in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system includes a processor and a memory configured to store captured light field image data and an image manipulation application, wherein the captured light field image data includes image data, pixel position data, and a depth map, and wherein the image manipulation application configures the processor to obtain captured light field image data, determine a virtual viewpoint for the captured light field image data, where the virtual viewpoint includes a virtual location and virtual depth information, compute a virtual depth map based on the captured light field image data and the virtual viewpoint, and generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map.

What is claimed is: 
     
         1 . A system configured to synthesize images using captured light field image data, comprising:
 a processor; and   a memory connected to the processor and configured to store captured light field image data and an image manipulation application;   wherein the captured light field image data comprises image data, pixel position data, and a depth map;   wherein the depth map comprises depth information for one or more pixels in the image data; and   wherein the image manipulation application configures the processor to:
 obtain captured light field image data; 
 determine a virtual viewpoint for the captured light field image data based on the pixel position data and the depth map for the captured light field image data, where the virtual viewpoint comprises a virtual location and virtual depth information; 
 compute a virtual depth map based on the captured light field image data and the virtual viewpoint; and 
 generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map, where the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map. 
   
     
     
         2 . The system of  claim 1 , wherein the virtual viewpoint corresponds to a focal plane in an array camera utilized to create the captured light field image data. 
     
     
         3 . The system of  claim 1 , wherein:
 the captured light field image data further comprises a reference viewpoint within the captured light field image data; and   the virtual viewpoint is a separate viewpoint within the captured light field image data from the reference viewpoint.   
     
     
         4 . The system of  claim 3 , wherein:
 the captured light field image data was captured by an array camera having an imager array comprising a plurality of imagers; and   the reference viewpoint corresponds to the viewpoint of a first imager within the imager array in the array camera.   
     
     
         5 . The system of  claim 4 , wherein the virtual viewpoint corresponds to the viewpoint of a second imager within the imager array, where the second imager is separate from the first imager. 
     
     
         6 . The system of  claim 4 , wherein the virtual viewpoint is a viewpoint that does not correspond to the viewpoint of any of the imagers within the imager array. 
     
     
         7 . The system of  claim 4 , wherein the virtual viewpoint is selected from a position selected from the group consisting of in front of the imager array and behind the imager array. 
     
     
         8 . The system of  claim 1 , wherein the image manipulation application further configures the processor to generate an image from the perspective of the virtual viewpoint by projecting pixels from the captured light field image data based on the pixel position data and the depth map, where the projected pixels are described in the image data and the depth map. 
     
     
         9 . The system of  claim 8 , wherein:
 the captured light field image data further comprises occluded pixel information describing pixels not visible from a reference viewpoint of the captured light field image data; and   the projected pixels include at least one occluded pixel in the occluded pixel information that is visible from the perspective of the virtual viewpoint.   
     
     
         10 . The system of  claim 8 , wherein:
 at least one projected pixel in the generated image is not described in the image data, the pixel position data, and the depth map; and   the image manipulation application further configures the processor to generate the at least one projected pixel by resampling the image data, the pixel position data, and the depth map.   
     
     
         11 . The system of  claim 8 , wherein a pinhole camera model is utilized to project pixels within the generated image based on light rays projecting from the virtual viewpoint, where each projected pixels is associated with at least one of the projected light rays. 
     
     
         12 . The system of  claim 11 , wherein projected pixel depth information is determined for at least one pixel in the generated image based on the depth map, the virtual viewpoint, and the light rays associated with the projected pixel. 
     
     
         13 . The system of  claim 12 , wherein the depth information for a projected pixel is based on minimizing the variance for the projected pixel across the image data within the captured light field image data. 
     
     
         14 . The system of  claim 11 , wherein the image manipulation application further configures the processor to combine projected pixels having the same location within the generated image. 
     
     
         15 . The system of  claim 14 , wherein the pixels are combined based on the weighted average of the pixels, where the weighted average of the pixels is the inverse of the distance from the imager from which the projected pixel originated to the virtual viewpoint. 
     
     
         16 . The system of  claim 1 , further comprising an input device configured to obtain input data indicative of a position within the captured light field image data. 
     
     
         17 . The system of  claim 16 , wherein the input device is a touchscreen interface. 
     
     
         18 . The system of  claim 16 , wherein the input device is a sensor configured to obtain spatial location information. 
     
     
         19 . The system of  claim 16 , wherein the input device is a camera configured to obtain input data selected from the group consisting of head tracking data and gaze tracking data. 
     
     
         20 . The system of  claim 16 , wherein the virtual viewpoint is selected based on the input data. 
     
     
         21 . The system of  claim 20 , wherein the image manipulation application further configures the processor to:
 obtain a first input data indicative of a first position within the captured light field image data;   determine a first virtual viewpoint based on the first input data;   generate a first image from the perspective of the first virtual viewpoint;   obtain a second input indicative of a second position within the captured light field image data, where the second position is separate from the first position;   determine a second virtual viewpoint based on the second input data;   generate at least one intermediate virtual viewpoint by interpolating between the first virtual viewpoint and the second virtual viewpoint;   generate at least one intermediate image based on the generated at least one intermediate virtual viewpoints, where each intermediate image is from the perspective of an intermediate virtual viewpoint; and   generate a second image from the perspective of the second virtual viewpoint.   
     
     
         22 . The system of  claim 1 , wherein the image is generated utilizing a super-resolution process. 
     
     
         23 . A process for generating an image from a virtual viewpoint, comprising:
 obtaining captured light field image data using an image manipulation device, where the captured light field image data comprises image data, pixel position data, and a depth map and where the depth map comprises depth information for one or more pixels in the image data;   determining a virtual viewpoint for the captured light field image data based on the pixel position data and the depth map for the captured light field image data using the image manipulation device, where the virtual viewpoint comprises a virtual location and virtual depth information;   computing a virtual depth map based on the captured light field image data and the virtual viewpoint using the image manipulation device; and   generating an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map using the image manipulation device, where the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map.