Methods for Full Parallax Compressed Light Field 3D Imaging Systems

Abstract

A compressed light field imaging system is described. The light field 3D data is analyzed to determine optimal subset of light field samples to be (acquired) rendered, while the remaining samples are generated using multi-reference depth-image based rendering. The light field is encoded and transmitted to the display. The 3D display directly reconstructs the light field and avoids data expansion that usually occurs in conventional imaging systems. The present invention enables the realization of full parallax 3D compressed imaging system that achieves high compression performance while minimizing memory and computational requirements.

What is claimed is: 
     
         1 . A method for light field imaging systems comprising,
 compressed capturing light field data from a light field to provide compressed light field data; and   reconstructing and displaying the light field from the compressed light field data at a light field display system;   wherein compressed capturing the light field data comprises analyzing 3D scene data to select reference hogels representing a 3D scene.   
     
     
         2 . The method of  claim 1 , wherein compressed capturing the light field data further comprises:
 creating disparity maps for the reference hogels, and   synthesizing target hogels from the reference hogels texture and disparity maps.   
     
     
         3 . The method of  claim 2  wherein the target hogels are synthesized using multiple-reference depth-image based rendering. 
     
     
         4 . The method of  claim 3 , further comprising selecting multiple reference hogels. 
     
     
         5 . The method of  claim 4 , further comprising forward warping of the disparity maps of the selected reference hogels. 
     
     
         6 . The method of  claim 5 , further comprising filtering of the forward warped disparity maps. 
     
     
         7 . The method of  claim 6 , further comprising merging the forward warped disparity maps into a single disparity map. 
     
     
         8 . The method of  claim 7 , further comprising backward warping of textures according to the merged disparity map. 
     
     
         9 . The method of  claim 8 , wherein backward warping of the textures comprises using fractional pixel shifting. 
     
     
         10 . The method of  claim 8 , further comprising capturing and using reference hogels in the compressed capturing using a higher resolution than the resolution of the light field display system. 
     
     
         11 . The method of  claim 8 , further comprising hole filling after backward warping of the textures. 
     
     
         12 . The method of  claim 1 , wherein compressed capturing the light field data comprises compressed rendering and display-matched compressing to form the compressed light field data, and wherein analyzing the 3D scene data to select the reference hogels representing the 3D scene comprises performing a visibility test to choose the reference hogels to be rendered in the compressed rendering. 
     
     
         13 . The method of  claim 12 , further comprising selecting a subset of reference hogels from a larger set of reference hogels. 
     
     
         14 . The method of  claim 12 , further comprising selecting an initial set of reference hogels, and adding more reference hogels to better represent objects of the 3D scene by frusta of the initial set of reference hogels. 
     
     
         15 . The method of  claim 12 , further comprising preprocessing the 3D scene data before performing the visibility test to extract information from the 3D scene data for the visibility test. 
     
     
         16 . The method of  claim 15  wherein preprocessing the 3D scene data comprises utilizing computer generated scenes captured by computer graphics methods. 
     
     
         17 . The method of  claim 15  further comprising preprocessing utilizing real world or live scenes captured by one or more cameras or camera types. 
     
     
         18 . The method of  claim 1 , further comprising rendering of the reference hogels. 
     
     
         19 . The method of  claim 18 , further comprising low-pass filtering of the rendered reference hogels to avoid non-resolvable features. 
     
     
         20 . The method of  claim 18 , further comprising obtaining per-pixel depth of the reference hogels. 
     
     
         21 . The method of  claim 20 , further comprising converting per-pixel depth to disparity and quantizing the disparity information during the depth to disparity conversion after reference hogel rendering. 
     
     
         22 . A method for light field imaging systems comprising:
 compressed capturing light field data from a light field to provide compressed light field data;   reconstructing and displaying the light field from the compressed light field data at a light field display system;   formatting the compressed light field data by an encoder to generate a bitstream for transmission to the light field display system for decoding, and the display of the light field; and   dividing the light field into N×N blocks of hogels, each for independent encoding during the compressed capturing and for independent decoding at the light field display system;   wherein the bitstream is matched to the light field display system.   
     
     
         23 . The method of  claim 22 , further comprising selecting one or more seed hogels for each of the N×N blocks of hogels for encoding, and encoding residual hogels relative to the seed hogels. 
     
     
         24 . The method of  claim 23 , wherein the encoding comprises texture encoding of seed and residual hogels. 
     
     
         25 . The method of  claim 24 , further comprising synthesizing a prediction for the residual hogels, using a texture and a disparity map of a plurality of seed hogels. 
     
     
         26 . The method of  claim 23 , wherein the encoding comprises disparity encoding of seed and residual hogels. 
     
     
         27 . The method of  claim 23 , further comprising coding the seed and residual hogels with a bit allocation algorithm. 
     
     
         28 . The method of  claim 27 , further comprising decoding the seed and residual hogels at the light field display system using a decoding algorithm, the decoding algorithm being complimentary to the bit allocation algorithm. 
     
     
         29 . The method of  claim 28 , further comprising parallel decoding of all seed and residual hogels at the light field display system with multiprocessing techniques using multiple decoding units. 
     
     
         30 . The method of  claim 29 , further comprising packetizing the bit allocation for distribution among multiple decoding units. 
     
     
         31 . A method for light field imaging systems comprising:
 using depth or disparity image-based rendering to perform compressed capturing of light field data from a light field to provide compressed light field data;   reconstructing and displaying the light field from the compressed light field data at a light field display system;   using depth or disparity image-based rendering to perform compressed capturing of a dynamic light field with temporal correlation;   using motion estimation in the compressed capturing of the light field data; and   reusing hardware or software that is used for the depth or disparity image-based rendering to perform both the motion estimation and motion compensation.   
     
     
         32 . A method for light field imaging systems comprising:
 compressed capturing light field data from a light field to provide compressed light field data;   reconstructing and displaying the light field from the compressed light field data at a light field display system; and   time multiplexing the compressed light field data by rotations and translations of a light field emitter of the light field display system.   
     
     
         33 . The method of  claim 32  further comprising temporally multiplexing the compressed light field data into smaller subsets organized spatially or angularly. 
     
     
         34 . The method of  claim 33  further comprising creating subsets that have a predetermined amount of correlation. 
     
     
         35 . The method of  claim 32  further comprising reusing display pixels, memory and compressed light field data by time multiplexed reconstruction of the compressed light field data, thereby increasing the field of view of the light field display system. 
     
     
         36 . The method of  claim 32  wherein time multiplexing the compressed light field data comprises creating multiplexing segments that naturally divide the compressed light field data into hogel modulation groups. 
     
     
         37 . The method of  claim 36  further comprising adaptive allocating the interface bandwidth, and wherein time multiplexing the compressed light field data comprises time multiplexing of a display-matched encoder's compression algorithm together with the adaptive allocation of the interface bandwidth. 
     
     
         38 . The method of  claim 32  further comprising dividing the compressed light field data into hogel groups that are used for parallel implementation of compressed rendering and display-matched encoding. 
     
     
         39 . A method for light field imaging systems comprising:
 compressed capturing light field data from a light field to provide compressed light field data;   wherein compressed capturing the light field data including:
 compressed rendering the light field data and display-matched encoding the rendered light field data that matches capabilities of a light field display system, and 
 analyzing 3D scene data to select reference hogels representing a 3D scene; and 
 reconstructing and displaying the display-matched encoding of the rendered light field data at the light field display system. 
   
     
     
         40 . The method of  claim 39 , wherein compressed capturing the light field data further includes synthesizing target hogels from the reference hogels. 
     
     
         41 . The method of  claim 40  wherein the target hogels are synthesized using multiple-reference depth-image based rendering. 
     
     
         42 . The method of  claim 39 , further comprising rendering of the reference hogels. 
     
     
         43 . The method of  claim 42  further comprising obtaining per-pixel depth of the reference hogels. 
     
     
         44 . The method of  claim 43 , further comprising converting per-pixel depth to disparity and quantizing the disparity during the depth to disparity conversion after reference hogel rendering. 
     
     
         45 . The method of  claim 39 , wherein the selected reference hogels are captured with a higher resolution than the resolution of the light field display system. 
     
     
         46 . The method of  claim 39 , further comprising dividing the light field data into N×N blocks of hogels, each for independent encoding during the compressed capturing and for independent decoding at the light field display system. 
     
     
         47 . The method of  claim 46 , further comprising selecting one or more seed hogels for each of the N×N blocks of hogels for encoding, and encoding residual hogels relative to the seed hogels. 
     
     
         48 . The method of  claim 47 , wherein the encoding comprises texture encoding of seed and residual hogels. 
     
     
         49 . The method of  claim 47 , wherein the encoding comprises disparity encoding of seed and residual hogels. 
     
     
         50 . The method of  claim 49 , further comprising parallel decoding of all seed and residual hogels at the light field display system using multiprocessing techniques using multiple decoding units. 
     
     
         51 . The method of  claim 39  further comprising performing a hierarchical compressed capturing of the light field data, and performing a hierarchical decompression of the compressed light field data at the light field display system. 
     
     
         52 . The method of  claim 39  further comprising compressed capturing of a dynamic light field by utilizing temporal correlation. 
     
     
         53 . The method of  claim 52 , further comprising motion estimation in the compressed capturing of the light field data. 
     
     
         54 . The method of  claim 53  wherein the compressed capturing uses depth or disparity image-based rendering, and further comprises reusing hardware and/or software that is used for the depth or disparity image-based rendering to perform both motion estimation and motion compensation. 
     
     
         55 . The method of  claim 39  further comprising time multiplexing the compressed light field data by rotations and/or translations of a light field emitter in the light field display system. 
     
     
         56 . The method of  claim 39  further comprising dividing the compressed light field data into hogel modulation groups that are used for parallel implementation of compressed rendering and display-matched encoding of the compressed capturing.