Reconstruction of obscured views in captured imagery using pixel replacement from secondary imagery

Abstract

An imagery processing system determines alternative pixel color values for pixels of captured imagery where the alternative pixel color values are obtained from alternative sources. A main imagery capture device, such as a camera, captures main imagery such as still images and/or video sequences, of a live action scene. Alternative devices capture imagery of the live action scene, in some spectra and form, and that alternative imagery is processed to provide alternatives for pixel ranges from the main imagery.

1 . A computer-implemented method of processing images from a main imaging device using capture device inputs from capture devices, the method comprising:
 obtaining a main image, wherein the main image comprises a main pixel array derived from a view of a live action scene;   obtaining a plurality of alternative images, wherein each alternative image in the plurality of images comprises an alternative array of pixels including volume information;   mapping pixels of the alternative array of pixels to pixels of the main pixel array;   determining a set of one or more replacement regions of the main pixel array;   selecting one or more corresponding alternative regions from one or more selected alternative images of the plurality of alternative images, mapped to the pixels of the main pixel array; and   generating a synthetic image comprising a synthetic pixel array wherein pixel color values of the synthetic pixel array are pixel color values of pixels from the main pixel array for regions not replaced and are pixel color values of pixels derived from the corresponding alternative regions of the one or more selected alternative images for the replacement regions.   
     
     
         2 . The method of  claim 1 , wherein mapping pixels of the alternative array of pixels to pixels of the main pixel array comprises, for at least one of the alternative array of pixels, translating, rotating, and/or shearing the pixel array of the at least one of the alternative array of pixels to align pixels with the main pixel array. 
     
     
         3 . The method of  claim 1 , wherein the set of one or more replacement regions comprises a single replacement region that forms a continuous region. 
     
     
         4 . The method of  claim 1 , wherein the set of one or more replacement regions comprises a plurality of discontinuous replacement regions. 
     
     
         5 . The method of  claim 1 , further comprising determining the set of one or more replacement regions using image segmentation to identify boundaries of objects appearing in the main pixel array. 
     
     
         6 . The method of  claim 1 , further comprising obtaining user input for determining the set of one or more replacement regions. 
     
     
         7 . The method of  claim 1 , wherein selecting one or more corresponding alternative regions comprises selecting a plurality of corresponding alternative regions, the method further comprising blending the plurality of corresponding alternative regions to derive the corresponding alternative regions of the one or more selected alternative images for the replacement regions. 
     
     
         8 . The method of  claim 1 , wherein selecting one or more corresponding alternative regions comprises determining, using a voting or weighting process, which of a plurality of corresponding alternative regions provides a best fit for the replacement regions. 
     
     
         9 . The method of  claim 8 , further comprising determining the best fit for the replacement regions using a color matching process. 
     
     
         10 . The method of  claim 9 , wherein the color matching process comprises a color matching process across a plurality of frames of a main image device video sequence. 
     
     
         11 . The method of  claim 9 , wherein the color matching process comprises a color matching process across the main pixel array. 
     
     
         12 . The method of  claim 1 , further comprising:
 providing the one or more determined alternative regions to a user for selection.   
     
     
         13 . The method of  claim 1 , further comprising
 providing the one or more determined alternative regions to an automated process for selection.   
     
     
         14 . An apparatus including a processor configured to perform the actions recited in  claim 1 . 
     
     
         15 . One or more tangible processor-readable media including instructions executable by one or more processors to perform the actions recited in  claim 1 .