Reconstruction of obscured views of captured imagery using arbitrary captured inputs

Abstract

An imagery processing system obtains capture inputs from capture devices that might have capture parameters and characteristics that differ from those of a main imagery capture device. By normalizing outputs of those capture devices, potentially arbitrary capture devices could be used for reconstructing portions of a scene captured by the main imagery capture device when reconstructing a plate of the scene to replace an object in the scene with what the object obscured in the scene. Reconstruction could be of one main image, a stereo pair of images, or some number, N, of images where N>2.

What is claimed is: 
     
         1 . A computer-implemented method of processing images from a main imaging device using capture device inputs from capture devices, the method comprising:
 receiving reference stage parameters and normalization parameters associated with the capture device;   storing the normalization parameters;   receiving main input from the main imaging device;   receiving the capture device inputs from the capture devices;   normalizing the capture device inputs using normalization parameters to form normalized captured inputs;   aligning two or more normalized captured inputs;   obtaining reconstruction selections;   reconstructing a plate based on the selections, the normalized aligned captured inputs, and the reconstruction selection to form a reconstructed plate; and   generating modified video content using the main input and the reconstructed plate.   
     
     
         2 . The method of  claim 1 , wherein the main imaging device is a camera, the method further comprising replacing pixels in digitized video content captured with the camera to modify a captured scene in the digitized video content, wherein pixel color values of replaced pixel are determined based, at least in part, on the reconstruction selections and the capture device inputs. 
     
     
         3 . The method of  claim 2 , further comprising:
 determining which objects in the captured scene are part of a plate and which objects are to be removed from the captured scene; and   using results of determining for plate reconstruction.   
     
     
         4 . The method of  claim 1 , wherein the reference stage parameters relate at least in part to what capture devices are being used and differences between the capture devices and the main imaging device. 
     
     
         5 . The method of  claim 1 , wherein the capture devices comprise one or more of a first camera having a resolution different than the main imaging device and/or a second camera optimized for a spectrum different than that of the main imaging device. 
     
     
         6 . The method of  claim 1 , further comprising configuring the capture devices based on the normalization parameters. 
     
     
         7 . An image processing system comprising:
 a first storage for a main scene capture, captured from a main imaging device;   a normalizer for normalizing inputs from capture devices;   a second storage for normalized capture data, output by the normalizer;   a preprocessor that processes reference inputs, reference stage parameters, and capture device settings and that generates normalizing parameters;   a third storage for the normalizing parameters, coupled to an input of the normalizer; and   a machine-learning reconstructor that generates reconstructed imagery based on reconstruction parameters and reconstruction input selection.   
     
     
         8 . The image processing system of  claim 7 , wherein the first storage, the second storage, and the third storage are portions of a shared computer memory storage system. 
     
     
         9 . A computer system for processing digital video, the system comprising:
 at least one processor; and   a computer-readable medium storing instructions, which when executed by the at least one processor, causes the system to carry out the method of  claim 1 .   
     
     
         10 . A non-transitory computer-readable storage medium storing instructions, which when executed by the at least one processor of a computer system, causes the computer system to carry out the method of  claim 1 . 
     
     
         11 . A computer system comprising:
 one or more processors; and   a storage medium storing instructions, which when executed by the at least one processor, cause the system to implement the method of  claim 1 .   
     
     
         12 . A carrier medium carrying reconstructed plate image information generated according to the method of  claim 1 .