Outputting warped images from captured video data

Abstract

Each image in a sequence of images includes three-dimensional locations of object features depicted in the image, and a first camera position of the camera when the image is captured. A gap is detected between first camera positions associated with a first continuous and first camera positions associated with a second continuous subset, the first camera positions associated with the second continuous subset adjusted to close the gap. A view path for a virtual camera is determined based on the first camera positions and the adjusted first camera positions. Second camera positions are determined for the virtual camera, for each of the second camera positions: one of the first camera positions associated with the sequence of images is selected and warped using the first camera position, the second camera position, and the three-dimensional locations of object features depicted in the selected image. A sequence of the warped images is outputted.

1 - 20 . (canceled) 
     
     
         21 . A method comprising, by a computing system:
 capturing a sequence of images using a camera;   determining, for each image in the sequence of images, (1) three-dimensional locations of object features depicted in the image and (2) a first camera position of the camera when the image is captured;   detecting a gap between (1) the first camera positions associated with a first continuous subset of the sequence of images and (2) the first camera positions associated with a second continuous subset of the sequence of images;   adjusting the first camera positions associated with the second continuous subset of the sequence of images to close the gap;   determining a view path for a virtual camera based on the first camera positions associated with the first continuous subset of the sequence of images and the adjusted first camera positions associated with the second continuous subset of the sequence of images;   determining second camera positions for the virtual camera, the second camera positions being separated by a predetermined interval along the view path;   for each of the second camera positions:
 selecting one of the first camera positions associated with the sequence of images; and 
 warping the image associated with the selected first camera position using the selected first camera position, the second camera position, and the three-dimensional locations of object features depicted in the selected image; and 
   outputting a sequence of the warped images.   
     
     
         22 . The method of claim  1 , further comprising:
 determining a plurality of control points based on the first camera positions associated with the sequence of images;   wherein determining the view path for the virtual camera comprises generating a spline using the plurality of control points.   
     
     
         23 . The method of claim  1 , further comprising:
 determining, for each of the second camera positions, a corresponding focus point, wherein the focus points are determined based in part on optimizing (1) a smoothness of a path corresponding to the focus points and (2) a compactness of the focus points;   wherein for each of the second camera positions, warping the image associated with the selected first camera position for that second camera position further uses the focus point corresponding to the second camera position.   
     
     
         24 . The method of claim  1 , further comprising:
 determining, for each of the second camera positions, a corresponding focus point, wherein the focus points are determined based in part on optimizing (1) a smoothness of a path corresponding to the focus points and (2) distances between the second camera positions and their respective focus points to be close to a predetermined target distance;   wherein for each of the second camera positions, warping of the image associated with the selected first camera position for that second camera position further uses the focus point corresponding to that second camera position.   
     
     
         25 . The method of claim  1 , wherein the sequence of images is captured via a user interface on a wireless device, the user interface comprising user instructions to move the wireless device in a curved path for capturing the sequence of images. 
     
     
         26 . The method of claim  1 , wherein for each of the second camera positions, warping the image associated with the selected first camera position comprises:
 generating a mesh corresponding to the image associated with the selected first camera position;   projecting, based on at least the selected first camera position, the three-dimensional locations of object features depicted in the image onto the mesh to generate a first set of projected points;   projecting, based on at least the second camera position, the three-dimensional locations of object features depicted in the image to generate a second set of projected points;   generating a warped mesh based on the first set of projected points and the second set of projected points; and   generating the warped image based on the image associated with the selected first camera position and the warped mesh.   
     
     
         27 . The method of claim  1 , further comprising:
 determining scaling factors for scaling the sequence of the warped images to satisfy a target resolution; and   in response to determining that the scaling factors exceed a predetermined failing threshold, generate an error message.   
     
     
         28 . The method of claim  1 , further comprising:
 determining scaling factors for scaling the sequence of the warped images to satisfy a target resolution; and   in response to determining that the scaling factors are within a predetermined acceptable range:
 scale the sequence of the warped images according to the scaling factors; 
 identifying missing pixel information in the scaled sequence of the warped images; and 
 in-painting the missing pixel information. 
   
     
     
         29 . The method of claim  1 , further comprising:
 determining scaling factors for scaling the sequence of the warped images to satisfy a target resolution; and   in response to determining that the scaling factors are below a predetermined threshold:
 scale the sequence of the warped images according to the scaling factors; and 
 crop the scaled sequence of the warped images to satisfy the target resolution. 
   
     
     
         30 . The method of claim  1 , further comprising:
 determining scaling factors for scaling the sequence of the warped images to satisfy a target resolution; and   in response to determining that one or more of the scaling factors exceeds a predetermined failing threshold:
 display an error message via a user interface on a wireless device, wherein the error message prompts a user to attempt to capture the sequence of images again. 
   
     
     
         31 . The method of claim  1 , wherein the object features represent a corner or surface of an object in the image. 
     
     
         32 . The method of claim  1 , further comprising:
 grouping the three-dimensional locations of object features depicted in the sequence of images into one or more clusters;   removing one or more outlier images from the sequence of images that include object features that exceed a threshold distance from the one or more clusters.   
     
     
         33 . The method of claim  1 , wherein the view path is defined by a polynomial equation. 
     
     
         34 . The method of claim  1 , further comprising:
 determining that two or more images in the sequence of images were captured from an identical first camera position; and   removing at least one of the two or more images that were captured from an identical first camera position from the sequence of images.   
     
     
         35 . The method claim  1 , wherein the gap is detected by determining the first camera positions associated with the first continuous subset of images and the first camera positions associated with the second continuous subset of images are parallel to each other. 
     
     
         36 . The method claim  1 , wherein the gap is detected by determining that one or more the first camera positions associated with the first continuous subset of images deviates a threshold distance from one or more of the first camera positions associated with the second continuous subset of images. 
     
     
         37 . The method of claim  1 , further comprising transmitting the outputted sequence of the warped images to an online retail platform or a social networking system. 
     
     
         38 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 capture a sequence of images using a camera;   determine, for each image in the sequence of images, (1) three-dimensional locations of object features depicted in the image and (2) a first camera position of the camera when the image is captured;   detect a gap between (1) the first camera positions associated with a first continuous subset of the sequence of images and (2) the first camera positions associated with a second continuous subset of the sequence of images;   adjust the first camera positions associated with the second continuous subset of the sequence of images to close the gap;   determine a view path for a virtual camera based on the first camera positions associated with the first continuous subset of the sequence of images and the adjusted first camera positions associated with the second continuous subset of the sequence of images;   determine second camera positions for the virtual camera, the second camera positions being separated by a predetermined interval along the view path;   for each of the second camera positions:
 select one of the first camera positions associated with the sequence of images; and 
 warp the image associated with the selected first camera position using the selected first camera position, the second camera position, and the three-dimensional locations of object features depicted in the selected image; and 
   output a sequence of the warped images.   
     
     
         39 . The media of claim  18 , wherein the gap is detected by determining the first camera positions associated with the first continuous subset of images and the first camera positions associated with the second continuous subset of images are parallel to each other. 
     
     
         40 . A system comprising:
 one or more processors; and   one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:   capture a sequence of images using a camera;   determine, for each image in the sequence of images, (1) three-dimensional locations of object features depicted in the image and (2) a first camera position of the camera when the image is captured;   detect a gap between (1) the first camera positions associated with a first continuous subset of the sequence of images and (2) the first camera positions associated with a second continuous subset of the sequence of images;   adjust the first camera positions associated with the second continuous subset of the sequence of images to close the gap;   determine a view path for a virtual camera based on the first camera positions associated with the first continuous subset of the sequence of images and the adjusted first camera positions associated with the second continuous subset of the sequence of images;   determine second camera positions for the virtual camera, the second camera positions being separated by a predetermined interval along the view path;   for each of the second camera positions:
 select one of the first camera positions associated with the sequence of images; and 
 warp the image associated with the selected first camera position using the selected first camera position, the second camera position, and the three-dimensional locations of object features depicted in the selected image; and 
   output a sequence of the warped images.