In camera implementation of selecting and stitching frames for panoramic imagery

Abstract

A system, method, and computer program product for selecting qualifying frames from an image sequence for use in subsequent stitching into a composite panoramic image are disclosed. Incoming frames from any source may be cropped and downscaled prior to evaluation against qualifying criteria relating to image overlap and local motion. Qualifying images are saved and/or output. The resulting panoramic image generally uses fewer qualifying images and appears smoother and has fewer artifacts than those of the prior art. The qualifying criterion for image overlap is a predetermined overlap margin or percentage between a current image and a previous image from the sequence. The qualifying criterion for image motion includes a maximum amount of local motion, often due to passing objects. The embodiments may process incoming images in real time or from stored sequences. Problems may trigger user warnings.

1 - 25 . (canceled) 
     
     
         26 . A method of selecting frames of an image sequence for panoramic stitching, comprising:
 evaluating frame overlap constraints between a current frame and a previously selected frame of the image sequence;   evaluating frame motion constraints between the current and previously selected frames, the evaluating of the frame motion constraints comprising determining whether local motion in an overlapping region between the current and previously selected frames exceeds a threshold; and   selecting the current frame for panoramic stitching in response to the frame overlap constraints and the frame motion constraints being met.   
     
     
         27 . The method of  claim 26 , wherein the frame overlap constraints comprise a specified directional overlap percentage with respect to the previously selected frame. 
     
     
         28 . The method of  claim 26 , wherein evaluating the frame motion constraints are comprises:
 performing motion estimation of the current frame with respect to the previously selected frame;   dividing the overlapping region between the current frame and the previously selected frame into blocks for subsequent analysis;   summing the number of pixels in each block having a difference in pixel illumination value for the current frame and the previously selected frame exceeding a predefined threshold; and   selectively determining that large local motion violates the frame motion constraints according to the percentage of blocks having pixel sums indicating high illumination value differences.   
     
     
         29 . The method of  claim 26 , further comprising downscaling the frames to be processed. 
     
     
         30 . The method of  claim 26 , further comprising warning a user if an error condition occurs, the error condition including at least one of: the frame motion constraints being violated, the frame overlap constraints being violated, excess camera tilt occurring in any orientation, and insufficient qualifying frames being saved. 
     
     
         31 . The method of  claim 30 , wherein the frame motion constraints are violated by at least one of: excess local motion, excess panning speed, insufficient panning speed, and excess hand motion drift in a direction perpendicular to a sweep direction. 
     
     
         32 . The method of  claim 26 , wherein the frame overlap constraints for the current and previously selected frames are evaluated against a predefined overlap margin and the frame motion constraints are evaluated for the current and previously selected frames and at least one additional frame of the image sequence within the predefined overlap margin. 
     
     
         33 . The method of  claim 26 , wherein evaluating the frame motion constraints comprises generating a histogram comprising a plurality of motion vectors for the current frame with respect to the previously selected frame, wherein a motion vector corresponding to the largest bin the in the histogram is employed in determining if the frame overlap constraints and the frame motion constraints are met. 
     
     
         34 . A device for selecting frames of an image sequence for panoramic stitching, comprising:
 an input interface configured to receive the image sequence;   a memory configured to store the image sequence; and   a processor operatively coupled to the memory and the input interface and configured to:
 evaluate frame overlap constraints between a current frame and a previously selected frame of the image sequence; 
 evaluate frame motion constraints between the current and previously selected frames, the evaluation of the frame motion constraints comprising determining whether local motion in an overlapping region between the current and previously selected frames exceeds a threshold; and 
 select the current frame for panoramic stitching in response to the frame overlap constraints and the frame motion constraints being met. 
   
     
     
         35 . The device of  claim 34 , wherein the frame overlap constraints comprise a specified directional overlap percentage with respect to the previously selected frame. 
     
     
         36 . The device of  claim 34 , wherein the processor is further configured to evaluate the frame motion constraints via:
 performing motion estimation of the current frame with respect to the previously selected frame;   dividing the overlapping region between the current frame and the previously selected frame into blocks for subsequent analysis;   summing the number of pixels in each block having a difference in pixel illumination value for the current frame and the previously selected frame exceeding a predefined threshold; and   selectively determining that large local motion violates the frame motion constraints according to the percentage of blocks having pixel sums indicating high illumination value differences.   
     
     
         37 . The device of  claim 34 , wherein the processor is further configured to warn a user if an error condition occurs, the error condition including at least one of: the frame motion constraints being violated, the frame overlap constraints being violated, excess camera tilt occurring in any orientation, and insufficient qualifying frames being saved. 
     
     
         38 . The device of  claim 34 , wherein the processor is further configured to: i) evaluate the frame overlap constraints for the current and previously selected frames against a predefined overlap margin and ii) evaluate the frame motion constraints for the current and previously selected frames and at least one additional frame of the image sequence within the predefined overlap margin. 
     
     
         39 . The device of  claim 34 , wherein the processor is further configured to: i) evaluate the frame motion constraints based on generating a histogram comprising a plurality of motion vectors for the current frame with respect to the previously selected frame and ii) determine if the frame overlap constraints and the frame motion constraints are met based on a motion vector corresponding to the largest bin the in the histogram. 
     
     
         40 . A non-transitory computer readable medium having stored thereon instructions that, when executed, cause a processor of a device to:
 evaluate frame overlap constraints between a current frame and a previously selected frame;   evaluate frame motion constraints between the current and previously selected frames, the evaluation of the frame motion constraints comprising determining whether local motion in an overlapping region between the current and previously selected frames exceeds a threshold; and   select the current frame for panoramic stitching in response to the frame overlap constraints and the frame motion constraints being met.   
     
     
         41 . The non-transitory computer readable medium of  claim 40 , wherein the frame overlap constraints comprise a specified directional overlap percentage with respect to the previously selected frame. 
     
     
         42 . The non-transitory computer readable medium of  claim 40 , further having stored thereon instructions that, when executed, cause the processor to evaluate the frame motion constraints via:
 performing motion estimation of the current frame with respect to the previously selected frame;   dividing the overlapping region between the current frame and the previously selected frame into blocks for subsequent analysis;   summing the number of pixels in each block having a difference in pixel illumination value for the current frame and the previously selected frame exceeding a predefined threshold; and   selectively determining that large local motion violates the frame motion constraints according to the percentage of blocks having pixel sums indicating high illumination value differences.   
     
     
         43 . The non-transitory computer readable medium of  claim 40 , further having stored thereon instructions that, when executed, cause the processor to warn a user if an error condition occurs, the error condition including at least one of: the frame motion constraints being violated, the frame overlap constraints being violated, excess camera tilt occurring in any orientation, and insufficient qualifying frames being saved. 
     
     
         44 . The non-transitory computer readable medium of  claim 40 , further having stored thereon instructions that, when executed, cause the processor to:
 evaluate the frame overlap constraints for the current and previously selected frames against a predefined overlap margin; and   evaluate the frame motion constraints for the current and previously selected frames and at least one additional frame of the image sequence within the predefined overlap margin.   
     
     
         45 . The non-transitory computer readable medium of  claim 40 , further having stored thereon instructions that, when executed, cause the processor to: evaluate the frame motion constraints via generating a histogram comprising a plurality of motion vectors for the current frame with respect to the previously selected frame; and employ a motion vector corresponding to the largest bin the in the histogram in determining if the frame overlap constraints and the frame motion constraints are met.