Image stitching with dynamic seam placement based on ego-vehicle state for surround view visualization

Abstract

In various examples, a state machine is used to select between a default seam placement or dynamic seam placement that avoids salient regions, and to enable and disable dynamic seam placement based on speed of ego-motion, direction of ego-motion, proximity to salient objects, active viewport, driver gaze, and/or other factors. Images representing overlapping views of an environment may be aligned to create an aligned composite image or surface (e.g., a panorama, a 360° image, bowl shaped surface) with overlapping regions of image data, and a default or dynamic seam placement may be selected based on driving scenario (e.g., driving direction, speed, proximity to nearby objects). As such, seams may be positioned in the overlapping regions of image data, and the image data may be blended at the seams to create a stitched image or surface (e.g., a stitched panorama, stitched 360° image, stitched textured surface).

What is claimed is: 
     
         1 . A method comprising:
 generating, using sensor data corresponding to a first time slice, two or more image frames representative of two or more overlapping viewpoints around an ego-object in an environment;   determining, based at least on one or more of an active viewport or a state of the ego-object, whether to use a default placement for a seam or a dynamic placement to avoid placing the seam in a salient region; and   generating a composite image based at least on stitching image data of the two or more image frames using the default placement or the dynamic seam placement of the seam.   
     
     
         2 . The method of  claim 1 , further comprising determining to use the dynamic placement based at least on at least one of the active viewport facing substantially forward in the environment or the ego-object moving substantially forward in the environment. 
     
     
         3 . The method of  claim 1 , further comprising determining to use the dynamic placement based at least on a speed of the ego-object being below a threshold speed, and at least one of the active viewport facing substantially forward in the environment or the ego-object moving substantially forward in the environment. 
     
     
         4 . The method of  claim 1 , further comprising determining to use the dynamic placement based at least on a determination that a closest detected object depicted in the two or more image frames is closer to the ego-object than a threshold proximity, and at least one of the active viewport facing substantially forward in the environment or the ego-object moving substantially forward in the environment. 
     
     
         5 . The method of  claim 1 , further comprising determining to use the default placement based at least on a speed of the ego-object being above a threshold speed. 
     
     
         6 . The method of  claim 1 , further comprising determining to use the default placement based at least on a determination that there are no detected objects depicted in the two or more image frames that are closer to the ego-object than a threshold proximity. 
     
     
         7 . The method of  claim 1 , further comprising determining to use the dynamic placement based at least on the active viewport facing substantially backwards in the environment or the ego-object moving substantially backwards in the environment. 
     
     
         8 . The method of  claim 1 , further comprising determining to use a previous seam placement from a previous set of image frames generated using sensor data captured during a previous time slice based at least on a determination that the previous seam placement avoids the salient region in the two or more image frames generated using sensor data captured during the first time slice. 
     
     
         9 . The method of  claim 1 , further comprising determining to use the dynamic placement based at least on a determination that a previous seam placement from a previous set of image frames generated using sensor data captured during a previous time slice does not avoid the salient region in the two or more image frames generated using sensor data captured during the first time slice. 
     
     
         10 . The method of  claim 1 , wherein the method is performed by at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for performing deep learning operations;   a system implemented using an edge device;   a system implemented using a robot;   a system for performing conversational AI operations;   a system for generating synthetic data;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         11 . A processor comprising:
 one or more circuits to:
 generate image data of two or more image frames corresponding to two or more overlapping viewpoints of an environment; 
 determine, based at least on at least one of an active viewport or ego-motion of an ego-object in the environment, whether to use a default placement for a seam or a dynamic placement based at least on content of the image data; and 
 generate a composite image based at least on stitching the image data using the default placement or the dynamic seam placement of the seam. 
   
     
     
         12 . The processor of  claim 11 , the one or more circuits further to determine to use the dynamic placement based at least on at least one of the active viewport facing substantially forward in the environment or the ego-motion being substantially forward in the environment. 
     
     
         13 . The processor of  claim 11 , the one or more circuits further to determine to use the dynamic placement based at least on a speed of the ego-motion being below a threshold speed, and at least one of the active viewport facing substantially forward in the environment or the ego-motion being substantially forward in the environment. 
     
     
         14 . The processor of  claim 11 , the one or more circuits further to determine to use the dynamic placement based at least on a determination that a closest detected object depicted in the aligned image data is closer to the ego-object than a threshold proximity, and at least one of the active viewport facing substantially forward in the environment or the ego-motion being substantially forward in the environment. 
     
     
         15 . The processor of  claim 11 , wherein the processor is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for performing deep learning operations;   a system implemented using an edge device;   a system implemented using a robot;   a system for performing conversational AI operations;   a system for generating synthetic data;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         16 . A system comprising:
 one or more processing units to determine, for two or more images generated using sensor data captured during a first time slice and representative of two or more overlapping viewpoints around an ego-object in an environment, whether to use a default placement for a seam or a dynamic placement that attempts to avoid placing the seam in a salient region and to generate a composite image frame based at least on stitching image data of the two or more image frames using the default placement or the dynamic seam placement of the seam.   
     
     
         17 . The system of  claim 16 , the one or more processing units further to use the default placement based at least on a determination that there are no detected objects depicted in the two or more image frames that are closer to the ego-object than a threshold proximity. 
     
     
         18 . The system of  claim 16 , the one or more processing units further to use the dynamic placement based at least on at least one of an active viewport facing substantially backwards in the environment or the ego-object moving substantially backwards in the environment. 
     
     
         19 . The system of  claim 16 , the one or more processing units further to use the dynamic placement based at least on a determination that a previous seam placement from a previous set of image frames generated using sensor data captured during a previous time slice does not avoid the salient region in the two or more image frames generated using sensor data captured during the first time slice. 
     
     
         20 . The system of  claim 16 , wherein the system is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing deep learning operations;   a system implemented using an edge device;   a system implemented using a robot;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for generating synthetic data; or   a system implemented at least partially using cloud computing resources.