System and method to improve multi-camera monocular depth estimation using pose averaging

Abstract

A method for multi-camera monocular depth estimation using pose averaging is described. The method includes determining a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes determining a multi-camera pose consistency constraint (PCC) loss associated with the multi-camera rig of the ego vehicle. The method further includes adjusting the multi-camera photometric loss according to the multi-camera PCC loss to form a multi-camera PCC photometric loss. The method also includes training a multi-camera depth estimation model and an ego-motion estimation model according to the multi-camera PCC photometric loss. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the trained multi-camera depth estimation model and the ego-motion estimation model.

1 . A method for multi-camera monocular depth estimation using pose averaging, comprising:
 adjusting a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle according to a multi-camera pose consistency constraint (PCC) loss associated with the multi-camera rig of the ego vehicle to form a multi-camera PCC photometric loss; and   training a multi-camera depth estimation model and an ego-motion estimation model according to the multi-camera PCC photometric loss.   
     
     
         2 . The method of  claim 1 , further comprising:
 capturing images of the scene surrounding the ego vehicle using the multi-camera rig of the ego vehicle, in which cameras of the multi-camera rig have a predetermined minimum overlap;   selecting target images and context images captured by the cameras of the multi-camera rig of the ego vehicle at a same time-step and at different time-steps; and   performing spatial-temporal transformations of the selected target images and the context images to determine the multi-camera photometric loss.   
     
     
         3 . The method of  claim 2 , in which performing the spatial-temporal transformations comprises warping target images and the context images captured by different cameras of the multi-camera rig at the different time-steps according to a predicted ego-motion of the ego vehicle and known extrinsics of the different cameras. 
     
     
         4 . The method of  claim 2 , in which performing the spatial-temporal transformations comprises:
 warping target images and source images captured by a same camera of the multi-camera rig at the different time-steps; and   warping the context images and target images between different cameras and captured at the different time-steps according to a predicted ego-motion and known extrinsics of the different cameras.   
     
     
         5 . The method of  claim 1 , further comprising:
 predicting a transformation from a current frame to a subsequent frame captured by each camera of the multi-camera rig;   transforming each predicted transformation to a coordinate frame of a canonical camera;   constraining a translation vector and a rotation matrix of each camera according to the canonical camera; and   generating the multi-camera PCC loss according to a translation loss and a rotation loss of the translation vector and the rotation matrix of each camera.   
     
     
         6 . The method of  claim 1 , in which training comprises enforcing pose consistency constraints to ensure cameras of the multi-camera rig follow a same rigid body motion to train scale-aware models without any ground-truth depth or ego-motion labels. 
     
     
         7 . The method of  claim 1 , in which training the depth estimation model and the ego-motion estimation model comprises leveraging cross-camera temporal contexts via spatio-temporal photometric constraints to increase an amount of overlap between cameras of the multi-camera rig using a predicted ego-motion of the ego vehicle. 
     
     
         8 . The method of  claim 1 , further comprising:
 predicting a 360° point cloud of a scene surrounding the ego vehicle according to the trained multi-camera depth estimation model and the ego-motion estimation model; and   planning a trajectory of the ego vehicle according to the 360° point cloud of the scene surrounding the ego vehicle.   
     
     
         9 . A non-transitory computer-readable medium having program code recorded thereon for multi-camera monocular depth estimation using pose averaging, the program code being executed by a processor and comprising:
 program code to adjust a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle according to a multi-camera pose consistency constraint (PCC) loss associated with the multi-camera rig of the ego vehicle to form a multi-camera PCC photometric loss; and   program code to train a multi-camera depth estimation model and an ego-motion estimation model according to the multi-camera PCC photometric loss.   
     
     
         10 . The non-transitory computer-readable medium of  claim 9 , further comprises:
 program code to capture images of the scene surrounding the ego vehicle using the multi-camera rig of the ego vehicle, in which cameras of the multi-camera rig have a predetermined minimum overlap;   program code to select target images and context images captured by the cameras of the multi-camera rig of the ego vehicle at a same time-step and at different time-steps; and   program code to perform spatial-temporal transformations of selected target images and the context images to determine the multi-camera photometric loss.   
     
     
         11 . The non-transitory computer-readable medium of  claim 10 , in which the program code to perform spatial-temporal transformations comprises program code to warp the target images and the context images captured by different cameras of the multi-camera rig at the different time-steps according to a predicted ego-motion of the ego vehicle and known extrinsics of the different cameras. 
     
     
         12 . The non-transitory computer-readable medium of  claim 10 , in which the program code to perform spatial-temporal transformations comprises:
 program code to warp the target images and source images captured by a same camera of the multi-camera rig at the different time-steps; and   program code to warp the context images and the target images between different cameras and captured at the different time-steps according to a predicted ego-motion and known extrinsics of the different cameras.   
     
     
         13 . The non-transitory computer-readable medium of  claim 9 , further comprising:
 program code to predict a transformation from a current frame to a subsequent frame captured by each camera of the multi-camera rig;   program code to transform each predicted transformation to a coordinate frame of a canonical camera;   program code to constrain a translation vector and a rotation matrix of each camera according to the canonical camera; and   program code to generate the multi-camera PCC loss according to a translation loss and a rotation loss of the translation vector and the rotation matrix of each camera.   
     
     
         14 . The non-transitory computer-readable medium of  claim 9 , in which the program code to train comprises program code to enforce pose consistency constraints to ensure cameras of the multi-camera rig follow a same rigid body motion to train scale-aware models without any ground-truth depth or ego-motion labels. 
     
     
         15 . The non-transitory computer-readable medium of  claim 9 , in which the program code to train the depth estimation model and the ego-motion estimation model comprises program code to leverage cross-camera temporal contexts via spatio-temporal photometric constraints to increase an amount of overlap between cameras of the multi-camera rig using a predicted ego-motion of the ego vehicle. 
     
     
         16 . The non-transitory computer-readable medium of  claim 9 , further comprising:
 program code to predict a 360° point cloud of a scene surrounding the ego vehicle according to the trained multi-camera depth estimation model and the ego-motion estimation model   program code to plan a trajectory of the ego vehicle according to the 360° point cloud of the scene surrounding the ego vehicle.