Modeling consistency in modalities of data for semantic segmentation

Abstract

Techniques and systems are provided for training a machine learning (ML) model. A technique can include generating a first set of features for objects in images, predicting image feature labels for the first set of features, comparing the predicted image feature labels to ground truth image feature labels to evaluate a first loss function, perform a perspective transform on the first set of features to generate a birds eye view (BEV) projected image features, combining the BEV projected image features and a first set of flattened features to generate combined image features, generating a segmented BEV map of the environment based on the combined image features, comparing the segmented BEV map to a ground truth segmented BEV map to evaluate a second loss function, and training the ML model for generation of segmented BEV maps based on the evaluated first loss function and the evaluated second loss function.

What is claimed is: 
     
         1 . An apparatus for training machine learning models, comprising:
 at least one memory comprising instructions; and   at least one processor coupled to the at least one memory and configured to:
 obtain one or more images of an environment; 
 generate, using a first machine learning model, a first set of features for one or more objects in the one or more images; 
 predict, using a second machine learning model, image feature labels for the first set of features; 
 compare the predicted image feature labels for the first set of features to ground truth image feature labels corresponding to the one or more images to evaluate a first loss function; 
 perform a perspective transform on the first set of features to generate a birds eye view (BEV) projected image features; 
 combine the BEV projected image features and a first set of flattened features to generate combined image features, wherein the first set of flattened features are generated based on a first three-dimensional representation of the environment; 
 generate, using the first machine learning model, a segmented BEV map of the environment based on the combined image features; 
 compare the segmented BEV map to a ground truth segmented BEV map to evaluate a second loss function; and 
 train the first machine learning model for generation of one or more segmented BEV maps based on the evaluated first loss function and the evaluated second loss function. 
   
     
     
         2 . The apparatus of  claim 1 , wherein the at least one processor is further configured to:
 obtain the first three-dimensional representation of the environment;   generate a second set of features for one or more objects in the first three-dimensional representation; and   flatten the second set of features from three dimensions to two dimensions to generate the first set of flattened features.   
     
     
         3 . The apparatus of  claim 1 , wherein the first three-dimensional representation of the environment comprises one of a radar point cloud or a lidar point cloud. 
     
     
         4 . The apparatus of  claim 1 , wherein the first three-dimensional representation of the environment comprises a lidar point cloud and wherein the at least one processor is further configured to:
 predict a predicted first set of flattened features based on the BEV projected image features;   compare the predicted first set of flattened features to the first set of flattened features to evaluate a third loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, and the evaluated third loss function.   
     
     
         5 . The apparatus of  claim 4 , wherein the at least one processor is further configured to:
 obtain a radar point cloud representation of the environment;   generate a third set of features for one or more objects in the radar point cloud representation;   flatten the third set of features from three dimensions to two dimensions to generate a second set of flattened features; and   generate the segmented BEV map of the environment based on the combined BEV projected image features, the first set of flattened features, and the second set of flattened features.   
     
     
         6 . The apparatus of  claim 5 , wherein the at least one processor is further configured to:
 predict a predicted second set of flattened features based on the BEV projected image features;   compare the predicted second set of flattened features to the second set of flattened features to evaluate a fourth loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, and the evaluated fourth loss function.   
     
     
         7 . The apparatus of  claim 6 , wherein the at least one processor is further configured to:
 perform an additional perspective transform on the predicted image feature labels to generate a predicted segmented BEV map;   compare the predicted segmented BEV map to the generated segmented BEV map to evaluate a fifth loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, the evaluated fourth loss function, and the evaluated fifth loss function.   
     
     
         8 . The apparatus of  claim 6 , wherein the at least one processor is further configured to:
 perform an additional perspective transform on the generated segmented BEV map to generate an additional predicted image feature labels;   compare the predicted image feature labels to the additional predicted image feature labels to evaluate a fifth loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, the evaluated fourth loss function, and the evaluated fifth loss function.   
     
     
         9 . The apparatus of  claim 1 , wherein the at least one processor is configured to:
 predict, using the second machine learning model that is separate from the first machine learning model, feature labels for the ground truth image feature labels.   
     
     
         10 . The apparatus of  claim 1 , wherein the at least one processor is configured to train the first machine learning model at least in part by back propagating the evaluated first loss function and the evaluated second loss function. 
     
     
         11 . A method for training machine learning models, comprising:
 obtaining one or more images of an environment;   generating, using a first machine learning model, a first set of features for one or more objects in the one or more images;   predicting, using a second machine learning model, image feature labels for the first set of features;   comparing the predicted image feature labels for the first set of features to ground truth image feature labels corresponding to the one or more images to evaluate a first loss function;   performing a perspective transform on the first set of features to generate a birds eye view (BEV) projected image features;   combining the BEV projected image features and a first set of flattened features to generate combined image features, wherein the first set of flattened features are generated based on a first three-dimensional representation of the environment;   generating, using the first machine learning model, a segmented BEV map of the environment based on the combined image features;   comparing the segmented BEV map to a ground truth segmented BEV map to evaluate a second loss function; and   training the first machine learning model for generation of one or more segmented BEV maps based on the evaluated first loss function and the evaluated second loss function.   
     
     
         12 . The method of  claim 11 , further comprising:
 obtaining the first three-dimensional representation of the environment;   generating a second set of features for one or more objects in the first three-dimensional representation; and   flattening the second set of features from three dimensions to two dimensions to generate the first set of flattened features.   
     
     
         13 . The method of  claim 11 , wherein the first three-dimensional representation of the environment comprises one of a radar point cloud or a lidar point cloud. 
     
     
         14 . The method of  claim 11 , wherein the first three-dimensional representation of the environment comprises a lidar point cloud and further comprising:
 predicting a predicted first set of flattened features based on the BEV projected image features;   comparing the predicted first set of flattened features to the first set of flattened features to evaluate a third loss function; and   training the first machine learning model based on the evaluated first loss function, the evaluated second loss function, and the evaluated third loss function.   
     
     
         15 . The method of  claim 14 , further comprising:
 obtaining a radar point cloud representation of the environment;   generating a third set of features for one or more objects in the radar point cloud representation;   flattening the third set of features from three dimensions to two dimensions to generate a second set of flattened features; and   generating the segmented BEV map of the environment based on the combined BEV projected image features, the first set of flattened features, and the second set of flattened features.   
     
     
         16 . The method of  claim 15 , further comprising:
 predicting a predicted second set of flattened features based on the BEV projected image features;   comparing the predicted second set of flattened features to the second set of flattened features to evaluate a fourth loss function; and   training the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, and the evaluated fourth loss function.   
     
     
         17 . The method of  claim 16 , further comprising:
 performing an additional perspective transform on the predicted image feature labels to generate a predicted segmented BEV map;   comparing the predicted segmented BEV map to the generated segmented BEV map to evaluate a fifth loss function; and   training the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, the evaluated fourth loss function, and the evaluated fifth loss function.   
     
     
         18 . The method of  claim 16 , further comprising:
 performing an additional perspective transform on the generated segmented BEV map to generate an additional predicted image feature labels;   comparing the predicted image feature labels to the additional predicted image feature labels to evaluate a fifth loss function; and   training the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, the evaluated fourth loss function, and the evaluated fifth loss function.   
     
     
         19 . The method of  claim 11 , further comprising:
 predicting, using the second machine learning model that is separate from the first machine learning model, feature labels for the ground truth image feature labels.   
     
     
         20 . The method of  claim 11 , further comprising training the first machine learning model at least in part by back propagating the evaluated first loss function and the evaluated second loss function. 
     
     
         21 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:
 obtain one or more images of an environment;   generate, using a first machine learning model, a first set of features for one or more objects in the one or more images;   predict, using a second machine learning model, image feature labels for the first set of features;   compare the predicted image feature labels for the first set of features to ground truth image feature labels corresponding to the one or more images to evaluate a first loss function;   perform a perspective transform on the first set of features to generate a birds eye view (BEV) projected image features;   combine the BEV projected image features and a first set of flattened features to generate combined image features, wherein the first set of flattened features are generated based on a first three-dimensional representation of the environment;   generate, using the first machine learning model, a segmented BEV map of the environment based on the combined image features;   compare the segmented BEV map to a ground truth segmented BEV map to evaluate a second loss function; and   train the first machine learning model for generation of one or more segmented BEV maps based on the evaluated first loss function and the evaluated second loss function.   
     
     
         22 . The non-transitory computer-readable medium of  claim 21 , wherein the instructions cause the at least one processor to:
 obtain the first three-dimensional representation of the environment;   generate a second set of features for one or more objects in the first three-dimensional representation; and   flatten the second set of features from three dimensions to two dimensions to generate the first set of flattened features.   
     
     
         23 . The non-transitory computer-readable medium of  claim 21 , wherein the first three-dimensional representation of the environment comprises one of a radar point cloud or a lidar point cloud. 
     
     
         24 . The non-transitory computer-readable medium of  claim 21 , wherein the first three-dimensional representation of the environment comprises a lidar point cloud and wherein the instructions cause the at least one processor to:
 predict a predicted first set of flattened features based on the BEV projected image features;   compare the predicted first set of flattened features to the first set of flattened features to evaluate a third loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, and the evaluated third loss function.   
     
     
         25 . The non-transitory computer-readable medium of  claim 24 , wherein the instructions cause the at least one processor to:
 obtain a radar point cloud representation of the environment;   generate a third set of features for one or more objects in the radar point cloud representation;   flatten the third set of features from three dimensions to two dimensions to generate a second set of flattened features; and   generate the segmented BEV map of the environment based on the combined BEV projected image features, the first set of flattened features, and the second set of flattened features.   
     
     
         26 . The non-transitory computer-readable medium of  claim 25 , wherein the instructions cause the at least one processor to:
 predict a predicted second set of flattened features based on the BEV projected image features;   compare the predicted second set of flattened features to the second set of flattened features to evaluate a fourth loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, and the evaluated fourth loss function.   
     
     
         27 . The non-transitory computer-readable medium of  claim 26 , wherein the instructions cause the at least one processor to:
 perform an additional perspective transform on the predicted image feature labels to generate a predicted segmented BEV map;   compare the predicted segmented BEV map to the generated segmented BEV map to evaluate a fifth loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, the evaluated fourth loss function, and the evaluated fifth loss function.   
     
     
         28 . The non-transitory computer-readable medium of  claim 26 , wherein the instructions cause the at least one processor to:
 perform an additional perspective transform on the generated segmented BEV map to generate an additional predicted image feature labels;   compare the predicted image feature labels to the additional predicted image feature labels to evaluate a fifth loss function; and   train the first machine learning model based on the evaluated first loss function, the evaluated second loss function, the evaluated third loss function, the evaluated fourth loss function, and the evaluated fifth loss function.   
     
     
         29 . The non-transitory computer-readable medium of  claim 21 , wherein the instructions cause the at least one processor to:
 predict, using the second machine learning model that is separate from the first machine learning model, feature labels for the ground truth image feature labels.   
     
     
         30 . The non-transitory computer-readable medium of  claim 21 , wherein the instructions cause the at least one processor to train the first machine learning model at least in part by back propagating the evaluated first loss function and the evaluated second loss function.