Scenario sensing method and related device thereof

Abstract

This application discloses a scenario sensing method and a related device. The method includes: when scenario sensing needs to be performed on the target scenario, image data of the target scenario and/or point cloud data of the target scenario may be obtained, and the image data and/or the point cloud data may be input into a target model; the target model may first perform feature extraction on the image data and/or the point cloud data to obtain a first BEV feature of the image data and/or a second BEV feature of the point cloud data; the target model may perform feature fusion on the first BEV feature and/or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and finally, the target model may perform further processing on the feature fusion result to obtain a sensing result of the target scenario.

1 . A scenario sensing method, wherein the method is implemented through a target model, and the method comprises:
 obtaining image data of a target scenario or point cloud data of the target scenario;   performing feature extraction on the image data or the point cloud data to obtain a first bird eye view (BEV) feature of the image data or a second BEV feature of the point cloud data;   performing feature fusion on the first BEV feature or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and   obtaining a sensing result of the target scenario based on the feature fusion result.   
     
     
         2 . The method according to  claim 1 , wherein performing the feature extraction on the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data comprises:
 encoding the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data.   
     
     
         3 . The method according to  claim 1 , wherein performing the feature fusion on the first BEV feature or the second BEV feature based on the preset third BEV feature to obtain the feature fusion result comprises:
 performing, based on the preset third BVE feature, processing based on a deformable attention mechanism on the first BEV feature or the second BEV feature to obtain a fourth BEV feature;   adding the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature;   performing processing based on a mixture of experts model on the fifth BEV feature to obtain a sixth BVE feature;   adding the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and   obtaining the feature fusion result based on the seventh BVE feature.   
     
     
         4 . The method according to  claim 3 , wherein performing, based on the preset third BVE feature, the processing based on the deformable attention mechanism on the first BEV feature or the second BEV feature to obtain the fourth BEV feature comprises:
 performing processing based on a first multilayer perceptron on the preset third BVE feature to obtain first location information of a first sub-feature of the first BEV feature or second location information of a second sub-feature of the second BEV feature;   obtaining the first sub-feature or the second sub-feature from the first BEV feature or the second BEV feature based on the first location information or the second location information;   performing processing based on a second multilayer perceptron on the third BVE feature to obtain first weight information of the first sub-feature of the first BEV feature or second weight information of the second sub-feature of the second BEV feature; and   performing weighted summation on the first sub-feature or the second sub-feature based on the first weight information or the second weight information to obtain the fourth BEV feature.   
     
     
         5 . The method according to  claim 3 , wherein performing the processing based on the mixture of experts model on the fifth BEV feature to obtain the sixth BVE feature comprises:
 performing processing based on a first expert model on the fifth BEV feature to obtain an eighth BEV feature;   performing processing based on a second expert model on the fifth BEV feature to obtain a ninth BEV feature; and   performing processing based on a third expert model on the eighth BEV feature and the ninth BEV feature to obtain the sixth BEV feature.   
     
     
         6 . The method according to  claim 3 , wherein performing the processing based on the mixture of experts model on the fifth BEV feature to obtain the sixth BVE feature comprises:
 dividing the fifth BEV feature into a plurality of third sub-features;   dividing the fifth BEV feature into a plurality of fourth sub-features;   performing processing based on a plurality of fourth expert models on the plurality of third sub-features and the plurality of fourth sub-features to obtain a plurality of fifth sub-features, wherein processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and   adding the plurality of fifth sub-features to obtain the sixth BVE feature.   
     
     
         7 . The method according to  claim 1 , wherein the point cloud data is data collected when a lidar is interfered with, and the image data is data collected when a camera is interfered with. 
     
     
         8 . A model training method, wherein the method comprises:
 obtaining image data of a target scenario or point cloud data of the target scenario;   processing the image data or the point cloud data through a to-be-trained model to obtain a sensing result of the target scenario, wherein the to-be-trained model is configured to: perform feature extraction on the image data or the point cloud data to obtain a first BEV feature of the image data or a second BEV feature of the point cloud data; perform feature fusion on the first BEV feature or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and obtain the sensing result of the target scenario based on the feature fusion result; and   training the to-be-trained model based on the sensing result of the target scenario to obtain a target model.   
     
     
         9 . The method according to  claim 8 , wherein the to-be-trained model is configured to:
 encode the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data.   
     
     
         10 . The method according to  claim 8 , wherein the to-be-trained model is configured to:
 perform, based on the preset third BVE feature, processing based on a deformable attention mechanism on the first BEV feature or the second BEV feature to obtain a fourth BEV feature;   add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature;   perform processing based on a mixture of experts model on the fifth BEV feature to obtain a sixth BVE feature;   add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and   obtain the feature fusion result based on the seventh BVE feature.   
     
     
         11 . The method according to  claim 10 , wherein the to-be-trained model is configured to:
 perform processing based on a first multilayer perceptron on the preset third BVE feature to obtain first location information of a first sub-feature of the first BEV feature or second location information of a second sub-feature of the second BEV feature;   obtain the first sub-feature or the second sub-feature from the first BEV feature or the second BEV feature based on the first location information or the second location information;   perform processing based on a second multilayer perceptron on the third BVE feature to obtain first weight information of the first sub-feature of the first BEV feature or second weight information of the second sub-feature of the second BEV feature; and   perform weighted summation on the first sub-feature or the second sub-feature based on the first weight information or the second weight information to obtain the fourth BEV feature.   
     
     
         12 . The method according to  claim 10 , wherein the to-be-trained model is configured to:
 perform processing based on a first expert model on the fifth BEV feature to obtain an eighth BEV feature;   perform processing based on a second expert model on the fifth BEV feature to obtain a ninth BEV feature; and   perform processing based on a third expert model on the eighth BEV feature and the ninth BEV feature to obtain the sixth BEV feature.   
     
     
         13 . The method according to  claim 10 , wherein the to-be-trained model is configured to:
 divide the fifth BEV feature into a plurality of third sub-features;   divide the fifth BEV feature into a plurality of fourth sub-features;   perform processing based on a plurality of fourth expert models on the plurality of third sub-features and the plurality of fourth sub-features to obtain a plurality of fifth sub-features, wherein processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and   add the plurality of fifth sub-features to obtain the sixth BVE feature.   
     
     
         14 . The method according to  claim 8 , wherein the point cloud data is data collected when a lidar is interfered with, and the image data is data collected when a camera is interfered with. 
     
     
         15 . A computer storage medium, wherein the computer storage medium stores one or more instructions, and when the one or more instructions are executed by one or more computers, the one or more computers are enabled to:
 obtain image data of a target scenario or point cloud data of the target scenario;   perform feature extraction on the image data or the point cloud data to obtain a first bird eye view (BEV) feature of the image data or a second BEV feature of the point cloud data;   perform feature fusion on the first BEV feature or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and   obtain a sensing result of the target scenario based on the feature fusion result.   
     
     
         16 . The computer storage medium according to  claim 15 , wherein performing the feature extraction on the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data comprises:
 encoding the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data.   
     
     
         17 . The computer storage medium according to  claim 15 , wherein performing the feature fusion on the first BEV feature or the second BEV feature based on the preset third BEV feature to obtain the feature fusion result comprises:
 performing, based on the preset third BVE feature, processing based on a deformable attention mechanism on the first BEV feature or the second BEV feature to obtain a fourth BEV feature;   adding the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature;   performing processing based on a mixture of experts model on the fifth BEV feature to obtain a sixth BVE feature;   adding the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and   obtaining the feature fusion result based on the seventh BVE feature.   
     
     
         18 . The computer storage medium according to  claim 17 , wherein performing, based on the preset third BVE feature, the processing based on the deformable attention mechanism on the first BEV feature or the second BEV feature to obtain the fourth BEV feature comprises:
 performing processing based on a first multilayer perceptron on the preset third BVE feature to obtain first location information of a first sub-feature of the first BEV feature or second location information of a second sub-feature of the second BEV feature;   obtaining the first sub-feature or the second sub-feature from the first BEV feature or the second BEV feature based on the first location information or the second location information;   performing processing based on a second multilayer perceptron on the third BVE feature to obtain first weight information of the first sub-feature of the first BEV feature or second weight information of the second sub-feature of the second BEV feature; and   performing weighted summation on the first sub-feature or the second sub-feature based on the first weight information or the second weight information to obtain the fourth BEV feature.   
     
     
         19 . The computer storage medium according to  claim 17 , wherein performing the processing based on the mixture of experts model on the fifth BEV feature to obtain the sixth BVE feature comprises:
 performing processing based on a first expert model on the fifth BEV feature to obtain an eighth BEV feature;   performing processing based on a second expert model on the fifth BEV feature to obtain a ninth BEV feature; and   performing processing based on a third expert model on the eighth BEV feature and the ninth BEV feature to obtain the sixth BEV feature.   
     
     
         20 . The computer storage medium according to  claim 17 , wherein performing the processing based on the mixture of experts model on the fifth BEV feature to obtain the sixth BVE feature comprises:
 dividing the fifth BEV feature into a plurality of third sub-features;   dividing the fifth BEV feature into a plurality of fourth sub-features;   performing processing based on a plurality of fourth expert models on the plurality of third sub-features and the plurality of fourth sub-features to obtain a plurality of fifth sub-features, wherein processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and   adding the plurality of fifth sub-features to obtain the sixth BVE feature.