Methods and apparatuses for vehicle appearance feature recognition, methods and apparatuses for vehicle retrieval, storage medium, and electronic devices

Abstract

The method for vehicle appearance feature recognition includes: multiple region segmentation results of a target vehicle are obtained from an image to be recognized; global feature data and multiple pieces of region feature data are extracted from the image to be recognized based on the multiple region segmentation results; and the global feature data and the multiple pieces of region feature data are fused to obtain appearance feature data of the target vehicle.

1 . A method for vehicle appearance feature recognition, comprising:
 obtaining a plurality of region segmentation results of a target vehicle from an image to be recognized;   extracting global feature data and a plurality of pieces of region feature data from the image to be recognized based on the plurality of region segmentation results; and   fusing the global feature data and the plurality of pieces of region feature data to obtain appearance feature data of the target vehicle,   wherein the fusing the global feature data and the plurality of pieces of region feature data comprises:   fusing the global feature data and the plurality of pieces of region feature data of the target vehicle by means of a third neural network for feature fusion,   wherein the third neural network has a first fully connected layer, a third computing layer, and a second fully connected layer which are connected to an output end of a second neural network,   wherein the fusing the global feature data and the plurality of pieces of region feature data of the target vehicle by means of a third neural network for feature fusion comprises:   obtaining weight values of a plurality of first region feature vectors by means of the first fully connected layer;   respectively weighting the plurality of first region feature vectors by means of the third computing layer according to the weight values to obtain corresponding a plurality of second region feature vectors; and   performing a mapping operation on the plurality of second region feature vectors and a global feature vector by means of the second fully connected layer to obtain an appearance feature vector of the target vehicle,   wherein the obtaining weight values of the a plurality of first region feature vectors by means of the first fully connected layer comprises:   performing a stitching operation on the plurality of first region feature vectors to obtain a stitched first region feature vector;   performing a mapping operation on the stitched first region feature vector by means of the first fully connected layer to obtain a set of scalars corresponding to the plurality of first region feature vectors; and   performing a normalization operation on scalars in the set of scalars to obtain the weight values of the plurality of first region feature vectors.   
     
     
         2 . The method according to  claim 1 , wherein the plurality of region segmentation results respectively corresponds to regions of different orientations of the target vehicle,
 wherein the plurality of region segmentation results comprise segmentation results of a front side, a rear side, a left side, and a right side of the target vehicle.   
     
     
         3 . The method according to  claim 1 , wherein the obtaining a plurality of region segmentation results of a target vehicle from an image to be recognized comprises:
 obtaining the plurality of region segmentation results of the target vehicle from the image to be recognized by means of a first neural network for region extraction.   
     
     
         4 . The method according to  claim 3 , wherein the first neural network has a first feature extraction layer and a first computing layer connected to a tail end of the first feature extraction layer,
 wherein the obtaining a plurality of region segmentation results of the target vehicle from the image to be recognized by means of a first neural network for region extraction comprises:   performing feature extraction on the image to be recognized by means of the first feature extraction layer to obtain a plurality of key points of the target vehicle; and   classifying the plurality of key points by means of the first computing layer to obtain a plurality of key point clusters, and respectively fusing feature maps of key points in the plurality of key point clusters, to obtain region segmentation results corresponding to the plurality of key point clusters.   
     
     
         5 . The method according to  claim 1 , wherein the extracting global feature data and a plurality of pieces of region feature data from the image to be recognized based on the plurality of region segmentation results comprises:
 extracting global feature data and the plurality of pieces of region feature data of the target vehicle from the image to be recognized by means of a second neural network for feature extraction based on the plurality of region segmentation results.   
     
     
         6 . The method according to  claim 5 , wherein the second neural network has a first processing subnet and a plurality of second processing subnets separately connected to an output end of the first processing subnet,
 wherein the first processing subnet has a second feature extraction layer, a first inception module, and a first pooling layer, and the second processing subnet has a second computing layer, a second inception module, and a second pooling layer which are connected to the output end of the first processing subnet.   
     
     
         7 . The method according to  claim 6 , wherein the extracting global feature data and the plurality of pieces of region feature data of the target vehicle from the image to be recognized by means of a second neural network for feature extraction based on the plurality of region segmentation results comprises:
 performing a convolution operation and a pooling operation on the image to be recognized by means of the second feature extraction layer to obtain a global feature map of the target vehicle;   performing a convolution operation and a pooling operation on the global feature map by means of the first inception module to obtain a first feature map set of the target vehicle; and   performing a pooling operation on feature maps in the first feature map set by means of the first pooling layer to obtain a global feature vector of the target vehicle.   
     
     
         8 . The method according to  claim 7 , wherein the extracting global feature data and the plurality of pieces of region feature data of the target vehicle from the image to be recognized by means of a second neural network for feature extraction based on the plurality of region segmentation results further comprises:
 performing point multiplication on the plurality of region segmentation results and the global feature map separately by means of the second computing layer, to obtain local feature maps respectively corresponding to the plurality of region segmentation results;   performing a convolution operation and a pooling operation on the local feature maps of the plurality of region segmentation results by means of the second inception module to obtain a second feature map set corresponding to the plurality of region segmentation results; and   performing a pooling operation on the second feature map set corresponding to the plurality of region segmentation results by means of the second pooling layer to obtain first region feature vectors corresponding to the plurality of region segmentation results.   
     
     
         9 . The method according to  claim 8 , wherein before the performing point multiplication on the plurality of region segmentation results and the global feature map separately by means of the second computing layer, the method further comprises:
 respectively scaling the plurality of region segmentation results to the same size as a size the global feature map by means of the second computing layer.   
     
     
         10 . The method according to  claim 4 , wherein the first feature extraction layer is an hourglass network structure. 
     
     
         11 . A method for vehicle retrieval, comprising:
 obtaining appearance feature data of a target vehicle in an image to be retrieved by means of the method according to  claim 1 ; and   searching a candidate vehicle image library for a target candidate vehicle image matching the appearance feature data.   
     
     
         12 . The method according to  claim 11 , wherein the searching a candidate vehicle image library for a target candidate vehicle image matching the appearance feature data comprises:
 determining cosine distances between the appearance feature vector of the target vehicle and appearance feature vectors of vehicles in a plurality of vehicle images to be selected in the candidate vehicle image library, separately; and   determining, according to the cosine distances, a target candidate vehicle image matching the target vehicle.   
     
     
         13 . The method according to  claim 12 , further comprising:
 obtaining at least one of a photographed time or a photographing position of the image to be retrieved and at least one of a photographed time or photographing positions of a plurality of vehicle images to be selected;   determining temporal-spatial distances between the target vehicle and vehicles in the plurality of vehicle images to be selected according to the at least one of the photographed time or the photographing position of the image to be retrieved and the at least one of the photographed time or the photographing positions of the plurality of vehicle images to be selected; and   determining, according to the cosine distances and the temporal-spatial distances, a target candidate vehicle image matching the target vehicle, in the candidate vehicle image library.   
     
     
         14 . The method according to  claim 13 , wherein the determining, according to the cosine distances and the temporal-spatial distances, a target candidate vehicle image matching the target vehicle, in the candidate vehicle image library comprises:
 obtaining the plurality of vehicle images to be selected from the candidate vehicle image library according to the cosine distances;   determining a temporal-spatial matching probability of the vehicle image to be selected and the target vehicle based on the photographed time and the photographing position of the vehicle image to be selected, respectively; and   determining, according to the cosine distances and the temporal-spatial matching probability, a target candidate vehicle image matching the target vehicle.   
     
     
         15 . An apparatus for vehicle appearance feature recognition, comprising:
 a memory storing processor-executable instructions; and   a processor arranged to execute the processor-executable instructions to perform steps of:   obtaining a plurality of region segmentation results of a target vehicle from an image to be recognized;   extracting global feature data and a plurality of pieces of region feature data from the image to be recognized based on the plurality of region segmentation results; and   fusing the global feature data and the plurality of pieces of region feature data to obtain appearance feature data of the target vehicle,   wherein the fusing the global feature data and the plurality of pieces of region feature data comprises:   fusing the global feature data and the plurality of pieces of region feature data of the target vehicle by means of a third neural network for feature fusion,   wherein the third neural network has a first fully connected layer, a third computing layer, and a second fully connected layer which are connected to an output end of a second neural network,   wherein the fusing the global feature data and the plurality of pieces of region feature data of the target vehicle by means of a third neural network for feature fusion comprises:   obtaining weight values of a plurality of first region feature vectors by means of the first fully connected layer;   respectively weighting the plurality of first region feature vectors by means of the third computing layer according to the weight values to obtain corresponding a plurality of second region feature vectors; and   performing a mapping operation on the plurality of second region feature vectors and a global feature vector by means of the second fully connected layer to obtain an appearance feature vector of the target vehicle,   wherein the obtaining weight values of a plurality of first region feature vectors by means of the first fully connected layer comprises:   performing a stitching operation on the plurality of first region feature vectors to obtain a stitched first region feature vector;   performing a mapping operation on the stitched first region feature vector by means of the first fully connected layer to obtain a set of scalars corresponding to the plurality of first region feature vectors; and   performing a normalization operation on scalars in the set of scalars to obtain the weight values of the plurality of first region feature vectors.   
     
     
         16 . The apparatus according to  claim 15 , wherein the plurality of region segmentation results respectively corresponds to regions of different orientations of the target vehicle,
 wherein the plurality of region segmentation results comprise segmentation results of a front side, a rear side, a left side, and a right side of the target vehicle.   
     
     
         17 . The apparatus according to  claim 15 , wherein the obtaining a plurality of region segmentation results of a target vehicle from an image to be recognized comprises:
 obtaining the plurality of region segmentation results of the target vehicle from the image to be recognized by means of a first neural network for region extraction.   
     
     
         18 . The apparatus according to  claim 17 , wherein the first neural network has a first feature extraction layer and a first computing layer connected to a tail end of the first feature extraction layer,
 wherein the obtaining a plurality of region segmentation results of the target vehicle from the image to be recognized by means of a first neural network for region extraction comprises:   performing feature extraction on the image to be recognized by means of the first feature extraction layer to obtain a plurality of key points of the target vehicle; and   classifying the plurality of key points by means of the first computing layer to obtain a plurality of key point clusters, and respectively fusing feature maps of key points in the plurality of key point clusters, to obtain region segmentation results corresponding to the plurality of key point clusters.   
     
     
         19 . An apparatus for vehicle retrieval, comprising:
 a memory storing processor-executable instructions; and   a processor arranged to execute the processor-executable instructions to perform steps of:   obtaining appearance feature data of a target vehicle in an image to be retrieved by means of the method according to  claim 1 ; and   searching a candidate vehicle image library for a target candidate vehicle image matching the appearance feature data.   
     
     
         20 . A non-transitory computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to implement steps of a method for vehicle appearance feature recognition, the method comprising:
 obtaining a plurality of region segmentation results of a target vehicle from an image to be recognized;   extracting global feature data and a plurality of pieces of region feature data from the image to be recognized based on the plurality of region segmentation results; and   fusing the global feature data and the plurality of pieces of region feature data to obtain appearance feature data of the target vehicle,   wherein the fusing the global feature data and the plurality of pieces of region feature data comprises:   fusing the global feature data and the plurality of pieces of region feature data of the target vehicle by means of a third neural network for feature fusion,   wherein the third neural network has a first fully connected layer, a third computing layer, and a second fully connected layer which are connected to an output end of a second neural network,   wherein the fusing the global feature data and the plurality of pieces of region feature data of the target vehicle by means of a third neural network for feature fusion comprises:   obtaining weight values of a plurality of first region feature vectors by means of the first fully connected layer;   respectively weighting the plurality of first region feature vectors by means of the third computing layer according to the weight values to obtain corresponding a plurality of second region feature vectors; and   performing a mapping operation on the plurality of second region feature vectors and a global feature vector by means of the second fully connected layer to obtain an appearance feature vector of the target vehicle,   wherein the obtaining weight values of a plurality of first region feature vectors by means of the first fully connected layer comprises:   performing a stitching operation on the plurality of first region feature vectors to obtain a stitched first region feature vector;   performing a mapping operation on the stitched first region feature vector by means of the first fully connected layer to obtain a set of scalars corresponding to the plurality of first region feature vectors; and   performing a normalization operation on scalars in the set of scalars to obtain the weight values of the plurality of first region feature vectors.