Image retrieval system

Abstract

In some examples, it is disclosed a method for generating an image retrieval system configured to rank a plurality of images of cargo from a dataset of images, in response to a query corresponding to an image of cargo of interest generated using penetrating radiation. The method may involve obtaining a plurality of annotated training images including cargo, each of the training images being associated with an annotation indicating a type of the cargo in the training image, and training the image retrieval system by applying a deep learning algorithm to the obtained annotated training images. The training may involve applying, to the annotated training images, a feature extraction convolutional neural network, and applying an aggregated generalized mean pooling layer associated with image spatial information.

1 . A method for generating an image retrieval system configured to rank a plurality of images of cargo from a dataset of images, in response to a query corresponding to an image of cargo of interest generated using penetrating radiation, the method comprising:
 obtaining a plurality of annotated training images comprising cargo, each of the training images being associated with an annotation indicating a type of the cargo in the training image; and   training the image retrieval system by applying a deep learning algorithm to the obtained annotated training images, the training comprising:
 applying, to the annotated training images, a feature extraction convolutional neural network, CNN, comprising a plurality of convolutional layers to generate a tensor χ of image features, and 
 applying, to the generated tensor χ, an aggregated generalized mean, AgGeM, pooling layer associated with image spatial information, the applying comprising:
 applying a generalized mean pooling layer, to generate a plurality of embedding vectors (f (p) ) p∈P ∈   K , such that:
   ( f   (p) ) p =[ f   1   (p)   , . . . ,f   K   (p) ] 
 
 
 with: 
   
       
         
           
             
               
                 f 
                 k 
                 
                   ( 
                   p 
                   ) 
                 
               
               = 
               
                 
                   ( 
                   
                     
                       1 
                       
                         
                           ❘ 
                           "\[LeftBracketingBar]" 
                         
                         
                           X 
                           k 
                         
                         
                           ❘ 
                           "\[RightBracketingBar]" 
                         
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           x 
                           ∈ 
                           
                             X 
                             k 
                           
                         
                       
                       
                         x 
                         p 
                       
                     
                   
                   ) 
                 
                 
                   1 
                   p 
                 
               
             
           
         
         
           with:
 P is a set of positive integers p representing pooling parameters of the generalized mean pooling layer, 
 the tensor χ=(χ k ) k∈{1, . . . , K}  having H×W activations for a feature map k∈{1, . . . , K}, the feature map resulting from the application of the feature extraction CNN on an training image, with H and W being respectively a height and a width of each of the feature maps, 
 K is a number of feature maps in a last convolutional layer of the feature extraction CNN, 
 x is a feature from the generated tensor χ, and 
 |χ k | is a cardinal of χ k  of the tensor χ, and 
 aggregating the generated plurality of embedding vectors (f (p) ) p∈P ∈   K  by applying weights α of a scoring layer associated with an attention mechanism to the plurality of embedding vectors (f (p) ) p∈P ∈   K , for each pooling parameter p belonging to P, the weights α being such that:
   α( f   (p) ;θ)
 
 
 with the weights α and a parameter θ being learnable by the image retrieval system to minimize an associated loss function, 
 wherein the aggregating is configured to generate a feature vector f such that:
     f =[ f   1   , . . . ,f   K ] 
   with: 
   ∀ k∈{ 1, . . . , K},f   k =Σ p∈P α( f   (p) ;θ)· f   k   (p) .
 
 
 
         
       
     
     
         2 . The method of  claim 1 , wherein training the image retrieval system further comprises:
 applying, to the generated tensor χ, an orderless feature pooling layer associated with image texture information,   wherein applying the orderless feature pooling layer comprises using a Gaussian mixture model, GMM, to generate orderless image descriptors of the image features, the applying comprising:
 mapping the image features x i , i∈{1, . . . , d} of the tensor χ=(χ k ) k∈{1, . . . , K}  to a group of clusters of a Gaussian Mixture Model, with diagonal variances Σ k  such that: 
   
       
         
           
             
               
                 ∀ 
                 
                   k 
                   ∈ 
                   
                     { 
                     
                       1 
                       , 
                       … 
                           
                       , 
                       K 
                     
                     } 
                   
                 
               
               , 
               
                 
                   ∑ 
                   k 
                 
                 
                   = 
                   
                     
                       1 
                       
                         α 
                         k 
                       
                     
                     · 
                     
                       
                         I 
                         d 
                       
                       . 
                     
                   
                 
               
             
           
         
         
           with I d  being the d×d identity matrix, 
           d=H×W being a dimension of the cluster k of (χ k ) k∈{1, . . . , K} , and 
           α k  being a smoothing factor that represents the inverse of the variance Σ k  in the k th  cluster, α k  being learnable by the image retrieval system to minimize the loss function, 
           applying a soft assignment algorithm by assigning weights ā k  associated with the feature x i , i∈{1, . . . , d}, to the cluster k of centre c k , such that: 
         
       
       
         
           
             
               
                 
                   
                     a 
                     ¯ 
                   
                   k 
                 
                 ( 
                 
                   x 
                   i 
                 
                 ) 
               
               = 
               
                 
                   e 
                   
                     
                       - 
                       
                         α 
                         k 
                       
                     
                     ⁢ 
                     
                       
                          
                         
                           
                             x 
                             i 
                           
                           - 
                           
                             c 
                             k 
                           
                         
                          
                       
                       2 
                     
                   
                 
                 
                   
                     Σ 
                     
                       
                         k 
                         ′ 
                       
                       = 
                       1 
                     
                     K 
                   
                   ⁢ 
                   
                     e 
                     
                       
                         - 
                         
                           α 
                           
                             k 
                             ′ 
                           
                         
                       
                       ⁢ 
                       
                         
                            
                           
                             
                               x 
                               i 
                             
                             - 
                             
                               c 
                               
                                 k 
                                 ′ 
                               
                             
                           
                            
                         
                         2 
                       
                     
                   
                 
               
             
           
         
         
           with c k  being a vector representing the centre of the k-th cluster, c k  being learnable by the image retrieval system to minimize the loss function, 
           c k , is the same as c k  for index k=k′ ranging from 1 to K, 
           M being a hyper-parameter representing a number of clusters to include in the group of the plurality of clusters of the Gaussian Mixture Model, and which can be chosen by an operator training the system, and 
           generating a matrix V of descriptors, such that:
     V =( ā   k ( x   i )) k∈{1, . . . ,M};i∈{1, . . . ,d} ∈   d×M .
 
 
         
       
     
     
         3 . The method of  claim 2 , wherein applying the aggregated generalized mean pooling layer and applying the orderless feature pooling layer are performed in parallel. 
     
     
         4 . The method of  claim 3 , wherein the training further comprises applying a bilinear model layer to a combined output associated with the aggregated generalized mean pooling layer and the orderless feature pooling layer, wherein the bilinear model layer is associated with a bilinear function Y ts  such that: 
       
         
           
             
               
                 Y 
                 ts 
               
               = 
               
                 
                   ∑ 
                   
                     i 
                     = 
                     1 
                   
                   I 
                 
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     J 
                   
                   
                     
                       ω 
                       ij 
                     
                     ⁢ 
                     
                       a 
                       i 
                       t 
                     
                     ⁢ 
                     
                       b 
                       j 
                       s 
                     
                   
                 
               
             
           
         
         with a t  being a vector with a dimension I and associated with an output of the orderless feature pooling layer, 
         b s  being a vector with a dimension J and associated with an output of the aggregated generalized mean pooling layer, and 
         ωij a weight configured to balance interaction between a t  and b s , ωij being learnable by the image retrieval system to minimize the loss function. 
       
     
     
         5 . The method of  claim 4 , wherein the vector a t  is obtained by applying an l 2  normalization layer and/or a fully connected layer to the matrix V of descriptors. 
     
     
         6 . The method of  claim 4 , wherein the vector b s  is obtained by applying a normalization layer, such as an l 2  normalization layer and/or a batch normalization layer, and/or a fully connected layer to the feature vector f. 
     
     
         7 . The method of  claim 3 , further comprising applying, to a combined output associated with the aggregated generalized mean pooling layer and the orderless feature pooling layer, and in order to obtain a final image representation of an image, at least one of:
 at least one normalization layer, such as an l 2  normalization layer, and/or   a fully connected layer.   
     
     
         8 . The method of  claim 1 , wherein each of the training images is further associated with a code of the Harmonised Commodity Description and Coding System, HS, the HS comprising hierarchical sections and chapters corresponding to the type of the cargo in the training image, and
 wherein training the image retrieval system further comprises taking into account, in the loss function associated with the image retrieval system, the hierarchical sections and chapters of the HS.   
     
     
         9 . The method of  claim 8 , wherein the loss function of the image retrieval system is such that:
     =Σ i=1   N  max(∥ϕ θ ( I   i   anchor )−ϕ θ ( I   i   similar )∥ 2   2 −∥ϕ θ ( I   i   anchor )−ϕ θ ( I   i   different )∥ 2   2 +β,0)+λΣ i=1   N  max(∥ψ η ( h   i   anchor )−ψ η ( h   i   similar )∥ 2   2 −∥ψ η ( h   i   anchor )−ψ η ( h   i   different )∥ 2   2 +δ,0)
   With I anchor , I similar  and I different  being three images, I anchor , I similar  and I different  being such that I similar  comprises cargo similar to the cargo of an anchor image I anchor , and I different  comprises cargo which is different from the cargo of the anchor image I anchor ,   ∥.∥ 2  being the Euclidean l 2  norm in    d , and d is a dimension of an image vectorial representation, and d may be chosen by an operator training the system,   N being the number of images in the dataset of images,   β being a hyper-parameter that controls the margin between similar images and different images, and which can be chosen by an operator training the system.   h i   anchor  an HS code of a training image corresponding to a query,   h i   similar  an HS code of a training image sharing a same hierarchical section and/or hierarchical chapter with the training image corresponding to the query,   h i   different  an HS code of a training image having a different hierarchical section and/or hierarchical chapter from the training image corresponding to the query,   ψ η  being a parametric function associated with the image retrieval system, η being a parameter learnable by the image retrieval system to minimize the loss function  ,   λ being a parameter controlling an importance given to the hierarchical structure of the HS-codes during the training, and   δ being a hyper-parameter that controls a margin between similar and different HS-codes, and which can be chosen by the operator training the system.   
     
     
         10 . (canceled) 
     
     
         11 . (canceled) 
     
     
         12 . The method of  claim 1 , wherein each of the training images is further associated with textual information corresponding to the type of cargo in the training image. 
     
     
         13 . (canceled) 
     
     
         14 . The method of  claim 1 , wherein the image retrieval system is configured to detect at least one image in the dataset of images comprising a cargo most similar to the cargo of interest in the inspection image, a similarity between cargos being based on a Euclidean distance between the cargos, the Euclidean distance being taken into account in a loss function of the image retrieval system applied to the training images. 
     
     
         15 . The method according to  claim 1 , wherein the method is performed at a computer system separate from a device configured to inspect containers. 
     
     
         16 . A method comprising:
 obtaining an inspection image of cargo of interest generated using penetrating radiation, the inspection image corresponding to a query;   applying, to the inspection image, an image retrieval system generated by the method according to  claim 1 ; and   ranking a plurality of images of cargo from a dataset of images, based on the applying.   
     
     
         17 . The method of  claim 16 , wherein ranking the plurality of images comprises outputting a ranked list of images comprising cargo corresponding to the cargo of interest in the inspection image. 
     
     
         18 . The method of  claim 16 , wherein ranking the plurality of images further comprises outputting an at least partial code of the Harmonised Commodity Description and Coding System, HS, the HS comprising hierarchical sections and chapters corresponding to a type of cargo in each of the plurality of ranked images. 
     
     
         19 . The method of  claim 16 , wherein ranking the plurality of images further comprises outputting at least partial textual information corresponding to the type of cargo in each of the plurality of ranked images. 
     
     
         20 . A method of producing a device configured to rank a plurality of images of cargo from a dataset of images generated using penetrating radiation, the method comprising:
 obtaining an image retrieval system generated by the method according to  claim 1 ; and   storing the obtained image retrieval system in a memory of the device.   
     
     
         21 . The method according to  claim 20 , wherein the storing comprises transmitting the generated image retrieval system to the device via a network, the device receiving and storing the image retrieval system, or
 wherein the image retrieval system is generated, stored and/or transmitted in the form of one or more of:   a data representation of the image retrieval system;   executable code for applying the image retrieval system to one or more inspection images.   
     
     
         22 . (canceled) 
     
     
         23 . A device configured to rank a plurality of images of cargo from a dataset of images generated using penetrating radiation, the device comprising a memory storing an image retrieval system generated by the method according to  claim 1 . 
     
     
         24 . (canceled) 
     
     
         25 . A computer program or a computer program product comprising instructions which, when executed by a processor, enable the processor to perform the method according to  claim 1 .