Memory-augmented neural network system

Abstract

The present disclosure relates to a method for classifying a query information element using the similarity between the query information element and a set of support information elements. A resulting set of similarity scores is transformed using a sharpening function such that the transformed scores are decreasing as negative similarity scores increase and the transformed scores are increasing as positive similarity scores increase. A class of the query information element is determined based on the transformed similarity scores.

What is claimed is: 
     
       1. A computer-implemented method (CIM) for classification, the method comprising:
 representing a set of support information elements in a vector space by a set of hypervectors, wherein the vector space is provided such that hypervectors representing different classes of the information elements are dissimilar; 
 representing at least one query information element by a respective hypervector in the vector space, wherein the respective hypervector representation of the at least one query information element is performed by a machine learning based classifier; 
 determining a similarity score of the query information element with the set of support information elements using the hypervectors, resulting in a set of similarity scores associated with the set of support information elements respectively; 
 transforming the set of similarity scores using a sharpening function such that the transformed scores are decreasing as negative similarity scores increase and the transformed scores are increasing as positive similarity scores increase; 
 providing an estimation of a class of the query information element based on the transformed similarity scores; 
 comparing the estimated class of the query information element with a true class of the query information element; and 
 determining an estimation error based on the comparison result; 
 wherein representing a set of support information elements in a vector space, representing at least one query information elements, determining a similarity score, transforming the set of similarity scores, providing an estimation, comparing the estimated class of the query information element, and determining an estimation error based on the comparison results are repeatedly executed to train the classifier with backpropagation for optimizing a predefined loss function, wherein the loss function further includes at least one of the regularizing terms: 
 
       
         
           
             
               
                 ℒ 
                 sc 
               
               = 
               
                 
                   
                     - 
                     1 
                   
                   mn 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     mn 
                   
                   ⁢ 
                   
                     
                       ( 
                       
                         
                           
                             1 
                             d 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 j 
                                 = 
                                 1 
                               
                               d 
                             
                             ⁢ 
                             
                               ( 
                               
                                 
                                   tanh 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         aK 
                                         1 
                                       
                                       ⁡ 
                                       
                                         ( 
                                         j 
                                         ) 
                                       
                                     
                                     ) 
                                   
                                 
                                 + 
                                 
                                   1 
                                   2 
                                 
                               
                               ) 
                             
                           
                         
                         - 
                         0.5 
                       
                       ) 
                     
                     2 
                   
                 
               
             
           
         
         
           
             and 
           
         
         
           
             
               
                 
                   ℒ 
                   aux 
                 
                 = 
                 
                   
                     
                       
                         - 
                         1 
                       
                       mnd 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         mn 
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             j 
                             = 
                             1 
                           
                           d 
                         
                         ⁢ 
                         
                           tanh 
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   aK 
                                   i 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   j 
                                   ) 
                                 
                               
                               + 
                               δ 
                             
                             ) 
                           
                         
                       
                     
                   
                   - 
                   
                     tanh 
                     ⁡ 
                     
                       ( 
                       
                         
                           
                             aK 
                             i 
                           
                           ⁡ 
                           
                             ( 
                             j 
                             ) 
                           
                         
                         - 
                         δ 
                       
                       ) 
                     
                   
                 
               
               , 
             
           
         
       
       where K i  is a support hypervector of dimension d, m is the number of classes, and n is the number of support information elements per class. 
     
     
       2. The CIM of  claim 1 , wherein the machine learning based classifier is a trained classifier. 
     
     
       3. The CIM of  claim 1 , wherein the machine learning based classifier is provided in accordance with a memory augmented neural network (MANN) architecture, wherein an explicit memory is configured to store the set of hypervectors of the set of support information elements. 
     
     
       4. The CIM of  claim 3 , the memory comprising a neuromorphic memory device with a crossbar array structure that includes input lines and output lines interconnected at junctions via electronic devices. 
     
     
       5. The CIM of  claim 4 , wherein the crossbar array structure comprises a single electronic device per junction, wherein the electronic device is a memristive device. 
     
     
       6. The CIM of  claim 1 , further comprising providing the hypervectors as binary vectors or bipolar vectors. 
     
     
       7. The CIM of  claim 1 , the classifier comprising a convolutional neural network (CNN). 
     
     
       8. The CIM of  claim 1 , the information element being an image or a set of characters. 
     
     
       9. The CIM of  claim 1 , the set of support information elements having a set of classes, wherein the query information element belongs to one of the set of classes.