Robust auto-associative memory with recurrent neural network

Abstract

Computer systems and computer-implemented methods recursively train a content-addressable auto-associative memory such that: (i) the content addressable auto-associative memory system is trained to produce an output pattern for each of the input examples; and (ii) a quantity of the learned parameters for the content-addressable auto-associative memory is equal to the number of input variables times a quantity that is independent of the number of input variables. The quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned.

1 . A computer system comprising:
 a set of processor cores; and   computer memory in communication with the set of processor cores, wherein the computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to recursively train a content-addressable auto-associative memory system with a plurality of learned parameters and with a plurality of input examples, wherein each input example is represented by a plurality of input variables, such that:
 the content addressable auto-associative memory system is trained to produce an output pattern for each of the input examples; and 
 a quantity of the learned parameters for the content-addressable auto-associative memory is equal to the number of input variables times a quantity that is independent of the number of input variables. 
   
     
     
         2 . The computer system of  claim 1 , wherein the software causes the set of processor cores to train the content-addressable auto-associative memory system such that the quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned. 
     
     
         3 . The computer system of  claim 1 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the content-addressable auto-associative memory system by back propagating partial derivatives of a loss function through the content-addressable auto-associative memory system. 
     
     
         4 . The computer system of  claim 1 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the content-addressable auto-associative memory system by, for each input example:
 randomly transforming the input example; and   recursively providing the randomly transformed input example to the content-addressable auto-associative memory system for training, until an output of the content-addressable auto-associative memory system converges to the input example.   
     
     
         5 . The computer system of  claim 4 , wherein the computer memory stores software that when executed by the set of processor cores further causes the set of processor cores to train the content-addressable auto-associative memory system with negative input examples. 
     
     
         6 . The computer system of  claim 5 , wherein the negative input examples comprise input examples where the output of the content-addressable auto-associative memory system, in operation, does not converge to an input example. 
     
     
         7 - 11 . (canceled) 
     
     
         12 . A computer system comprising:
 a set of processor cores; and   computer memory in communication with the set of processor cores, wherein the computer memory stores software that when executed by the set of processor cores, causes the set of processor cores to recursively train a recurrent neural network with a plurality of input examples, such that:
 the recurrent neural network comprises a deep neural network that comprises N+1 layers, numbered 0, . . . , N, wherein N>3, and wherein layer 0 is an input layer and layer N is an output layer of the recurrent neural network, and wherein layers 1 to N−1 are between the input layer and the output layer; 
   the recurrent neural network is trained to produce an output pattern for each of the input examples;   a target for the output pattern for each input example is the input example; and   the recurrent neural network comprises a plurality of directed arcs, wherein at least some of the directed arcs are between a node in one layer of the recurrent neural network and a node in another layer of the recurrent neural network.   
     
     
         13 - 15 . (canceled) 
     
     
         16 . The computer system of  claim 12 , wherein the software causes the set of processor cores to train the recurrent neural network such that the only directed arcs in the recurrent neural network from a higher numbered layer to a lower numbered layer are from a node in the output layer N to a node in the input layer 0. 
     
     
         17 . The computer system of  claim 16 , wherein:
 the output layer of the recurrent neural network comprise a plurality of output layer nodes;   the input layer of the recurrent neural network comprise a plurality of input layer nodes;   the quantity of input layer nodes equals the quantity of output layer nodes, such that each output layer node has one and only one corresponding input layer node; and   the only directed arcs in the recurrent neural network that are from a higher numbered layer to a lower numbered layer are directed arcs from the output layer to the input layer, wherein there is a directed arc from each output layer node to its associated input layer node.   
     
     
         18 . The computer system of  claim 12 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by back propagating partial derivatives of a loss function through the recurrent neural network. 
     
     
         19 . The computer system of claim  13 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by back propagating partial derivatives of a loss function through the recurrent neural network. 
     
     
         20 . The computer system of  claim 12 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by back propagating partial derivatives of a loss function through the recurrent neural network. 
     
     
         21 . The computer system of  claim 17 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to train the recurrent neural network by, for each input example:
 randomly transforming the input example; and   recursively providing the randomly transformed input example to the content-addressable auto-associative memory system for training, until an output of the content-addressable auto-associative memory system converges to the input example.   
     
     
         22 . The computer system of  claim 21 , wherein the computer memory stores software that when executed by the set of processor cores causes the set of processor cores to transform an input example by performing a distortion on the input example that comprises a distortion selected from the group consisting of:
 translating the input example;   rotating the input example;   linearly transforming the input example;   degrading the input example; and   subsampling the input example.   
     
     
         23 . The computer system of  claim 21 , wherein the transformations of the input examples are controlled by one or more hyperparameters. 
     
     
         24 . The computer system of  claim 23 , further comprising:
 a second set of processor cores; and   second computer memory in communication with the second set of processor cores, wherein the second computer memory stores software that when executed by the second set of processor cores causes the second set of processor cores to implement a machine-learning learning coach that learns, through machine learning, values for the one or more hyperparameters.   
     
     
         25 . The computer system of  claim 21 , wherein the computer memory stores software that when executed by the set of processor cores further causes the set of processor cores to train the recurrent neural network with negative input examples. 
     
     
         26 . The computer system of  claim 25 , wherein the negative input examples comprise input examples where the output of the recurrent neural network, in operation, does not converge to an input example. 
     
     
         27 . The computer system of  claim 12 , wherein at least some of the input examples are labeled examples that have, for each such input example, a classification category label such that the recurrent neural network is trained to act as a classifier. 
     
     
         28 . The computer system of  claim 27 , wherein the classification category labels comprise error-correcting encoding. 
     
     
         29 . The computer system of  claim 12 , wherein:
 the input examples are digital images; and   the recurrent neural network is trained to, in operation, retrieve one of the digital images in response to receiving as input a portion of the digital image.   
     
     
         30 . The computer system of  claim 12 , wherein:
 the input examples are audio files; and   the recurrent neural network is trained to, in operation, retrieve one of the audio files in response to receiving as input a portion of the audio file.   
     
     
         31 . The computer system of  claim 12 , wherein:
 the input examples are document files; and   the recurrent neural network is trained to, in operation, retrieve one of the document files in response to receiving as input a portion of the document file.   
     
     
         32 . A method comprising:
 recursively train, by a computer system that comprises a set of processor cores, a content-addressable auto-associative memory system with a plurality of learned parameters and with a plurality of input examples, wherein each input example is represented by a plurality of input variables, such that:
 the content addressable auto-associative memory system is trained to produce an output pattern for each of the input examples; and 
 a quantity of the learned parameters for the content-addressable auto-associative memory is equal to the number of input variables times a quantity that is independent of the number of input variables. 
   
     
     
         33 . The method of  claim 32 , wherein training the content-addressable auto-associative memory system comprises raining the content-addressable auto-associative memory system such that the quantity of learned parameters for the content-addressable auto-associative memory system can be varied based on the number of input examples to be learned. 
     
     
         34 . The method of  claim 32 , wherein training the content-addressable auto-associative memory system comprises back propagating partial derivatives of a loss function through the content-addressable auto-associative memory system. 
     
     
         35 . The method of  claim 32 , wherein training the content-addressable auto-associative memory system comprises, for each input example:
 randomly transforming the input example; and   recursively providing the randomly transformed input example to the content-addressable auto-associative memory system for training, until an output of the content-addressable auto-associative memory system converges to the input example.   
     
     
         36 . The method of  claim 35 , wherein training the content-addressable auto-associative memory system comprises training the content-addressable auto-associative memory system with negative input examples. 
     
     
         37 . The method of  claim 36 , wherein the negative input examples comprise input examples where the output of the content-addressable auto-associative memory system, in operation, does not converge to an input example. 
     
     
         38 . The method of  claim 35 , wherein at least some of the input examples are labeled examples that have, for each such input example, a classification category label such that the r content-addressable auto-associative memory system is trained to act as a classifier. 
     
     
         39 . The method of  claim 38 , wherein the classification category labels comprise error-correcting encoding. 
     
     
         40 . The method of  claim 32 , wherein:
 the input examples are digital images; and   the content-addressable auto-associative memory system is trained to, in operation, retrieve one of the digital images in response to receiving as input a portion of the digital image.   
     
     
         41 . The method of  claim 32 , wherein:
 the input examples are audio files; and   the content-addressable auto-associative memory system is trained to, in operation, retrieve one of the audio files in response to receiving as input a portion of the audio file.   
     
     
         42 . The method of  claim 32 , wherein:
 the input examples are document files; and   the content-addressable auto-associative memory system is trained to, in operation, retrieve one of the document files in response to receiving as input a portion of the document file.   
     
     
         43 . A method comprising:
 training, recursively, by a computer system that comprises a set of processor cores, a recurrent neural network with a plurality of input examples, such that:
 the recurrent neural network comprises a deep neural network that comprises N+1 layers, numbered 0, . . . , N, wherein N>3, and wherein layer 0 is an input layer and layer N is an output layer of the recurrent neural network, and wherein layers 1 to N−1 are between the input layer and the output layer; 
 the recurrent neural network is trained to produce an output pattern for each of the input examples; 
 a target for the output pattern for each input example is the input example; and 
 the recurrent neural network comprises a plurality of directed arcs, wherein at least some of the directed arcs are between a node in one layer of the recurrent neural network and a node in another layer of the recurrent neural network. 
   
     
     
         44 . A method comprising:
 training, recursively, by a computer system that comprises a set of processor cores, a recurrent neural network with a plurality of input examples, such that:
 the recurrent neural network is trained to produce an output pattern for each of the input examples; and 
 a quantity of learned parameters for the recurrent neural network is equal to the number of input variables times a quantity that is independent of the number of input variables. 
   
     
     
         45 - 46 . (canceled) 
     
     
         47 . The method of  claim 43 , wherein training the recurrent neural network comprises training the recurrent neural network such that the only directed arcs in the recurrent neural network from a higher numbered layer to a lower numbered layer are from a node in the output layer N to a node in the input layer 0. 
     
     
         48 . The method of  claim 47 , wherein:
 the output layer of the recurrent neural network comprise a plurality of output layer nodes;   the input layer of the recurrent neural network comprise a plurality of input layer nodes;   the quantity of input layer nodes equals the quantity of output layer nodes, such that each output layer node has one and only one corresponding input layer node; and   the only directed arcs in the recurrent neural network that are from a higher numbered layer to a lower numbered layer are directed arcs from the output layer to the input layer, wherein there is a directed arc from each output layer node to its associated input layer node.   
     
     
         49 . The method of  claim 43 , wherein training the recurrent neural network comprises back propagating partial derivatives of a loss function through the recurrent neural network. 
     
     
         50 . The method of  claim 44 , wherein training the recurrent neural network comprises back propagating partial derivatives of a loss function through the recurrent neural network. 
     
     
         51 . (canceled) 
     
     
         52 . The method of  claim 48 , wherein training the recurrent neural network comprises, for each input example:
 randomly transforming the input example; and   recursively providing the randomly transformed input example to the content-addressable auto-associative memory system for training, until an output of the content-addressable auto-associative memory system converges to the input example.   
     
     
         53 . The method of  claim 52 , wherein transforming an input example by performing a distortion on the input example that comprises a distortion selected from the group consisting of:
 translating the input example;   rotating the input example;   linearly transforming the input example;   degrading the input example; and   subsampling the input example.   
     
     
         54 - 55 . (canceled) 
     
     
         56 . The method of  claim 52 , wherein training the recurrent neural network comprises training the recurrent neural network with negative input examples. 
     
     
         57 - 62 . (canceled)