Fixed point integer implementations for neural networks

Abstract

Techniques related to implementing neural networks for speech recognition systems are discussed. Such techniques may include processing a node of the neural network by determining a score for the node as a product of weights and inputs such that the weights are fixed point integer values, applying a correction to the score based on a correction value associated with at least one of the weights, and generating an output from the node based on the corrected score.

What is claimed is: 
     
         1 . A system to process a node of a neural network comprising:
 a memory to store weights associated with the node of the neural network; and   one or more processors coupled to the memory, the one or more processors to:
 determine a score for the node of the neural network based on inputs to the node and the weights associated with the node, wherein the weights comprise a subset of first weights having associated correction values and a subset of second weights not having associated correction values; 
 apply a plurality of corrections to the score responsive to the correction values associated with the subset of first weights, each correction comprising a product of the associated correction value and a corresponding input of the inputs to the node; and 
 generate an output from the node based on the corrected score. 
   
     
     
         2 . The system of  claim 1 , wherein the weights comprise fixed point integer values converted from floating point values and the correction values of the subset of first weights are associated with fixed point integer value weights having a non-zero most significant bit. 
     
     
         3 . The system of  claim 1 , wherein the corrected score comprises a 32 bit fixed point integer value. 
     
     
         4 . The system of  claim 1 , wherein the weights have an associated scaling factor, the neural network comprises a neural network layer including the node, and the scaling factor is a maximum scaling factor value that provides a corrections count for the neural network layer that is less than a predetermined corrections count limit. 
     
     
         5 . The system of  claim 1 , wherein the node comprises a hidden layer node and the one or more processors to generate the output from the node based on the corrected score comprises the one or more processors to apply an activation function to the corrected score to generate the output. 
     
     
         6 . The system of  claim 1 , wherein the neural network comprises a speech recognition neural network, the one or more processors to:
 convert received speech to a speech recording;   extract feature vectors associated with time windows of the speech recording;   provide the feature vectors as input to the neural network;   generate classification scores from the speech recognition neural network based at least in part on the output from the node; and   determine a sequence of textual elements based on the classification scores.   
     
     
         7 . The system of  claim 1 , the one or more processors to:
 modify the score, prior to said applying the plurality of corrections, based on a bias associated with the node.   
     
     
         8 . The system of  claim 1 , wherein the score for the node comprises a sum of products of the inputs to the node and the weights associated with the node. 
     
     
         9 . A method for processing a node of a neural network comprising:
 determining a score for the node of the neural network based on inputs to the node and weights associated with the node, wherein the weights comprise a subset of first weights having associated correction values and a subset of second weights not having associated correction values;   applying a plurality of corrections to the score responsive to the correction values associated with the subset of first weights, each correction comprising a product of the associated correction value and a corresponding input of the inputs to the node; and   generating an output from the node based on the corrected score.   
     
     
         10 . The method of  claim 9 , wherein the weights comprise fixed point integer values converted from floating point values and the correction values of the subset of first weights are associated with fixed point integer value weights having a non-zero most significant bit. 
     
     
         11 . The method of  claim 9 , wherein the corrected score comprises a 32 bit fixed point integer value. 
     
     
         12 . The method of  claim 9 , wherein the weights have an associated scaling factor, the neural network comprises a neural network layer including the node, and the scaling factor is a maximum scaling factor value that provides a corrections count for the neural network layer that is less than a predetermined corrections count limit. 
     
     
         13 . The method of  claim 9 , wherein the node comprises a hidden layer node and generating the output from the node based on the corrected score comprises applying an activation function to the corrected score to generate the output. 
     
     
         14 . The method of  claim 9 , wherein the neural network comprises a speech recognition neural network, the method further comprising:
 converting received speech to a speech recording;   extracting feature vectors associated with time windows of the speech recording;   providing the feature vectors as input to the neural network;   generating classification scores from the speech recognition neural network based at least in part on the output from the node; and   determining a sequence of textual elements based on the classification scores.   
     
     
         15 . At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to process a node of a neural network by:
 determining a score for the node of the neural network based on inputs to the node and weights associated with the node, wherein the weights comprise a subset of first weights having associated correction values and a subset of second weights not having associated correction values;   applying a plurality of corrections to the score responsive to the correction values associated with the subset of first weights, each correction comprising a product of the associated correction value and a corresponding input of the inputs to the node; and   generating an output from the node based on the corrected score.   
     
     
         16 . The non-transitory machine readable medium of  claim 15 , wherein the weights comprise fixed point integer values converted from floating point values and the correction values of the subset of first weights are associated with fixed point integer value weights having a non-zero most significant bit. 
     
     
         17 . The non-transitory machine readable medium of  claim 15 , wherein the corrected score comprises a 32 bit fixed point integer value. 
     
     
         18 . The non-transitory machine readable medium of  claim 15 , wherein the weights have an associated scaling factor, the neural network comprises a neural network layer including the node, and the scaling factor is a maximum scaling factor value that provides a corrections count for the neural network layer that is less than a predetermined corrections count limit. 
     
     
         19 . The non-transitory machine readable medium of  claim 15 , wherein the node comprises a hidden layer node and generating the output from the node based on the corrected score comprises applying an activation function to the corrected score to generate the output. 
     
     
         20 . The non-transitory machine readable medium of  claim 15 , wherein the neural network comprises a speech recognition neural network, the non-transitory machine readable medium further comprising instructions that, in response to being executed on the computing device, cause the computing device to perform speech recognition by:
 converting received speech to a speech recording;   extracting feature vectors associated with time windows of the speech recording.