Dual-momentum gradient optimization with reduced memory requirements

Abstract

Systems and methods related to dual-momentum gradient optimization with reduced memory requirements are described. An example method in a system comprising a gradient optimizer and a memory configured to store momentum values associated with a neural network model comprising L layers is described. The method includes retrieving from the memory a first set of momentum values and a second set of momentum values, corresponding to a layer of the neural network model, having a selected storage format. The method further includes converting the first set of momentum values to a third set of momentum values having a training format associated with the gradient optimizer and converting the second set of momentum values to a fourth set of momentum values having a training format associated with the gradient optimizer. The method further includes performing gradient optimization using the third set of momentum values and the fourth set of momentum values.

1 .- 20 . (canceled) 
     
     
         21 . A method for operating a system comprising a dual-momentum gradient optimizer and a memory configured to store momentum values associated with a neural network model comprising L layers, wherein Lis an integer greater than one, the method comprising:
 using a first momentum decompressor, decompressing a first set of compressed momentum values associated with a first momentum of a dual-momentum algorithm to generate a first set of decompressed momentum values;   using a second momentum decompressor, decompressing a second set of compressed momentum values associated with a second momentum of the dual-momentum algorithm, different from the first momentum, to generate a second set of decompressed momentum values;   using the dual-momentum gradient optimizer, (1) generating a first set of new momentum values based on the first set of decompressed momentum values and (2) generating a second set of new momentum values based on the second set of decompressed values, wherein the dual-momentum gradient optimizer is configured to operate in a burst mode such that successive burst cycles result in streaming of gradients through the dual-momentum gradient optimizer;   using a first compressor, compressing the first set of new momentum values into a selected storage format associated with the memory; and   using a second compressor, compressing the second set of new momentum values into the selected storage format associated with the memory.   
     
     
         22 . The method of  claim 21 , wherein each of the first set of decompressed momentum values and the second set of decompressed momentum values corresponds to a training format for the dual-momentum gradient optimizer. 
     
     
         23 . The method of  claim 22 , wherein the training format comprises a single-precision format or a double-precision format, and wherein the method further comprises padding extra bits to form single-precision format decompressed momentum values or double-precision format decompressed momentum values. 
     
     
         24 . The method of  claim 22 , wherein the training format comprises a 32-bit precision format and wherein each of the first set of compressed momentum values and the second set of compressed momentum values correspond to an 8-bit compressed format, and wherein the 8-bit compressed format is converted into the 32-bit precision format by inserting one zero bit between a most significant bit of an exponent and padding twenty-three bits right after a least significant bit of the exponent. 
     
     
         25 . The method of  claim 21 , wherein the selected storage format comprises a reduced single-precision format. 
     
     
         26 . The method of  claim 25 , wherein values corresponding to the reduced single-precision format are formed by removing a most significant bit of each of the first set of new momentum values and by truncating all mantissa bits of each of the first set of new momentum values, and wherein values corresponding to the reduced single-precision format are formed by removing the most significant bit of each of the second set of new momentum values and by truncating all mantissa bits of each of the second set of new momentum values. 
     
     
         27 . The method of  claim 21 , wherein the dual-gradient optimizer is configured to implement an adaptive moment estimation algorithm. 
     
     
         28 . A system comprising:
 a first momentum decompressor for decompressing a first set of compressed momentum values associated with a first momentum of a dual-momentum algorithm to generate a first set of decompressed momentum values;   a second momentum decompressor for decompressing a second set of compressed momentum values associated with a second momentum of the dual-momentum algorithm, different from the first momentum, to generate a second set of decompressed momentum values;   a dual-momentum gradient optimizer for: (1) generating a first set of new momentum values based on the first set of decompressed momentum values and (2) generating a second set of new momentum values based on the second set of decompressed values, wherein the dual-momentum gradient optimizer is configured to operate in a burst mode such that successive burst cycles result in streaming of gradients through the dual-momentum gradient optimizer;   a first compressor for compressing the first set of new momentum values into a selected storage format associated with a memory; and   a second compressor for compressing the second set of new momentum values into the selected storage format associated with the memory.   
     
     
         29 . The system of  claim 28 , wherein each of the first set of decompressed momentum values and the second set of decompressed momentum values corresponds to a training format for the dual-momentum gradient optimizer. 
     
     
         30 . The system of  claim 29 , wherein the training format comprises a single-precision format or a double-precision format, and wherein the system is further configured to pad extra bits to form single-precision format decompressed momentum values or double-precision format decompressed momentum values. 
     
     
         31 . The system of  claim 29 , wherein the training format comprises a 32-bit precision format, wherein each of the first set of compressed momentum values and the second set of compressed momentum values correspond to an 8-bit compressed format, and wherein the 8-bit compressed format is converted into the 32-bit precision format by inserting one zero bit between a most significant bit of an exponent and padding twenty-three bits right after a least significant bit of the exponent. 
     
     
         32 . The system of  claim 28 , wherein the selected storage format comprises a reduced single-precision format. 
     
     
         33 . The system of  claim 32 , wherein the first compressor forms values corresponding to the reduced single-precision format by removing a most significant bit of each of the first set of new momentum values and by truncating all mantissa bits of each of the first set of new momentum values, and wherein the second compressor forms values corresponding to the reduced single-precision format by removing the most significant bit of each of the second set of new momentum values and by truncating all mantissa bits of each of the second set of new momentum values. 
     
     
         34 . The system of  claim 28 , wherein the dual-gradient optimizer is configured to implement an adaptive moment estimation algorithm. 
     
     
         35 . The system of  claim 28 , wherein the dual-gradient gradient optimizer is implemented using a field programmable gate array (FPGA). 
     
     
         36 . A system comprising:
 a first momentum decompressor for decompressing a first set of compressed momentum values associated with a first momentum of a dual-momentum adaptive moment estimation algorithm to generate a first set of decompressed momentum values;   a second momentum decompressor for decompressing a second set of compressed momentum values associated with a second momentum of the dual-momentum adaptive moment estimation algorithm, different from the first momentum, to generate a second set of decompressed momentum values;   a dual-momentum gradient optimizer for: (1) generating a first set of new momentum values based on the first set of decompressed momentum values and (2) generating a second set of new momentum values based on the second set of decompressed values;   a first compressor for compressing the first set of new momentum values into a selected storage format associated with a memory; and   a second compressor for compressing the second set of new momentum values into the selected storage format associated with the memory.   
     
     
         37 . The system of  claim 36 , wherein each of the first set of decompressed momentum values and the second set of decompressed momentum values corresponds to a training format for the dual-momentum gradient optimizer. 
     
     
         38 . The system of  claim 37 , wherein the training format comprises a single-precision format or a double-precision format, and wherein the system is further configured to pad extra bits to form single-precision format decompressed momentum values or double-precision format decompressed momentum values. 
     
     
         39 . The system of  claim 38 , wherein the selected storage format comprises a reduced single-precision format. 
     
     
         40 . The system of  claim 39 , wherein the dual-gradient optimizer is configured to implement an adaptive moment estimation algorithm.