Hyperbolic functions for machine learning acceleration

Abstract

The present disclosure relates generally to techniques for enhancing recurrent neural networks (RNNs) implemented on an integrated circuit. In particular, approximations of activation functions used in an RNN, such as sigmoid and hyperbolic tangent, may be implemented in an integrated circuit, which may result in increased efficiencies, reduced latency, increased accuracy, and reduced resource consumption involved with implementing machine learning.

What is claimed is: 
     
         1 . A tangible, non-transitory, machine-readable medium, comprising machine-readable instructions that, when executed by one or more processors, cause the processors to:
 receive an input to an activation function;   evaluate a piecewise polynomial function to generate a first approximation result corresponding to a first input, wherein a first interval of inputs to the activation function comprises the first input, wherein the piecewise polynomial function approximates the activation function on the first interval;   determine, using an identity of the activation function and the first approximation result, a second approximation result corresponding to a second input, wherein a second interval of inputs to the activation function comprises the second input;   determine a first saturation value of the activation function corresponding to a third input, wherein a third interval of inputs to the activation function comprises the third input;   determine a second saturation value of the activation function corresponding to a fourth input, wherein a fourth interval of inputs to the activation function comprises the fourth input;   determine whether the first interval, the second interval, the third interval, or the fourth interval comprise the input; and   in response to determining the first interval comprises the input, selecting the first approximation result as an approximate value of the activation function at the input.   
     
     
         2 . The tangible, non-transitory, machine-readable medium of  claim 1 , comprising machine-readable instructions that, when executed by one or more processors, cause the processors to divide the first interval into a number of sub-intervals, wherein the piecewise polynomial function comprises a polynomial function for each sub-interval. 
     
     
         3 . The tangible, non-transitory, machine-readable medium of  claim 2 , wherein an accuracy of the approximate value compared to a corresponding actual value of the activation function is based at least in part on the number of sub-intervals, a degree of the piecewise polynomial function, or a combination thereof. 
     
     
         4 . The tangible, non-transitory, machine-readable medium of  claim 1 , comprising machine-readable instructions that, when executed by one or more processors, cause the processors to:
 in response to determining the second interval comprises the input, selecting the second approximation result as the approximate value;   in response to determining the third interval comprises the input, selecting the first saturation value as the approximate value; and   in response to determining the fourth interval comprises the input, selecting the second saturation value as the approximate value.   
     
     
         5 . The tangible, non-transitory, machine-readable medium of  claim 1 , wherein an accuracy of the approximate value compared to a corresponding actual value of the activation function is based at least in part on a size of the first interval. 
     
     
         6 . The tangible, non-transitory, machine-readable medium of  claim 2 , wherein an amount of resources used to find the approximate value is based at least in part on the number of sub-intervals, a degree of the piecewise polynomial function, or a combination thereof. 
     
     
         7 . The tangible, non-transitory, machine-readable medium of  claim 1 , wherein the activation function comprises a hyperbolic tangent. 
     
     
         8 . A hardware-implemented method to approximate an activation function, comprising:
 receiving, at a barrel shifter, an input to an activation function;   evaluating, using multiply-add circuitry, a piecewise polynomial function to generate a first approximation result corresponding to a first input, wherein a first interval of inputs to the activation function comprises the first input, wherein the piecewise polynomial function approximates the activation function on the first interval;   determining, using a subtractor, an identity of the activation function, and the first approximation result, a second approximation result corresponding to a second input, wherein a second interval of inputs to the activation function comprises the second input;   determining a first saturation value of the activation function corresponding to a third input, wherein a third interval of inputs to the activation function comprises the third input;   determining a second saturation value of the activation function corresponding to a fourth input, wherein a fourth interval of inputs to the activation function comprises the fourth input;   determining whether the first interval, the second interval, the third interval, or the fourth interval comprise the input; and   in response to determining the first interval comprises the input, selecting, using a multiplexer, the first approximation result as an approximate value of the activation function at the input.   
     
     
         9 . The hardware-implemented method of  claim 8 , comprising dividing the first interval into a number of sub-intervals, wherein the piecewise polynomial function comprises a polynomial function for each sub-interval. 
     
     
         10 . The hardware-implemented method of  claim 9 , wherein an accuracy of the approximate value compared to a corresponding actual value of the activation function is based at least in part on the number of sub-intervals, a degree of the piecewise polynomial function, or a combination thereof. 
     
     
         11 . The hardware-implemented method of  claim 9 , comprising:
 in response to determining the second interval comprises the input, selecting, using the multiplexer, the second approximation result as the approximate value   in response to determining the third interval comprises the input, selecting, using the multiplexer, the first saturation value as the approximate value; and   in response to determining the fourth interval comprises the input, selecting, using the multiplexer, the second saturation value as the approximate value.   
     
     
         12 . The hardware-implemented method of  claim 8 , wherein an accuracy of the approximate value compared to a corresponding actual value of the activation function is based at least in part on a size of the first interval. 
     
     
         13 . A method of approximating an activation function based on an input to an integrated circuit device, comprising:
 outputting a coefficient from a plurality of coefficients of a coefficient table, wherein the plurality of coefficients of the coefficient table are indexed to an index that is based at least in part on the input, wherein the coefficient represents a coefficient in a polynomial approximating the activation function over a number of sub-intervals in a first interval;   generating a first approximation result using multiply-add circuitry, wherein the multiply-add circuitry is configured to evaluate the polynomial using a mathematical operation using the input and the coefficient; and   generating a second approximation result using a subtractor and based on an identity of the activation function and the first approximation result, wherein the second approximation result corresponds to an approximation of the activation function in a second interval, wherein the second interval comprises a second number of sub-intervals.   
     
     
         14 . The method of approximating the activation function of  claim 13 , comprising:
 receiving, at a multiplexer, a first saturation value, wherein the first saturation value comprises a first asymptote value of the activation function;   receiving, at the multiplexer, a second saturation value, wherein the second saturation value comprises an additional asymptote value of the activation function; and   receiving, at the multiplexer, the first approximation result;   receiving, at the multiplexer, the second approximation result; and   selecting, using the multiplexer and based at least in part on the input, an approximate output of the activation function among at least the first saturation value, the second saturation value, the first approximation result, and the second approximation result.   
     
     
         15 . The method of approximating the activation function of  claim 13 , wherein the input comprises a floating point number having a first precision, and, wherein a barrel shifter is configured to receive the input and configured to generate the index, wherein the index is a fixed-point number having a second precision, wherein the second precision is less than or equal to the first precision. 
     
     
         16 . The method of approximating the activation function of  claim 13 , wherein the activation function comprises sigmoid. 
     
     
         17 . The method of approximating the activation function of  claim 13 , wherein the activation function comprises hyperbolic tangent. 
     
     
         18 . The method of approximating the activation function of  claim 13 , wherein the mathematical operation has a first precision, the input has a second precision, the first approximation result has the second precision, and the first precision is greater than or equal to the second precision. 
     
     
         19 . The method of approximating the activation function of  claim 13 , comprising:
 storing the coefficient in the coefficient table, wherein the coefficient has a lower precision than a precision of the multiply-add circuitry; and   converting, using conversion circuitry, the coefficient from the lower precision to the precision of the multiply-add circuitry.   
     
     
         20 . The method of approximating the activation function of  claim 13 , outputting a second coefficient from a plurality of coefficients of a second coefficient table, wherein the plurality of coefficients of the second coefficient table are indexed to an index that is based at least in part on the input, wherein the second coefficient represents a second coefficient in a second polynomial approximating the activation function over a second number of sub-intervals in a second interval;
 generating a second approximation result using second multiply-add circuitry and based on the input and the second coefficient, wherein the second multiply-add circuitry is configured to evaluate the second polynomial using a mathematical operation using the input and the second coefficient; and   selecting, at a multiplexer and based at least in part on the input, an approximate output of the activation function among at least the first approximation result and the second approximation result.