Recurrent neural networks for online sequence generation

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence from a source sequence. In one aspect, the system includes a recurrent neural network configured to, at each time step, receive am input for the time step and process the input to generate a progress score and a set of output scores; and a subsystem configured to, at each time step, generate the recurrent neural network input and provide the input to the recurrent neural network; determine, from the progress score, whether or not to emit a new output at the time step; and, in response to determining to emit a new output, select an output using the output scores and emit the selected output as the output at a next position in the output order.

What is claimed is: 
     
       1. A computer system for online speech recognition configured to generate a target sequence comprising a plurality of outputs arranged according to an output order from a source sequence comprising a plurality of inputs arranged according to an input order, wherein the plurality of inputs include voice tokens and the plurality of outputs include text tokens, the computer system comprising:
 a recurrent neural network implemented by one or computers, wherein the recurrent neural network is configured to, at each of a plurality of time steps:
 receive a recurrent neural network input for the time step comprising (i) a current input for the time step, (ii) a most recently emitted output from the target sequence, and (iii) a current internal state of the recurrent neural network; and 
 process the recurrent neural network input in accordance with the current internal state of the recurrent neural network to generate (i) a progress score and (ii) a set of output scores; and 
 
 a subsystem implemented by the one or more computers, wherein the subsystem is configured to: at each of the plurality of time steps and beginning before the last input in the input order is received:
 generate the recurrent neural network input for the time step and provide the recurrent neural network input as input to the recurrent neural network; 
 determine, from the progress score, whether or not to emit a new output at the time step; 
 in response to determining to emit a new output:
 select an output using the output scores; 
 emit the selected output as the output at a next position in the output order; and 
 update the current internal state of the recurrent neural network; and 
 
 in response to determining not to emit a new output:
 update the current internal state of the recurrent neural network without emitting a new output. 
 
 
 
     
     
       2. The system of  claim 1 , wherein the recurrent neural network is a Grid Long Short-Term Memory (LSTM) neural network. 
     
     
       3. The system of  claim 1 , wherein the recurrent neural network is a stacked deep LSTM neural network. 
     
     
       4. The system of  claim 1 , wherein the recurrent neural network input further comprises a progress input indicating whether or not an output was emitted at a time step immediately preceding the time step. 
     
     
       5. The system of  claim 1 , wherein the source sequence comprises k inputs, wherein k is an integer equal to or greater than one, wherein the plurality of time steps are ordered according to a time step order, and wherein, when i is less than or equal to k, the current input for an i-th time step in the time step order is the i-th input in the input order. 
     
     
       6. The system of  claim 5 , when i is greater than k, the current input for an i-th time step in the time step order is a last input in the input order. 
     
     
       7. The system of  claim 5 , when i is greater than k, the current input for an i-th time step in the time step order is a placeholder input. 
     
     
       8. The system of  claim 1 , wherein determining, from the progress score, whether or not to emit a new output at the time step comprises:
 sampling from a two-point distribution in accordance with the progress score to select either an emit value or a do not emit value for a binary variable; and 
 determining to emit a new output when the emit value is selected and determining not to emit a new output when the do not emit value is selected. 
 
     
     
       9. The system of  claim 1 , wherein the set of output scores includes a respective score for each output in a vocabulary of possible outputs and a score for an end of sequence output, and wherein selecting an output comprises selecting an output having a highest output score. 
     
     
       10. The system of  claim 9 , wherein the subsystem is further configured to terminate the output sequence if the selected output is the end of sequence output. 
     
     
       11. A computer-implemented method for online speech recognition to generate a target sequence comprising a plurality of outputs arranged according to an output order from a source sequence comprising a plurality of inputs arranged according to an input order, wherein the plurality of inputs include voice tokens and the plurality of outputs include text tokens, the method comprising:
 before the last input in the input order is received: 
 generating a recurrent neural network input for a time step and providing the recurrent neural network input as input to the recurrent neural network, wherein the recurrent neural network is configured to, at each of a plurality of time steps:
 receive the recurrent neural network input for the time step comprising (i) a current input for the time step, (ii) a most recently emitted output from the target sequence, and (iii) a current internal state of the recurrent neural network; and 
 process the recurrent neural network input in accordance with the current internal state of the recurrent neural network to generate (i) a progress score and (ii) a set of output scores before the last input in the input order is received; 
 
 determining, from the progress score, whether or not to emit a new output at the time step; and 
 in response to determining to emit a new output:
 selecting an output using the output scores; 
 emitting the selected output as the output at a next position in the output order, and 
 updating the current internal state of the recurrent neural network. 
 
 
     
     
       12. The method of  claim 11 , wherein the recurrent neural network is a Grid Long Short-Term Memory (LSTM) neural network. 
     
     
       13. The method of  claim 11 , wherein the recurrent neural network is a stacked deep LSTM neural network. 
     
     
       14. The method of  claim 11 , wherein the recurrent neural network input further comprises a progress input indicating whether or not an output was emitted at a time step immediately preceding the time step. 
     
     
       15. The method of  claim 11 , wherein the source sequence comprises k inputs, wherein k is an integer equal to or greater than one, wherein the plurality of time steps are ordered according to a time step order, and wherein, when i is less than or equal to k, the current input for an i-th time step in the time step order is the i-th input in the input order. 
     
     
       16. The method of  claim 15 , when i is greater than k, the current input for an i-th time step in the time step order is a last input in the input order. 
     
     
       17. The method of  claim 15 , when i is greater than k, the current input for an i-th time step in the time step order is a placeholder input. 
     
     
       18. The method of  claim 11 , wherein determining, from the progress score, whether or not to emit a new output at the time step comprises:
 sampling from a two-point distribution in accordance with the progress score to select either an emit value or a do not emit value for a binary variable; and 
 determining to emit a new output when the emit value is selected and determining not to emit a new output when the do not emit value is selected. 
 
     
     
       19. The method of  claim 11 , wherein the set of output scores includes a respective score for each output in a vocabulary of possible outputs and a score for an end of sequence output, and wherein selecting an output comprises selecting an output having a highest output score. 
     
     
       20. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform online speech recognition operations for generating a target sequence comprising a plurality of outputs arranged according to an output order from a source sequence comprising a plurality of inputs arranged according to an input order, wherein the plurality of inputs include voice tokens and the plurality of outputs include text tokens, the operations comprising:
 before the last input in the input order is received: 
 generating a recurrent neural network input for a time step and providing the recurrent neural network input as input to the recurrent neural network, wherein the recurrent neural network is configured to, at each of a plurality of time steps:
 receive the recurrent neural network input for the time step comprising (i) a current input for the time step, (ii) a most recently emitted output from the target sequence, and (iii) a current internal state of the recurrent neural network; and 
 process the recurrent neural network input in accordance with the current internal state of the recurrent neural network to generate (i) a progress score and (ii) a set of output scores before the last input in the input order is received; 
 
 determining, from the progress score, whether or not to emit a new output at the time step; and 
 in response to determining to emit a new output:
 selecting an output using the output scores; 
 emitting the selected output as the output at a next position in the output order, and 
 updating the current internal state of the recurrent neural network.