Generating output sequences from input sequences using neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output sequences from input sequences. One of the methods includes obtaining an input sequence having a first number of inputs arranged according to an input order; processing each input in the input sequence using an encoder recurrent neural network to generate a respective encoder hidden state for each input in the input sequence; and generating an output sequence having a second number of outputs arranged according to an output order, each output in the output sequence being selected from the inputs in the input sequence, comprising, for each position in the output order: generating a softmax output for the position using the encoder hidden states that is a pointer into the input sequence; and selecting an input from the input sequence as the output at the position using the softmax output.

What is claimed is: 
     
         1 - 20 . (canceled) 
     
     
         21 . A computer-implemented method comprising:
 processing an input sequence having a first number of inputs arranged according to an input order using an encoder neural network to generate a respective encoder hidden state for each input in the input sequence; and   generating an output sequence having a second number of outputs arranged according to an output order, each output in the output sequence being selected from the inputs in the input sequence, comprising, for each position in the output order:
 generating, using an encoder hidden state, a respective output for the position in the output order, the respective output providing a pointer into the input sequence; and 
 selecting, using the respective output, an input from the input sequence as the output for the output sequence at the position in the output order. 
   
     
     
         22 . The method of  claim 21 , wherein the respective output for the position comprises a softmax output for the position. 
     
     
         23 . The method of  claim 22 , wherein generating the softmax output for the position comprises, for the initial position in the output order:
 processing a predetermined initial output using a decoder neural network to generate an initial decoder hidden state;   generating, from the initial decoder hidden state and the hidden states for the inputs in the input sequence, the softmax output for the initial position, wherein the softmax output comprises a respective output score for each position in the input order; and   selecting an input at the highest-scoring position in the input order according to the output scores as the output at the initial position in the output order.   
     
     
         24 . The method of  claim 23 , wherein the encoder neural network and the decoder neural network are each a respective recurrent neural network. 
     
     
         25 . The method of  claim 23 , wherein processing the predetermined initial output using the decoder neural network to generate an initial decoder hidden state comprises:
 initializing an internal state of the decoder neural network to the encoder hidden state for the last input in the input sequence; and   processing the predetermined initial output using the decoder neural network to update the initialized internal state to the initial decoder hidden state.   
     
     
         26 . The method of  claim 23 , wherein generating the softmax output for the position comprises, for each position after the initial position in the output order:
 processing an output at the preceding position in the output order using the decoder neural network to generate a decoder hidden state for the output at the preceding position;   generating, from the decoder hidden state for the output at the preceding position and the hidden states for the inputs in the input sequence, the softmax output for the position, wherein the softmax output comprises a respective output score for each position in the input order; and   selecting an input at the highest-scoring position in the input order according to the output scores as the output at the position in the output order.   
     
     
         27 . The method of  claim 26 , wherein generating the softmax output for the position comprises:
 generating an attention vector having a dimensionality equal to the first number of inputs in the input sequence, comprising, for each position in the input order:
 combining the encoder hidden state for the input at the position in the input order and the decoder hidden state for the output at the preceding position in the output order to generate a value at a corresponding position in the vector; and 
 generating the respective output scores by processing the attention vector through a softmax output layer. 
   
     
     
         28 . The method of  claim 21 , further comprising, for each position in the output order:
 determining whether or not the selected output at the position is a designated input;   when the selected output at the position is not the designated input, determining that an output should be generated for a subsequent position in the output order; and   when the selected output at the position is the designated input, determining that the position is the last position in the output order.   
     
     
         29 . The method of  claim 21 , further comprising:
 generating one or more additional output sequences using a beam search technique;   determining a respective sequence score for the output sequence and for each additional output sequence; and   selecting an output sequence having a highest sequence score as a final output sequence for the input sequence.   
     
     
         30 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
 processing an input sequence having a first number of inputs arranged according to an input order using an encoder neural network to generate a respective encoder hidden state for each input in the input sequence; and   generating an output sequence having a second number of outputs arranged according to an output order, each output in the output sequence being selected from the inputs in the input sequence, comprising, for each position in the output order:
 generating, using an encoder hidden state, a respective output for the position in the output order, the respective output providing a pointer into the input sequence; and 
 selecting, using the respective output, an input from the input sequence as the output for the output sequence at the position in the output order. 
   
     
     
         31 . The system of  claim 30 , wherein the respective output for the position comprises a softmax output for the position. 
     
     
         32 . The system of  claim 31 , wherein generating the softmax output for the position comprises, for the initial position in the output order:
 processing a predetermined initial output using a decoder neural network to generate an initial decoder hidden state;   generating, from the initial decoder hidden state and the hidden states for the inputs in the input sequence, the softmax output for the initial position, wherein the softmax output comprises a respective output score for each position in the input order; and   selecting an input at the highest-scoring position in the input order according to the output scores as the output at the initial position in the output order.   
     
     
         33 . The system of  claim 32 , wherein the encoder neural network and the decoder neural network are each a respective recurrent neural network 
     
     
         34 . The system of  claim 32 , wherein processing the predetermined initial output using the decoder neural network to generate an initial decoder hidden state comprises:
 initializing an internal state of the decoder neural network to the encoder hidden state for the last input in the input sequence; and   processing the predetermined initial output using the decoder neural network to update the initialized internal state to the initial decoder hidden state.   
     
     
         35 . The system of  claim 32 , wherein generating the softmax output for the position comprises, for each position after the initial position in the output order:
 processing an output at the preceding position in the output order using the decoder neural network to generate a decoder hidden state for the output at the preceding position;   generating, from the decoder hidden state for the output at the preceding position and the hidden states for the inputs in the input sequence, the softmax output for the position, wherein the softmax output comprises a respective output score for each position in the input order; and   selecting an input at the highest-scoring position in the input order according to the output scores as the output at the position in the output order.   
     
     
         36 . The system of  claim 35 , wherein generating the softmax output for the position comprises:
 generating an attention vector having a dimensionality equal to the first number of inputs in the input sequence, comprising, for each position in the input order:
 combining the encoder hidden state for the input at the position in the input order and the decoder hidden state for the output at the preceding position in the output order to generate a value at a corresponding position in the vector; and 
 generating the respective output scores by processing the attention vector through a softmax output layer. 
   
     
     
         37 . The system of  claim 30 , wherein the operations further comprise, for each position in the output order:
 determining whether or not the selected output at the position is a designated input;   when the selected output at the position is not the designated input, determining that an output should be generated for a subsequent position in the output order; and   when the selected output at the position is the designated input, determining that the position is the last position in the output order.   
     
     
         38 . The system of  claim 30 , wherein the operations further comprise:
 generating one or more additional output sequences using a beam search technique;   determining a respective sequence score for the output sequence and for each additional output sequence; and   selecting an output sequence having a highest sequence score as a final output sequence for the input sequence.   
     
     
         39 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
 processing an input sequence having a first number of inputs arranged according to an input order using an encoder neural network to generate a respective encoder hidden state for each input in the input sequence; and   generating an output sequence having a second number of outputs arranged according to an output order, each output in the output sequence being selected from the inputs in the input sequence, comprising, for each position in the output order:
 generating, using an encoder hidden state, a respective output for the position in the output order, the respective output providing a pointer into the input sequence; and 
 selecting, using the respective output, an input from the input sequence as the output for the output sequence at the position in the output order. 
   
     
     
         40 . The computer storage media of  claim 39 , wherein generating the softmax output for the position comprises, for the initial position in the output order:
 processing a predetermined initial output using a decoder neural network to generate an initial decoder hidden state;   generating, from the initial decoder hidden state and the hidden states for the inputs in the input sequence, the softmax output for the initial position, wherein the softmax output comprises a respective output score for each position in the input order; and   selecting an input at the highest-scoring position in the input order according to the output scores as the output at the initial position in the output order.