Encoder-decoder models for sequence to sequence mapping

Abstract

Methods, systems, and apparatus for performing speech recognition. In some implementations, acoustic data representing an utterance is obtained. The acoustic data corresponds to time steps in a series of time steps. One or more computers process scores indicative of the acoustic data using a recurrent neural network to generate a sequence of outputs. The sequence of outputs indicates a likely output label from among a predetermined set of output labels. The predetermined set of output labels includes output labels that respectively correspond to different linguistic units and to a placeholder label that does not represent a classification of acoustic data. The recurrent neural network is configured to use an output label indicated for a previous time step to determine an output label for the current time step. The generated sequence of outputs is processed to generate a transcription of the utterance, and the transcription of the utterance is provided.

What is claimed is: 
     
         1 . An apparatus for training a recurrent neural network, the apparatus comprising:
 processing circuitry configured to   obtain a plurality of training examples, each training example comprising (i) an input acoustic sequence of scores indicative of acoustic data at each of multiple time steps in a series of time steps, the input acoustic sequence representing a known utterance, and (ii) a corresponding target sequence of linguistic units representing a transcription of the known utterance,   represent possible alignments between the input acoustic sequence and the target sequence of linguistic units as a lattice, each node in the lattice represents a respective recurrent neural network state of the recurrent neural network, each recurrent neural network state being dependent on a respective time step from the series of time steps and a respective position in the target sequence of linguistic units, and first and second subsets of transitions between nodes in the lattice representing probabilities of observing respective subsequent linguistic units and blank labels in the target sequence of linguistic units, respectively, and   train the recurrent neural network through the lattice.   
     
     
         2 . The apparatus of  claim 1 , wherein the processing circuitry is further configured to
 define, as an output label representing one of the blank labels, a generated output for a first time step in the series of time steps.   
     
     
         3 . The apparatus of  claim 1 , wherein the processing circuitry is further configured to
 perform a forward-backward calculation through the lattice to calculate a first loss function.   
     
     
         4 . The apparatus of  claim 3 , wherein the training the recurrent neural network includes
 training the recurrent neural network to minimize the first loss function using the plurality of training examples.   
     
     
         5 . The apparatus of  claim 4 , wherein the first loss function is a log likelihood loss function. 
     
     
         6 . The apparatus of  claim 3 , wherein the performing the forward-backward calculation through the lattice includes
 determining values of multiple forward variables, each forward variable corresponding to a respective time step from {1, . . . , t} and representing a probability of outputting a particular sequence of n linguistic units up to the respective time step, t being an integer greater than zero and n being an integer greater than or equal to zero.   
     
     
         7 . The apparatus of  claim 6 , wherein the performing the forward-backward calculation through the lattice includes
 determining that two different transitions between start node (t−1, n−1) and end node (t, n) exist in the lattice, the two different transitions comprising a first transition through a first intermediate node (t, n−1) and a second transition through a second intermediate node (t−1, n),   updating the recurrent neural network state for the end node to equal the recurrent neural network state corresponding to the start node (t−1, n−1) if a sum of the forward variable for the start node (t−1, n−1) and the probability of outputting a linguistic unit at the start node (t−1, n−1) is greater than a sum of the forward variable for the second intermediate node (t−1, n) and the probability of outputting one of the blank labels at the second intermediate node (t−1, n), and   updating the recurrent neural network state for the end node to equal the recurrent neural network state corresponding to the second intermediate node (t−1, n) if the sum of the forward variable for the start node (t−1, n−1) and the probability of outputting a linguistic unit at the start node (t−1, n−1) is not greater than the sum of the forward variable for the second intermediate node (t−1, n) and the probability of outputting the one of the blank labels at the second intermediate node (t−1, n).   
     
     
         8 . The apparatus of  claim 6 , wherein the performing the forward-backward calculation through the lattice includes
 determining values of multiple backward variables, each backward variable representing a probability of outputting a particular sequence of N−n linguistic units from the time t, N being a length of the target sequence of linguistic units.   
     
     
         9 . The apparatus of  claim 8 , wherein the performing the forward-backward calculation through the lattice includes
 calculating the first loss function by summing path possibilities through the lattice.   
     
     
         10 . The apparatus of  claim 1 , wherein the training the recurrent neural network includes
 training the recurrent neural network to minimize a second loss function using the plurality of training examples.   
     
     
         11 . The apparatus of  claim 10 , wherein the second loss function is an expected loss function. 
     
     
         12 . The apparatus of  claim 10 , wherein the training the recurrent neural network to minimize the second loss function includes
 performing a back propagation technique using the second loss function to adjust recurrent neural network parameters to trained recurrent neural network parameters.   
     
     
         13 . The apparatus of  claim 1 , wherein the processing circuitry is further configured to
 process a received input acoustic sequence to generate a sequence of outputs, the generated sequence of outputs indicating output labels from among a predetermined set of output labels including output labels that respectively correspond to different linguistic units and to a blank label that does not represent a classification of the received input acoustic sequence.   
     
     
         14 . The apparatus of  claim 13 , wherein the processing the received input acoustic sequence includes
 defining a first unit in the sequence of outputs as the blank label.   
     
     
         15 . The apparatus of  claim 13 , wherein the processing the received input acoustic sequence includes
 estimate a conditional probability distribution representing a probability of an alignment between the scores indicative of the acoustic data and the sequence of outputs, wherein   the conditional probability distribution comprises a product of output conditional probabilities for each time step in the series of time steps, each output conditional probability representing a probability of an output for a respective time step given the score for the respective time step and an output for a preceding time step.   
     
     
         16 . The apparatus of  claim 13 , wherein the processing the received input acoustic sequence includes
 removing the blank label from the sequence of outputs; and   concatenating the remaining of the sequence of outputs as a most likely output sequence of linguistic units.   
     
     
         17 . The apparatus of  claim 1 , wherein the first and second subsets of transitions are diagonal and horizontal transitions, respectively. 
     
     
         18 . The apparatus of  claim 16 , wherein the recurrent neural network comprises one or more long short-term memory neural network layers and a softmax output layer. 
     
     
         19 . A method for training a recurrent neural network, the method comprising:
 obtaining a plurality of training examples, each training example comprising (i) an input acoustic sequence of scores indicative of acoustic data at each of multiple time steps in a series of time steps, the input acoustic sequence representing a known utterance, and (ii) a corresponding target sequence of linguistic units representing a transcription of the known utterance;   representing possible alignments between the input acoustic sequence and the target sequence of linguistic units as a lattice, each node in the lattice represents a respective recurrent neural network state of the recurrent neural network, each recurrent neural network state being dependent on a respective time step from the series of time steps and a respective position in the target sequence of linguistic units, and first and second subsets of transitions between nodes in the lattice representing probabilities of observing respective subsequent linguistic units and blank labels in the target sequence of linguistic units, respectively; and   training the recurrent neural network through the lattice.   
     
     
         20 . A non-transitory computer-readable storage medium comprising instructions stored thereon that are executable by a processing device and upon such execution cause the processing device to train a recurrent neural network, the training comprising:
 obtaining a plurality of training examples, each training example comprising (i) an input acoustic sequence of scores indicative of acoustic data at each of multiple time steps in a series of time steps, the input acoustic sequence representing a known utterance, and (ii) a corresponding target sequence of linguistic units representing a transcription of the known utterance;   representing possible alignments between the input acoustic sequence and the target sequence of linguistic units as a lattice, each node in the lattice represents a respective recurrent neural network state of the recurrent neural network, each recurrent neural network state being dependent on a respective time step from the series of time steps and a respective position in the target sequence of linguistic units, and first and second subsets of transitions between nodes in the lattice representing probabilities of observing respective subsequent linguistic units and blank labels in the target sequence of linguistic units, respectively; and   training the recurrent neural network through the lattice.