Neural machine translation systems

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. The method comprises obtaining a first sequence of words in a source language, generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word, dividing the modified sequence of words into wordpieces using a wordpiece model, generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens.

What is claimed is: 
     
         1 . A method of generating an output sequence of words that represents a second sequence of words that is a translation of a first sequence of words from a source language into a target language, the method comprising:
 obtaining the first sequence of words in the source language;   generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word;   dividing the modified sequence of words into wordpieces using a wordpiece model;   generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and   generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens, wherein the output sequence of words represents the second sequence of words that is a translation of the first sequence of words from the source language into the target language.   
     
     
         2 . The method of  claim 1 , wherein the wordpiece model comprises a vocabulary with a predetermined number of basic characters and one or more special designated characters. 
     
     
         3 . The method of  claim 1 , wherein the wordpiece model is shared between the source language and the target language. 
     
     
         4 . The method of  claim 1 , wherein the wordpiece model comprises a vocabulary comprising between 8 k and 32 k wordpieces. 
     
     
         5 . The method of  claim 1 , wherein the neural machine translation system comprises:
 an encoder neural network comprising:
 an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, 
 an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token, 
 a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and 
 a plurality of hidden LSTM layers configured to process each combined representation in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence; and 
   a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.   
     
     
         6 . The method of  claim 5 , wherein the decoder subsystem comprises:
 a decoder neural network, the decoder neural network comprising:
 a plurality of LSTM layers arranged in a stack one on top of the other and configured to, for each of a plurality of positions in the output sequence:
 receive an attention context vector and an output token at a preceding position in the output sequence, and 
 process the attention context vector and the output token at the preceding position in the output sequence to generate an LSTM output for the position, and 
 
 a softmax output layer, the softmax output layer configured to, for each of the plurality of positions, receive the LSTM output for the position and to generate a respective score for each output token in a vocabulary of output tokens. 
   
     
     
         7 . The method of  claim 6 , wherein the decoder subsystem further comprises:
 an attention subsystem, wherein the attention subsystem is configured to, for each of the plurality of positions:
 receive an attention input comprising a bottom layer output generated by a bottom LSTM layer in the stack of LSTM layers in the decoder neural network for the preceding position; and 
 process the attention input to generate the attention context vector. 
   
     
     
         8 . A system comprising:
 one or more computers; and   one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by one or more computers, cause one or more computers to perform operations comprising:
 obtaining a first sequence of words in a source language; 
 generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word; 
 dividing the modified sequence of words into wordpieces using a wordpiece model; 
 generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and 
 generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens, wherein the output sequence of words represents a second sequence of words that is a translation of the first sequence of words from the source language into a target language. 
   
     
     
         9 . The system of  claim 8 , wherein the wordpiece model comprises a vocabulary with a predetermined number of basic characters and one or more special designated characters. 
     
     
         10 . The system of  claim 8 , wherein the wordpiece model is shared between the source language and the target language. 
     
     
         11 . The system of  claim 8 , wherein the wordpiece model comprises a vocabulary comprising between 8 k and 32 k wordpieces. 
     
     
         12 . The system of  claim 8 , wherein the neural machine translation system comprises:
 an encoder neural network comprising:
 an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, 
 an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token, 
 a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and 
 a plurality of hidden LSTM layers configured to process each combined representation in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence; and 
   a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.   
     
     
         13 . The system of  claim 12 , wherein the decoder subsystem comprises:
 a decoder neural network, the decoder neural network comprising:
 a plurality of LSTM layers arranged in a stack one on top of the other and configured to, for each of a plurality of positions in the output sequence:
 receive an attention context vector and an output token at a preceding position in the output sequence, and 
 process the attention context vector and the output token at the preceding position in the output sequence to generate an LSTM output for the position, and 
 
 a softmax output layer, the softmax output layer configured to, for each of the plurality of positions, receive the LSTM output for the position and to generate a respective score for each output token in a vocabulary of output tokens. 
   
     
     
         14 . The system of  claim 13 , wherein the decoder subsystem further comprises:
 an attention subsystem, wherein the attention subsystem is configured to, for each of the plurality of positions:
 receive an attention input comprising a bottom layer output generated by a bottom LSTM layer in the stack of LSTM layers in the decoder neural network for the preceding position; and 
 process the attention input to generate the attention context vector. 
   
     
     
         15 . One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
 obtaining a first sequence of words in a source language;   generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word;   dividing the modified sequence of words into wordpieces using a wordpiece model;   generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and   generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens, wherein the output sequence of words represents a second sequence of words that is a translation of the first sequence of words from the source language into a target language.   
     
     
         16 . The one or more non-transitory computer storage media of  claim 15 , wherein the wordpiece model comprises a vocabulary with a predetermined number of basic characters and one or more special designated characters. 
     
     
         17 . The one or more non-transitory computer storage media of  claim 15 , wherein the wordpiece model is shared between the source language and the target language. 
     
     
         18 . The one or more non-transitory computer storage media of  claim 15 , wherein the wordpiece model comprises a vocabulary comprising between 8 k and 32 k wordpieces. 
     
     
         19 . The one or more non-transitory computer storage media of  claim 15 , wherein the neural machine translation system comprises:
 an encoder neural network comprising:
 an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, 
 an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token, 
 a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and 
 a plurality of hidden LSTM layers configured to process each combined representation in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence; and 
   a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.   
     
     
         20 . The one or more non-transitory computer storage media of  claim 19 , wherein the decoder subsystem comprises:
 a decoder neural network, the decoder neural network comprising:
 a plurality of LSTM layers arranged in a stack one on top of the other and configured to, for each of a plurality of positions in the output sequence:
 receive an attention context vector and an output token at a preceding position in the output sequence, and 
 process the attention context vector and the output token at the preceding position in the output sequence to generate an LSTM output for the position, and 
 
 a softmax output layer, the softmax output layer configured to, for each of the plurality of positions, receive the LSTM output for the position and to generate a respective score for each output token in a vocabulary of output tokens.