Proper noun recognition in end-to-end speech recognition

Abstract

A method for training a speech recognition model with a minimum word error rate loss function includes receiving a training example comprising a proper noun and generating a plurality of hypotheses corresponding to the training example. Each hypothesis of the plurality of hypotheses represents the proper noun and includes a corresponding probability that indicates a likelihood that the hypothesis represents the proper noun. The method also includes determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criteria. The penalty criteria indicating that the corresponding probability satisfies a probability threshold, and the associated hypothesis incorrectly represents the proper noun. The method also includes applying a penalty to the minimum word error rate loss function.

What is claimed is: 
     
         1 . A computer-implemented method that when executed on data processing hardware causes the data processing to perform operations comprising:
 receiving a training example comprising a ground-truth transcription, the ground-truth transcription comprising a word sequence that includes a proper noun; and   training a speech recognition model with a fuzz training loss function by:
 generating an original beam of hypotheses corresponding to the training example, each hypothesis in the original beam of hypotheses comprising a respective sequence of words and a corresponding probability that indicates a likelihood that the hypothesis correctly identifies the ground-truth transcription; 
 generating a fuzz hypothesis corresponding to the training example that comprises a respective sequence of words that includes an incorrect proper noun alternative word and a corresponding probability that indicates a likelihood that the fuzz hypothesis correctly identifies the ground-truth transcription; 
 determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses; and 
 based on determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses, applying a penalty to the training loss function. 
   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the incorrect proper noun alternative word included in the respective sequence of words of the fuzz hypothesis comprises a phonetic similarity to the proper noun included in the word sequence of the ground-truth transcription. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein the incorrect proper noun alternative word included in the respective sequence of words of the fuzz hypothesis comprises an alternative spelling to a spelling of the proper noun included in the word sequence of the ground-truth transcription. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the operations further comprise, prior to training the speech recognition model with the fuzz training loss function:
 identifying the proper noun in the word sequence of the ground-truth transcription; and   generating a set number of different alternative words for the identified proper noun,   wherein generating the fuzz hypothesis during training the speech recognition model comprises randomly selecting one of the different alternative words for the identified proper noun as the incorrect proper noun alternative word included in respective sequence of words of the fuzz hypothesis.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein generating the fuzz hypothesis comprises:
 identifying an occurrence of the proper noun in the respective sequence of words of one of the hypotheses from the original beam of hypotheses; and   replacing the occurrence of proper noun in the respective sequence of words of the one of the hypotheses from the original beam of hypotheses with the incorrect proper noun alternative word.   
     
     
         6 . The computer-implemented method of  claim 5 , wherein the operations further comprise combining the original beam of hypotheses with the fuzz hypothesis. 
     
     
         7 . The computer-implemented method of  claim 5 , wherein the operations further comprise substituting the fuzz hypothesis for the one of the hypotheses from the original beam search that comprises the respective sequence of words including the occurrence of the proper noun. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the speech recognition model comprises a two-pass architecture comprising:
 a first pass network comprising a recurrent neural network transducer (RNN-T) decoder; and   a second pass network comprising a listen-attend-spell (LAS) decoder.   
     
     
         9 . The computer-implemented method of  claim 8 , wherein the speech recognition model further comprises a shared encoder, the shared encoder encoding acoustic frames for each of the first pass network and the second pass network. 
     
     
         10 . The computer-implemented method of  claim 8 , wherein the operations further comprise:
 training the RNN-T decoder; and   prior to training the LAS decoder with the fuzz training loss function, training the LAS decoder while parameters of the trained RNN-T decoder remain fixed.   
     
     
         11 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 receiving a training example comprising a ground-truth transcription, the ground-truth transcription comprising a word sequence that includes a proper noun; and 
 training a speech recognition model with a fuzz training loss function by:
 generating an original beam of hypotheses corresponding to the training example, each hypothesis in the original beam of hypotheses comprising a respective sequence of words and a corresponding probability that indicates a likelihood that the hypothesis correctly identifies the ground-truth transcription; 
 generating a fuzz hypothesis corresponding to the training example that comprises a respective sequence of words that includes an incorrect proper noun alternative word and a corresponding probability that indicates a likelihood that the fuzz hypothesis correctly identifies the ground-truth transcription; 
 determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses; and 
 based on determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses, applying a penalty to the training loss function. 
 
   
     
     
         12 . The system of  claim 11 , wherein the incorrect proper noun alternative word included in the respective sequence of words of the fuzz hypothesis comprises a phonetic similarity to the proper noun included in the word sequence of the ground-truth transcription. 
     
     
         13 . The system of  claim 11 , wherein the incorrect proper noun alternative word included in the respective sequence of words of the fuzz hypothesis comprises an alternative spelling to a spelling of the proper noun included in the word sequence of the ground-truth transcription. 
     
     
         14 . The system of  claim 11 , wherein the operations further comprise, prior to training the speech recognition model with the fuzz training loss function:
 identifying the proper noun in the word sequence of the ground-truth transcription; and   generating a set number of different alternative words for the identified proper noun,   wherein generating the fuzz hypothesis during training the speech recognition model comprises randomly selecting one of the different alternative words for the identified proper noun as the incorrect proper noun alternative word included in respective sequence of words of the fuzz hypothesis.   
     
     
         15 . The system of  claim 11 , wherein generating the fuzz hypothesis comprises:
 identifying an occurrence of the proper noun in the respective sequence of words of one of the hypotheses from the original beam of hypotheses; and   replacing the occurrence of proper noun in the respective sequence of words of the one of the hypotheses from the original beam of hypotheses with the incorrect proper noun alternative word.   
     
     
         16 . The system of  claim 15 , wherein the operations further comprise combining the original beam of hypotheses with the fuzz hypothesis. 
     
     
         17 . The system of  claim 15 , wherein the operations further comprise substituting the fuzz hypothesis for the one of the hypotheses from the original beam search that comprises the respective sequence of words including the occurrence of the proper noun. 
     
     
         18 . The system of  claim 11 , wherein the speech recognition model comprises a two-pass architecture comprising:
 a first pass network comprising a recurrent neural network transducer (RNN-T) decoder; and   a second pass network comprising a listen-attend-spell (LAS) decoder.   
     
     
         19 . The system of  claim 18 , wherein the speech recognition model further comprises a shared encoder, the shared encoder encoding acoustic frames for each of the first pass network and the second pass network. 
     
     
         20 . The system of  claim 18 , wherein the operations further comprise:
 training the RNN-T decoder; and   prior to training the LAS decoder with the fuzz training loss function, training the LAS decoder while parameters of the trained RNN-T decoder remain fixed.