Streaming Automatic Speech Recognition With Non-Streaming Model Distillation

Abstract

A method for training a streaming automatic speech recognition student model includes receiving a plurality of unlabeled student training utterances. The method also includes, for each unlabeled student training utterance, generating a transcription corresponding to the respective unlabeled student training utterance using a plurality of non-streaming automated speech recognition (ASR) teacher models. The method further includes distilling a streaming ASR student model from the plurality of non-streaming ASR teacher models by training the streaming ASR student model using the plurality of unlabeled student training utterances paired with the corresponding transcriptions generated by the plurality of non-streaming ASR teacher models.

What is claimed is: 
     
         1 . A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
 receiving an unlabeled student training utterance;   at each of a plurality of non-streaming automated speech recognition (ASR) teacher models, predicting an initial transcription for the unlabeled student training utterance;   for each of the initial transcriptions predicted by the plurality of non-streaming ASR teacher models:
 aligning the initial transcription with the other initial descriptions to define a sequence of alignment frames; and 
 dividing the initial transcription into transcription segments, each transcription segment corresponding to a respective alignment frame in the sequence of alignment frames; 
   for each alignment frame in the sequence of alignment frames, selecting a most repeated transcription segment across all of the initial transcriptions; and   concatenating the most repeated transcription segment of each respective frame to form a transcription of the unlabeled student training utterance.   
     
     
         2 . The method of  claim 1 , wherein the streaming ASR student model comprises a recurrent neural network transducer (RNN-T) architecture. 
     
     
         3 . The method of  claim 1 , wherein the streaming ASR student model comprises a conformer-based encoder. 
     
     
         4 . The method of  claim 1 , wherein each non-streaming ASR teacher model comprises a connectionist temporal classification (CTC) architecture. 
     
     
         5 . The method of  claim 4 , wherein the CTC architecture comprises a language model configured to capture contextual information for a respective utterance. 
     
     
         6 . The method of  claim 1 , wherein each non-streaming ASR teacher model comprises a conformer-based encoder. 
     
     
         7 . The method of  claim 1 , wherein the plurality of non-streaming ASR teacher models comprise at least two different recurrent neural network architectures. 
     
     
         8 . The method of  claim 7 , wherein a first non-streaming ASR teacher model comprises a recurrent neural network architecture and a second non-streaming ASR teacher model comprises a connectionist temporal classification (CTC) architecture. 
     
     
         9 . The method of  claim 1 , wherein the transcription segments divided from the initial transcriptions predicted by the plurality of non-streaming ASR teacher models comprise wordpieces or words. 
     
     
         10 . The method of  claim 1 , wherein one of the transcription segments divided from the initial transcription predicted by at least one of the plurality of non-streaming ASR teacher models comprises a blank produced by the at least one of the plurality of non-streaming ASR teacher models. 
     
     
         11 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 receiving an unlabeled student training utterance; 
 at each of a plurality of non-streaming automated speech recognition (ASR) teacher models, predicting an initial transcription for the unlabeled student training utterance; 
 for each of the initial transcriptions predicted by the plurality of non-streaming ASR teacher models:
 aligning the initial transcription with the other initial descriptions to define a sequence of alignment frames; and 
 dividing the initial transcription into transcription segments, each transcription segment corresponding to a respective alignment frame in the sequence of alignment frames; 
 
 for each alignment frame in the sequence of alignment frames, selecting a most repeated transcription segment across all of the initial transcriptions; and 
 concatenating the most repeated transcription segment of each respective frame to form a transcription of the unlabeled student training utterance. 
   
     
     
         12 . The system of  claim 11 , wherein the streaming ASR student model comprises a recurrent neural network transducer (RNN-T) architecture. 
     
     
         13 . The system of  claim 11 , wherein the streaming ASR student model comprises a conformer-based encoder. 
     
     
         14 . The system of  claim 11 , wherein each non-streaming ASR teacher model comprises a connectionist temporal classification (CTC) architecture. 
     
     
         15 . The system of  claim 14 , wherein the CTC architecture comprises a language model configured to capture contextual information for a respective utterance. 
     
     
         16 . The system of  claim 11 , wherein each non-streaming ASR teacher model comprises a conformer-based encoder. 
     
     
         17 . The system of  claim 11 , wherein the plurality of non-streaming ASR teacher models comprise at least two different recurrent neural network architectures. 
     
     
         18 . The system of  claim 17 , wherein a first non-streaming ASR teacher model comprises a recurrent neural network architecture and a second non-streaming ASR teacher model comprises a connectionist temporal classification (CTC) architecture. 
     
     
         19 . The system of  claim 11 , wherein the transcription segments divided from the initial transcriptions predicted by the plurality of non-streaming ASR teacher models comprise wordpieces or words. 
     
     
         20 . The system of  claim 11 , wherein one of the transcription segments divided from the initial transcription predicted by at least one of the plurality of non-streaming ASR teacher models comprises a blank produced by the at least one of the plurality of non-streaming ASR teacher models.