Semiautomated relay method and apparatus

Abstract

A relay for captioning a hearing user's (HU's) voice signal during a phone call between an HU and a hearing assisted user (AU), the HU using an HU device and the AU using an AU device where the HU voice signal is transmitted from the HU device to the AU device, the relay comprising a display screen, a processor linked to the display and programmed to perform the steps of receiving the HU voice signal from the AU device, transmitting the HU voice signal to a remote automatic speech recognition (ASR) server running ASR software that converts the HU voice signal to ASR generated text, the remote ASR server located at a remote location from the relay, receiving the ASR generated text from the ASR server, present the ASR generated text for viewing by a call assistant (CA) via the display and transmitting the ASR generated text to the AU device.

1 . A relay for captioning a hearing user's (H U′s) voice signal during a phone call between an HU and a hearing assisted user (AU), the HU using an HU device and the AU using an AU device including an AU device display screen, where the HU voice signal is transmitted from the HU device to the AU device, the relay completely separate from the AU device and comprising:
 a relay display screen; 
 a processor linked to the relay display screen and programmed to perform the steps of: 
 receiving the HU voice signal from the AU device, wherein the AU device received the HU voice signal from the HU device; 
 transmitting the HU voice signal to a remote automatic speech recognition (ASR) server running ASR software that converts the HU voice signal to ASR generated text, the remote ASR server located at a remote location from the relay; 
 receiving the ASR generated text from the ASR server; 
 present the ASR generated text for viewing by a call assistant (CA) via the relay display; and 
 transmitting the ASR generated text to the AU device immediately upon receiving the ASR generated text from the ASR. 
 
     
     
         2 . The relay of  claim 1  further including an interface that enables a CA to make changes to the ASR generated text presented on the relay display. 
     
     
         3 . The relay of  claim 2  wherein the processor is further programmed to transmit CA corrections made to the ASR generated text to the AU device with instructions to modify the ASR generated text previously sent to the AU device. 
     
     
         4 . The relay of  claim 1  wherein the relay separates the HU voice signal into voice signal slices, the step of transmitting the HU voice signal to the ASR server includes independently transmitting the voice signal slices to the remote ASR server for captioning and wherein the step of receiving the ASR generated text from the relay includes receiving separate ASR generated text segments for each of the slices and cobbling the separate segments together to form a stream of ASR generated text. 
     
     
         5 . The relay of  claim 4  wherein at least some of the voice signal slices overlap. 
     
     
         6 . The relay of  claim 4  wherein at least some of the voice signal slices are relatively short and some of the voice signal slices are relatively long and wherein the short voice signal slices are consecutive and do not overlap and wherein at least some relatively long voice signal slices overlap at least first and second of the relatively short voice signal slices. 
     
     
         7 . The relay of  claim 5  wherein at least some of the ASR generated text associated with overlapping voice signal slices is inconsistent, the relay applying a rule set to identify which inconsistent ASR generated text to use in the stream of ASR generated text. 
     
     
         8 . The relay of  claim 1  wherein the ASR server generates ASR error corrections for the ASR generated text, the relay further programmed to perform the steps of receiving ASR error corrections, using the error corrections to automatically correct at least some of the errors in the ASR generated text on the relay display screen and transmitting the ASR error corrections to the AU device. 
     
     
         9 . The relay of  claim 8  further including an interface that enables a CA to make changes to the ASR generated text presented on the relay display screen, the processor further programmed to transmit CA corrections made to the ASR generated text to the AU device with instructions to modify the ASR generated text previously sent to the AU device. 
     
     
         10 . The relay of  claim 9  wherein, after a CA makes a change to ASR generated text, the text prior thereto becomes firm so that no ASR error corrections are made to the text subsequent thereto. 
     
     
         11 . The relay of  claim 1  wherein the relay further includes a speaker and wherein the processor broadcasts the HU voice signal to the CA via the speaker as the ASR generated text is presented on the relay display screen. 
     
     
         12 . The relay of  claim 11  wherein the processor aligns broadcast of the HU voice signal with ASR generated text presented on the display screen. 
     
     
         13 . The relay of  claim 11  wherein the processor presents the ASR generated text on the on the display screen immediately upon reception and transmits the ASR generated text immediately upon reception and broadcasts the HU voice signal under control of the CA using an interface. 
     
     
         14 . The relay of  claim 13  wherein, as word in the HU voice signal is broadcast to the CA, text corresponding to the broadcast word in on the display screen is visually distinguished from other text on the display screen. 
     
     
         15 . A relay for captioning a hearing user's (H U′s) voice signal during a phone call between an HU and a hearing assisted user (AU), the HU using an HU device and the AU using an AU device including an AU device display screen where the HU voice signal is transmitted from the HU device to the AU device, the relay comprising:
 a relay display screen; 
 an interface device; 
 a processor linked to the relay display screen and the interface device, the processor programmed to perform the steps of: 
 receiving the HU voice signal from the AU device, wherein the AU device received the HU voice signal from the HU device; 
 separating the HU voice signal into voice signal slices; 
 separately transmitting the HU voice signal slices to a remote automatic speech recognition (ASR) server that is located at a remote location from the relay; 
 receiving separate ASR generated text segments for each of the slices and cobbling the separate segments together to form a stream of ASR generated text; 
 present the stream of ASR generated text as it is received from the ASR server for viewing by a call assistant (CA) via the display; and 
 transmitting the stream of ASR generated text to the AU device as the stream is received from the ASR server. 
 
     
     
         16 . The relay of  claim 15  wherein ASR error corrections to the ASR generated text are received from the ASR server and at least some of the ASR error corrections are used to correct the text on the display, the relay receives CA error corrections to the text on the display and uses those corrections to correct text on the display. Inventors: Robert M. Engelke Serial No.:  17 / 847 , 809  Amendment Page  6   
     
     
         17 . The relay of  claim 16  wherein, once a CA corrects an error in the text on the display, ASR error corrections for text prior to the CA corrected text on the display are not used to make error corrections on the display. 
     
     
         18 . The relay of  claim 17  wherein all ASR generated text presented on the display is transmitted to the AU device and all ASR error corrections and CA text corrections that are presented on the display are transmitted as correction text to the AU device. 
     
     
         19 . An caption device for use by a hard of hearing assisted user (AU) to assist the AU during voice communications with a hearing user (HU) using an HU device, the caption device comprising:
 a display screen;   a memory;   at least one communication link element for linking to a communication network;   a speaker;   a processor linked to each of the display screen, the memory, the speaker and the communication link, the processor programmed to perform the steps of:   receiving an HU voice signal from the HU device during a call;   broadcasting the HU voice signal to the AU via the speaker;   storing at least a most recent portion of the HU voice signal in the memory prior to receiving a command from the AU to start a captioning session;   receiving a command from the AU to start a captioning session;   upon receiving the command, obtaining a text caption corresponding to the stored HU voice signal; and   presenting the text caption to the AU via the display.   
     
     
         20 . The device of  claim 19  wherein the step of obtaining a text caption includes initiating a process whereby an automated speech recognition (ASR) program converts the stored HU voice signal to text. 
     
     
         21 . The device of  claim 20  wherein the processor runs the ASR program. 
     
     
         22 . The device of  claim 21  wherein the step of initiating the process includes establishing a link to a remote relay, and transmitting the stored HU voice signal to the relay, the step of obtaining further including receiving the text caption from the relay. 
     
     
         23 . The device of  claim 19  further including, subsequent to receiving the command, obtaining text captions for additional HU voice signals received during the ongoing call. 
     
     
         24 . The device of  claim 23  wherein the step of obtaining text caption of the stored HU voice signal includes initiating a process whereby the HU voice signal is converted to text via an automatic speech recognition (ASR) engine and wherein the step of obtaining text captions form additional HU voice signal received during the ongoing call further includes transmitting the additional HU voice signal to a relay and receiving text captions back from the relay.