Semiautomated relay method and apparatus

Abstract

A captioning relay for captioning hearing user (HU) voice signals comprising a plurality of separate captioning resources and a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments, and wherein the number of captioning resources is less than the number of ongoing calls.

What is claimed is: 
     
         1 . A communication system for enabling communication between an assisted user (AU) and a hearing user (HU) where the HU uses and HU communication device, the system comprising:
 an AU communication device including:   a display;   at least a first processor linked to the display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
 receiving an HU voice signal from the HU device; 
 providing the HU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device; 
 generating first text captions corresponding to the HU voice signal using the ASR engine; 
 automatically determining whether the generated first text captions meet a first accuracy threshold; 
 when the first text captions meet the first accuracy threshold, presenting the first text captions via the display; 
 only when the first text captions fail to meet the first accuracy threshold: 
   (i) transmitting the HU voice signal corresponding to the first text captions to a remote relay;   (ii) receiving second text captions corresponding to at least a subset of the first text captions from the relay; and   (iii) presenting the second text captions via the display.   
     
     
         2 . The communication system of  claim 1  further including the step of, when the first text captions fail to meet the first accuracy threshold, transmitting the first text captions to the remote relay. 
     
     
         3 . The communication system of  claim 2  further including the remote relay, the relay further including a call assistant (CA) workstation including an interface device, a second memory, a second display, and at least a second processor linked to the interface device and the second memory, the second memory having stored thereon software such that, when executed by the at least a second processor, the second processor receives the HU voice signal, presented to the HU voice signal to the CA, presents the first text captions to the CA via the second display, and receives CA error corrections via the interface device to generate the second text captions. 
     
     
         4 . The communication system of  claim 3  wherein the first processor generates confidence factors for captioned text and automatically determines accuracy based on the confidence factors. 
     
     
         5 . The communication system of  4  wherein the second processor visually distinguishes low confidence text words in the first text captions presented to the CA on the second display. 
     
     
         6 . The communication system of  claim 5  wherein the low confidence text words are visually distinguished via highlighting those words in a distinguishing color. 
     
     
         7 . The communication system of  claim 3  wherein the relay further includes a speaker, the second processor providing the HU voice signal to the CA by broadcasting the HU voice signal via the speaker. 
     
     
         8 . The communication system of  claim 1  wherein the AU device is a wireless communication device. 
     
     
         9 . The communication system of  claim 1  wherein the AU device is a smart phone device. 
     
     
         10 . The communication system of  claim 1  wherein the AU device further includes a speaker linked to the first processor and wherein the first processor further performs an operation to broadcast the HU voice signal via the speaker. 
     
     
         11 . The communication system of  claim 1  further including the remote relay, the relay further including a call assistant (CA) workstation including an interface device, a second memory, and at least a second processor linked to the interface device and the second memory, the second memory having stored thereon software such that, when the software is executed by the at least a second processor, the at least a second processor receives the HU voice signal, presents the HU voice signal to the CA, generates second text captions associated with the HU voice signal based on CA input via the interface device, and transmits the second text captions to the AU device. 
     
     
         12 . The communication system of  claim 11  wherein the relay further includes a microphone, the CA input including the CA revoicing the HU voice signal into the microphone, the second processor running another ASR trained to the CA's voice to generate CA voice captions. 
     
     
         13 . The communication system of  claim 11  wherein the CA voice captions are presented on a display screen to the CA, the second processor further receiving CA error corrections via the interface device and generating the second text captions. 
     
     
         14 . The communication system of  claim 1  wherein the first text captions are presented via the display regardless of whether or not the first text captions meet the first accuracy threshold and prior to receiving the second text captions. 
     
     
         15 . The communication system of  claim 14  wherein the second text captions are used to perform in line correction to the first text captions presented via the display. 
     
     
         16 . The communication system of  claim 15  wherein any corrections to the first text captions on the display are visually distinguished from text captions that are corrected. 
     
     
         17 . The communication system of  claim 1  wherein the AU device further includes a microphone for capturing an AU voice signal, the first processor further programmed to perform operations including receiving the AU voice signal, providing the AU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device to generate AU voice signal captions and presenting the AU voice signal captions via the display. 
     
     
         18 . The communication system of  claim 17  further including the step of, when the first text captions fail to meet the first accuracy threshold, transmitting the first text captions to the remote relay. 
     
     
         19 . The communication system of  claim 17  wherein the first text captions are transmitted to the remote relay and the AU voice signal captions are not transmitted to the remote relay. 
     
     
         20 . The communication system of  claim 17  wherein the AU voice signal captions are visually distinguished from the text captions associated with the HU voice signal as they are presented on the display. 
     
     
         21 . The communication system of  claim 20  wherein the AU voice signal captions are presented in a first column on the display and the text captions are presented in a second column on the display. 
     
     
         22 . The communication system of  claim 1  wherein the AU device further includes a selectable input that, when selected, causes the first processor to start transmitting the HU voice signal to the relay for call assistant assisted captioning services. 
     
     
         23 . The communication system of  claim 22  wherein, upon selection of the selectable input, the first processor also transmits the first captions to the relay to be presented to a call assistant. 
     
     
         24 . A communication system including an assisted user's (AU's) captioning device for use by an assisted user when communicating with a hearing user (HU) using a hearing user's (HU's) device, the AU's device comprising:
 a display;   at least a first processor linked to the display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
 receiving an HU voice signal from the HU device; 
 providing the HU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device; 
 generating first text captions corresponding to the HU voice signal using the ASR engine; 
 presenting the first text captions via the display; 
 automatically determining whether the generated first text captions meet a first accuracy threshold; 
 when the first text captions fail to meet thea first accuracy threshold: 
   (i) transmitting the HU voice signal corresponding to the first text captions to a remote relay;   (ii) receiving second text captions corresponding to at least a subset of the first text captions from the relay; and   (iii) presenting the second text captions via the display.   
     
     
         25 . The communication system of  claim 24  further including the step of, when the first text captions fail to meet the first accuracy threshold, transmitting the first text captions to the remote relay. 
     
     
         26 . The communication system of  claim 24  wherein the first processor presents the second text captions via the display by performing in line correction of the first text captions. 
     
     
         27 . A captioning relay system for use with an assisted user's (AU's) captioning device for use by an assisted user when communicating with a hearing user (HU) using a hearing user's (HU's) device, the relay system comprising:
 a display;   at least a first processor linked to the display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor performs operations including:
 receiving an HU voice signal from the AU device; 
 receiving first text captions from the AU device that were generated by an automated speech recognition engine run by an AU device processor; 
 audibly broadcasting the HU voice signal to a call assistant (CA) at the relay; 
 presenting the first text captions on the display for viewing by the CA; 
 receiving input from the CA correcting errors in the first text captions to generate second text captions; and 
 transmitting the second text captions to the AU device. 
   
     
     
         28 . A communication system for enabling communication between an assisted user (AU) and a hearing user (HU) where the HU uses and HU communication device, the system comprising:
 an AU communication device including:   a display;   at least a first processor linked to the display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor performs operations including:
 (i) receiving an HU voice signal from the HU device; 
 (ii) generating first text captions corresponding to the HU voice signal using an automated speech recognition (ASR) engine; 
 (iii) presenting the first text captions via the display; 
 (iv) receiving second text captions corresponding to at least a subset of the HU voice signal; and 
 (v) presenting the second text captions via the display. 
   a relay including at least a second processor and a second memory having stored thereon software such that, when the software is executed by the at least a second processor, the at least a second processor performs operations including, only when the first text captions fail to meet a first accuracy threshold:
 (i) receiving at least a portion of the HU voice signal corresponding to the first text captions; 
 (ii) generating second text captions corresponding to at least a portion of the received HU voice signal; and 
 (iii) transmitting the second text captions to the AU device. 
   
     
     
         29 . The communication system of  28  further including automatically determining whether the generated first text captions meet a first accuracy threshold, the step of receiving at least a portion of the HU voice signal including receiving only HU voice signal corresponding to portions of the first text captions that do not meet the first accuracy threshold. 
     
     
         30 . The communication system of  claim 29  wherein the at least a first processor automatically determines whether the generated first text captions meet the first accuracy threshold and then transmits only portions of the HU voice signal to the relay that correspond to portions of the first text caption that do not meet the first accuracy threshold. 
     
     
         31 . A communication system for enabling communication between an assisted user (AU) and a hearing user (HU) where the HU uses and HU communication device, the system comprising:
 an AU communication device including:   a display;   at least a first processor linked to the display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor performs operations including:
 (i) receiving an HU voice signal from the HU device; 
 (ii) generating first text captions corresponding to the HU voice signal using an automated speech recognition (ASR) engine; 
 (iii) presenting the first text captions via the display; 
 (iv) receiving second text captions from a relay corresponding to at least a subset of the HU voice signal; and 
 (v) presenting the second text captions via the display. 
   a relay including at least a second processor and a second memory having stored thereon software such that, when the software is executed by the at least a second processor, the at least a second processor performs operations including:
 (i) receiving at least a portion of the HU voice signal corresponding to the first text captions; 
 (ii) generating second text captions corresponding to at least a portion of the received HU voice signal; and 
 (iii) transmitting the second text captions to the AU device.