Semiautomated relay method and apparatus

Abstract

A captioning relay for captioning hearing user (HU) voice signals comprising a plurality of separate captioning resources and a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments, and wherein the number of captioning resources is less than the number of ongoing calls.

What is claimed is: 
     
         1 . A communication system for enabling communication between an assisted user (AU) and a hearing user (HU) where the HU uses an HU device to communicate via voice with the AU, the system comprising:
 at least a first processor;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
 receiving an HU voice signal that originated at the HU device; 
 providing the HU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor; 
 generating first text captions corresponding to the HU voice signal using the ASR engine; 
 presenting the first text captions via a first display associated with an AU's communication device; 
 automatically determining a first time at which the most recently generated portion of the first text captions fails to meet a first accuracy threshold; 
 only for HU voice signal received after the first time, presenting the HU voice signal to a human call assistant (CA); 
   (ii) generating CA generated text captions associated with the HU voice signal received after the first time based on CA input; and   (iii) sending the CA generated text captions to the AU communication device to be presented on the first display.   
     
     
         2 . The communication system of  claim 1  further including a second display, the at least a first processor further performing operations including, persistently generating first text captions corresponding to the HU voice signal subsequent to the first time that are presented on both the first and second displays on the second display, the CA input including corrections of errors in the captions presented on the second display. 
     
     
         3 . The communication system of  claim 1  wherein the at least a first processor includes a first processor and a second processor, the AU device is a portable computing device, the AU device including the first processor, a relay remote from the AU device including the second processor, the first processor running the ASR engine to receive the HU voice signal originating at the HU device, generate the first text captions, present the first text captions on the first display, and automatically identify the first time. 
     
     
         4 . The communication system of  claim 3  wherein the first processor further performs an operation of transmitting the HU voice signal received after the first time to the remote relay. 
     
     
         5 . The communication system of  claim 4  further including a second display, the first processor further performing operations including, persistently generating first text captions for HU voice signal received subsequent to the first time that are presented on the first display and that are also transmitted after the first time to the relay, the second processor presenting the captions received after the first time on the second display. 
     
     
         6 . The communication system of  claim 5  wherein the CA input includes corrections of errors in the captions presented on the second display. 
     
     
         7 . The communication system of  claim 6  wherein the relay further includes a first speaker, the second processor providing the HU voice signal to the CA by broadcasting the HU voice signal via the first speaker. 
     
     
         8 . The communication system of  claim 7  wherein the AU device further includes a second speaker, the first processor broadcasting the HU voice signal to the AU via the second speaker. 
     
     
         9 . The communication system of  claim 1  wherein the at least a first processor generates confidence factors for captioned text and automatically determines accuracy based on the confidence factors. 
     
     
         10 . The communication system of  claim 9  further including a second display, the at least a first processor further performing operations including, persistently generating first text captions for HU voice signal received subsequent to the first time that are presented on both the first display and the second display and, wherein, low confidence text presented on the second display is visually distinguished from other text presented on the second display. 
     
     
         11 . The communication system of  claim 1  wherein the AU device is a wireless communication device. 
     
     
         12 . The communication system of  claim 3  wherein the AU device further includes a speaker linked to the first processor and wherein the first processor further performs an operation to broadcast the HU voice signal via the speaker. 
     
     
         13 . The communication system of  claim 1  further including a second display and a speaker, the at least a first processor further performing operations including, persistently generating first text captions for HU voice signal received subsequent to the first time that are presented on both the first display and the second display, and broadcasting the HU voice signal received after the first time via the speaker to the CA and, wherein the CA input includes corrections of errors in the captions presented on the second display. 
     
     
         14 . The communication system of  claim 1  wherein the at least a first processor includes first and second processors, the further including a relay remote from the AU device, the relay further including a call assistant (CA) workstation including an interface device, a second memory, and at least a second processor linked to the interface device and the second memory, the second memory having stored thereon software such that, when the software is executed by the at least a second processor, the at least a second processor receives the HU voice signal after the first time, presents the HU voice signal to the CA, generates CA generated text captions associated with the HU voice signal based on CA input via the interface device, and transmits the CA generated text captions to the AU device. 
     
     
         15 . The communication system of  claim 14  wherein the relay further includes a microphone, the CA input including the CA revoicing the HU voice signal into the microphone, the second processor running another ASR trained to the CA's voice to generate CA voice captions. 
     
     
         16 . The communication system of  claim 15  wherein the CA workstation includes a second display, the CA voice captions are presented on the second display, the second processor further receiving CA error corrections via the interface device and generating the CA generated text captions. 
     
     
         17 . The communication system of  claim 1  wherein first text captions are presented via the display regardless of whether or not the first text captions meet the first accuracy threshold. 
     
     
         18 . The communication system of  claim 1  wherein the AU device further includes a selectable input that, when selected at a second time, causes the at least a first processor to present the HU voice signal to the CA to generate CA generated text captions thereafter irrespective of an accuracy of the first text captions prior to the second time. 
     
     
         19 . The communication system of  claim 18  further including a second display wherein, upon selection of the selectable input, the at least a first processor persistently uses the ASR engine to generate first text captions subsequent to the selection and presents the first text captions to the CA via the second display to be error corrected. 
     
     
         20 . A communication system for enabling communication between an assisted user (AU) and a hearing user (HU) where the HU uses and HU device to communicate with the AU, the system comprising:
 an portable wireless AU device including:   a first display;   at least a first processor linked to the first display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
 receiving an HU voice signal originating at the HU device; 
 providing the HU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device; 
 generating first text captions corresponding to the HU voice signal using the ASR engine; 
 presenting the first text captions via the first display; 
 automatically determining a trigger time at which a most recently generated portion of the first text captions fails to meet a first accuracy threshold; 
 only for HU voice signal received after the trigger time, transmitting the HU voice signal to a relay; and 
   a relay remote from the AU device, the relay including a second processor that runs software to perform operations including:
 (i) presenting the HU voice signal to a human call assistant (CA); 
 (ii) generating CA generated text captions for the HU voice signal presented to the CA based on CA input; and 
 (iii) transmitting the CA generated text captions to the AU communication device to be presented on the first display. 
   
     
     
         21 . An assisted user's (AU's) captioned device for use by an assisted user when communicating with a hearing user (HU) using a hearing user's (HU's) device, the AU's device comprising:
 a display;   at least a first processor linked to the display;   a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
 receiving an HU voice signal originating at the HU device; 
 providing the HU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device; 
 generating first text captions corresponding to the HU voice signal using the ASR engine; 
 presenting the first text captions via the display; 
 automatically determining a trigger time at which a most recently generated portion of the first text captions fails to meet a first accuracy threshold; 
 only for HU voice signal received after the trigger time: 
   (i) transmitting the HU voice signal to a relay;   (ii) receiving CA generated text captions corresponding to the HU voice signal transmitted to the relay; and   (iii) presenting the CA generated text captions via the display.   
     
     
         22 . The communication system of  claim 21  wherein the ASR engine continues to generate first text captions after the trigger time where that are presented via the display. 
     
     
         23 . The communication system of  claim 22  wherein the at least a first processor further, only for first text captions corresponding to HU voice signal transmitted to the relay, transmits the first text captions to the relay. 
     
     
         24 . The communication system of  claim 22  wherein the step of presenting the CA generated text captions via the display includes using the CA generated text captions to perform in line corrections to the first text captions previously presented on the display.