Semiautomated relay method and apparatus

Abstract

A captioning relay for captioning hearing user (HU) voice signals comprising a plurality of separate captioning resources and a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments, and wherein the number of captioning resources is less than the number of ongoing calls.

What is claimed is: 
     
         1 . An apparatus for assisting a hearing assisted user (AU) during a voice call between the AU and a hearing user (HU) using an HU communication device, the apparatus comprising:
 an AU communication device including a speaker, a microphone, a display screen and a processor, the processor programmed to perform the steps of, during an on-going voice call:   obtaining AU voice signal via the speaker;   transmitting the AU voice signal to the HU device;   receiving HU voice signal from the HU device wherein the HU voice signal includes consecutive HU voice signal segments;   broadcasting the HU voice signal via the speaker;   periodically establishing and maintaining a communication link to a remote call assistant (CA) for a linked segment of the ongoing call where the linked segments are separated by intervening periods during which there is no communication link to the CA;   transmitting at least a subset of the HU voice signal segments to the CA during the linked segments and foregoing transmitting at least a subset of the HU voice signal segments to the CA during the intervening periods;   receiving caption information from the CA for the HU voice signal segments transmitted to the CA; and   using the caption information received from the CA to present captions via the display screen.   
     
     
         2 . The apparatus of  claim 1  wherein the HU does not speak during at least some periods during the on-going call and wherein the step of foregoing transmitting the HU voice signal segments to the CA includes foregoing transmitting the HU voice signal segments during which the HU does not speak. 
     
     
         3 . The apparatus of  claim 2  wherein only a subset of the words spoken by the HU are transmitted to the CA during the on-going call. 
     
     
         4 . The apparatus of  claim 3  further including an automated speech recognition (ASR) engine that receives the HU voice signal, converts the HU voice signal to text, and assigns confidence factors to different segments of the text indicating likelihood that the segments accurately represent associated HU voice signal segments. 
     
     
         5 . The apparatus of  claim 4  wherein the processor links to a CA to transmit only voice signal segments that are associated with low confidence factor captions. 
     
     
         6 . The apparatus of  claim 5  wherein the caption information received from the CA is used to correct errors made by the ASR engine. 
     
     
         7 . The apparatus of  claim 5  wherein captions generated by the ASR engine that correspond to the low confidence factor captions are also transmitted to the CA. 
     
     
         8 . The apparatus of  claim 7  wherein the CA error corrects the ASR engine captions to generate the caption information. 
     
     
         9 . The apparatus of  claim 8  wherein the processor uses the caption information received from the CA to make in line corrections to the captions presented via the display screen. 
     
     
         10 . The apparatus of  claim 1  further including an automatic speech recognition (ASR) engine that receives the HU voice signal and transcribes the received voice signal to captions. 
     
     
         11 . The apparatus of  claim 10  wherein the captions generated by the ASR engine are presented via the display screen. 
     
     
         12 . The apparatus of  claim 11  wherein the captions generated by the ASR are presented via the display screen immediately after the ASR generates those captions. 
     
     
         13 . The apparatus of  claim 12  wherein HU voice signal segments are broadcast via the speaker substantially simultaneously with display of associated ASR generated caption segments. 
     
     
         14 . The apparatus of  claim 11  wherein the processor runs the ASR engine locally on the AU's device. 
     
     
         15 . The apparatus of  claim 1  wherein the processor links to and delinks from the CA automatically without any user action. 
     
     
         16 . The apparatus of  claim 15  wherein quiet segments of the HU voice signal are not transmitted to the CA. 
     
     
         17 . An apparatus for assisting a hearing assisted user (AU) during a voice call between the AU and an HU using an HU communication device, the apparatus comprising:
 an AU communication device including a speaker, a microphone, a display screen and a processor, the processor programmed to perform the steps of, during an on-going voice call:   obtaining AU voice signal via the microphone;   transmitting the AU voice signal to the HU device;   receiving HU voice signal from the HU device, the HU voice signal including signal segments corresponding to phrases spoken by the HU as well as silent segments between spoken phrases;   broadcasting the HU voice signal via the speaker;   transmitting at least a portion of the signal segments that correspond to phrases spoken by the HU to a remote call assistant (CA) where the transmitted signal segments do not include the silent segments;   receiving caption information from the CA for the HU voice signal segments transmitted to the CA; and   using the caption information received from the CA to present captions via the display screen.   
     
     
         18 . The apparatus of  claim 17  wherein the processor further runs an automated speech recognition (ASR) engine that generates ASR generated captions for the HU voice signal and confidence factors for each segment of the ASR generated captions and wherein only HU voice signal segments associated with low confidence ASR generated captions are transmitted to the CA. 
     
     
         19 . The apparatus of  claim 18  wherein the processor is delinked from the CA during periods when no HU voice signal segment is transmitted to the CA and establishes new communication links to the CA when an HU voice signal segment is to be transmitted to the CA. 
     
     
         20 . A method for assisting a hearing assisted user (AU) during a voice call between the AU and an HU using an HU communication device, the method comprising the steps of:
 obtaining AU voice signal;   transmitting the AU voice signal to the HU device;   receiving HU voice signal from the HU device, the HU voice signal including signal segments corresponding to phrases spoken by the HU as well as silent segments between spoken phrases;   broadcasting the HU voice signal via the speaker;   transmitting at least a portion of the signal segments that correspond to phrases spoken by the HU to a remote call assistant (CA) where the transmitted signal segments do not include the silent segments;   receiving caption information from the CA for the HU voice signal segments transmitted to the CA; and   using the caption information received from the CA to present captions via a display screen.