Semiautomated relay method and apparatus

Abstract

A captioning relay for captioning hearing user (HU) voice signals comprising a plurality of separate captioning resources and a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments, and wherein the number of captioning resources is less than the number of ongoing calls.

What is claimed is: 
     
         1 . A captioning method for providing captioning services to an assisted user (AU) using an AU device to communicate via voice with a hearing user using a HU device, the AU device including a display for presenting captions corresponding to voice signal during an AU-HU communication session, the method for use with a captioning system including at least a first captioning resource type required to facilitate a first captioning process and a second captioning resource type required to facilitate a second captioning process, wherein the first captioning process is a revoicing captioning process, the method comprising the steps of, during a first AU-HU communication session:
 receiving a first voice signal;   determining a first availability of the first captioning resource type;   selecting, based at least in part on the first availability of the first captioning resource type, a first resource of the first captioning resource type instead of a resource of the second captioning resource type to generate a transcript of the first voice signal;   obtaining revoiced audio generated by a revoicing of the first voice signal by a captioning assistant (CA);   generating first caption text corresponding to the revoiced first voice signal using a first automatic speech recognition (ASR) system; and   transmitting the first caption text to the AU device.   
     
     
         2 . The captioning method of  claim 1  further including, during a second AU-HU communication session that occurs subsequent to the first AU-HU communication session, receiving a second voice signal, determining a second availability of the first captioning resource type, selecting, based at least in part on the second availability of the first captioning resource type, a second resource of the second captioning resource type instead of a resource of the first captioning resource type, to generate second caption text corresponding to the second voice signal. 
     
     
         3 . The captioning method of  claim 2  wherein the second captioning process includes the steps of providing a voice signal to a second ASR that is trained differently than the first ASR. 
     
     
         4 . The captioning method of  claim 3  wherein the second captioning process is also a revoicing captioning process. 
     
     
         5 . The captioning method of  claim 2  wherein the second caption text is transmitted to the AU device as captions of the second voice signal. 
     
     
         6 . The captioning method of  claim 2  wherein the second AU-HU communication session is between the AU device and a second HU device. 
     
     
         7 . The captioning method of  claim 1  wherein the captioning resources are located at a relay remote from the AU device and the HU device, the method further including transmitting the first voice signal from at least one of the AU and HU devices to the relay. 
     
     
         8 . The captioning method of  claim 1  wherein the first voice signal is an HU voice signal and does not include an AU voice signal. 
     
     
         9 . The captioning method of  claim 1  wherein the ASR is trained specifically for the voice of the CA. 
     
     
         10 . The captioning method of  claim 1  wherein the second captioning process includes providing a voice signal to a second ASR to generate captions. 
     
     
         11 . The captioning method of  claim 1  wherein the first captioning resource type includes a CA that specializes in handling HU voice signals with a specific characteristic. 
     
     
         12 . The captioning method of  claim 1  further including presenting the first voice signal to a second ASR engine that generates second text captions that are transmitted to the AU device. 
     
     
         13 . A captioning method for providing captioning services to an assisted user (AU) using an AU device to communicate via voice with a hearing user using a HU device, the AU device including a display for presenting captions corresponding to voice signal during an AU-HU communication session, the method for use with a captioning system including at least a first captioning resource type required to facilitate a first captioning process and a second captioning resource type required to facilitate a second captioning process, wherein the second captioning process is a revoicing captioning process, the method comprising the steps of, during a first AU-HU communication session:
 receiving a first voice signal;   determining a first availability of the second captioning resource type;   selecting, based at least in part on the first availability of the second captioning resource type, a first resource of the first captioning resource type instead of a resource of the second captioning resource type to generate a transcript of the first voice signal;   generating first caption text corresponding to the first voice signal using a first automatic speech recognition (ASR) system associated with the selected first resource; and   transmitting the first caption text to the AU device.   
     
     
         14 . The captioning method of  claim 13  wherein, during the first AU-HU communication session, a second resource of the second captioning resource type becomes available, the method further including, in response to the second resource becoming available, switching from the first resource to the second resource to generate the first caption text which is then transmitted to the AU device. 
     
     
         15 . The captioning method of  claim 14  wherein the method further includes, in response to the second resource becoming available, providing the first voice signal to the second resource, obtaining, by the second resource, revoiced audio generated by a revoicing of the first voice signal by a call assistant (CA), generating first caption text of the revoiced audio using an automatic speech recognition (ASR) engine, and transmitting the first caption text to the AU device. 
     
     
         16 . The captioning method of  claim 13  wherein the first resource includes an ASR for converting a voice signal to the first caption text. 
     
     
         17 . The captioning method of  claim 13  wherein the captioning resources are located at a relay remote from the AU device and the HU device. 
     
     
         18 . A captioning method for providing captioning services to an assisted user (AU) using an AU device to communicate via voice with a hearing user using a HU device, the AU device including a display for presenting captions corresponding to voice signal during an AU-HU communication session, the method for use with a captioning system including at least a first captioning resource type required to facilitate a first captioning process and a second captioning resource type required to facilitate a second captioning process, wherein the first captioning process is a revoicing captioning process, the method comprising the steps of, during a first AU-HU communication session:
 receiving a first voice signal;   determining a first availability of the first captioning resource type;   based at least in part on the first availability of the first captioning resource type, selecting one of:   (i) a first resource of the first captioning resource type instead of a resource of the second captioning resource type to generate text captions of the first voice signal; and   (ii) a second resource of the second captioning resource type to generate text captions of the first voice signal;   in response to selecting the first resource:
 (a) obtaining revoiced audio generated by a revoicing of the first voice signal by a captioning assistant (CA); 
 (b) generating first caption text corresponding to the revoiced first voice signal using a first automatic speech recognition (ASR) engine; and 
   transmitting the first caption text to the AU device; and   in response to selecting the second resource:
 (a) generating first caption text corresponding to the first voice signal using a second ASR engine. 
   
     
     
         19 . The method of  claim 18  wherein the first caption text is transmitted to the AU device for display. 
     
     
         20 . The method of  claim 18  wherein the first and second ASR engines are different. 
     
     
         21 . The method of  claim 18  wherein the second captioning process further includes obtaining revoiced audio generated by a revoicing of the first voice signal by a second captioning assistant (CA) and generating first caption text corresponding to the revoiced first voice signal by the second captioning assistant using the second ASR engine.