Semiautomated relay method and apparatus

Abstract

A captioning relay for captioning hearing user (HU) voice signals comprising a plurality of separate captioning resources and a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments, and wherein the number of captioning resources is less than the number of ongoing calls.

What is claimed is: 
     
         1 . A captioning relay for captioning hearing user (HU) voice signals of HU's that use HU communication devices to participate in voice communication calls with assisted users (AUs) that use AU communication devices, each call including at least one HU generating voice signals that are transmitted from an HU communication device to at least one AU communication device, the relay comprising:
 (i) a plurality of separate captioning resources, each captioning resource configured to receive voice signal segments and generate captions corresponding to the received voice signal segments, captioning resources that are not currently captioning a voice signal segment being in a standby mode;   (ii) a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and that provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments; and   wherein the number of captioning resources is less than the number of ongoing calls.   
     
     
         2 . The captioning relay of  claim 1  wherein captioning resources that are not currently captioning voice signal segments are in a standby mode and wherein, the administrator module provides voice signal segments to captioning resources that are in the standby mode prior to providing voice signal segments to captioning resources currently captioning voice signal segments. 
     
     
         3 . The captioning relay of  claim 1  wherein the administrator module provides voice signal segments to captioning resources in a round robin fashion. 
     
     
         4 . The captioning relay of  claim 1  wherein each of the captioning resources includes an automated speech recognition (ASR) engine. 
     
     
         5 . The captioning relay of  claim 4  wherein the relay further comprises a plurality of call assistant (CA) workstations where CAs correct errors in in at least some of the ASR generated captions and, wherein, the administrator module provides at least a subset of the caption segments that need to be corrected to CA workstations in a first in, first out order, the administrator module providing each caption signal segment to be corrected from each call to any one of the CA workstations to for correction without regard to which CA workstation corrected prior captions generated during the call. 
     
     
         6 . The captioning relay of  claim 4  wherein each ASR generates caption segments for HU voice signal segments consecutively in the same order that the voice signal segments are received and sends those caption segments back to the administrator module and wherein the administrator module tracks the order of voice signal segments provided to each ASR and relates that order to specific ones of the ongoing calls so that caption segments received back from the ASRs can be associated with specific ones of the ongoing calls. 
     
     
         7 . The captioning relay of  claim 1  wherein each of at least some of the captioning resources includes a call assistant (CA) that listens to voice signal segments and performs at least some process to generate captions corresponding to the voice signal segments. 
     
     
         8 . The captioning relay of  claim 1  wherein the captioning administrator module maintains correlation between the ongoing calls and the captions generated for HU voice signal segments for the ongoing calls. 
     
     
         9 . The captioning relay of  claim 1  wherein the administrator module tracks the number of ongoing calls and automatically activates and deactivates captioning resources as a function of the number of ongoing calls. 
     
     
         10 . The captioning relay of  claim 9  wherein, as the number of ongoing calls increases, the captioning administrator module increases the number of captioning resources. 
     
     
         11 . The captioning relay of  claim 1  wherein at least some of the captioning resources include automated speech recognition (ASR) engines and others include call assistants trained to listen to HU voice signal segments and to generate captions for those voice signal segments. 
     
     
         12 . The captioning relay of  claim 1  wherein the administrator module maintains a separate call log for each of the ongoing calls, the administrator module storing correspondence data that correlates each voice signal segment received for an associated call with one of the caption segments received back from the captioning resources so that the caption segment can be provided to an AU device associated with the call ad presented by the AU device to an AU in the order in which the corresponding voice signal segment was received. 
     
     
         13 . The captioning relay of  claim 1  wherein the administrator module receives streaming HU voice signals related to the ongoing calls and divides those streaming voice signals into the voice signal segments to be captioned. 
     
     
         14 . The captioning relay of  claim 13  wherein the administrator module divides the streaming voice signals into segments based at least in part on when silent periods occur during the streaming voice signals. 
     
     
         15 . The captioning relay of  claim 1  wherein the administrator module receives streaming captions back from each of the captioning resources and divides the streaming captions into call specific segments that are transmitted to AU devices associated with the calls. 
     
     
         16 . The captioning relay of  claim 15  wherein the captioning administrator module transmits silence to each of the standby ASRs to maintain connections to those ASRs while the ASRs remain in standby mode. 
     
     
         17 . The captioning relay of  claim 1  wherein the administrator module inserts segment markers between different voice signal segments prior to providing those segments to the captioning resources, the segment markers including words that are captioned and recognizable by the administrator module as markers when captions are received back from the captioning resources so that the administrator module can separate the captions into caption segments. 
     
     
         18 . A captioning relay for captioning hearing user (HU) voice signals of HU's that use HU communication devices to participate in voice communication calls with assisted users (AUs) that use AU communication devices, each call including at least one HU generating voice signals that are transmitted from an HU communication device to at least one AU communication device, the relay comprising:
 (i) a plurality of separate captioning resources, each captioning resource configured to receive voice signal segments and generate captions corresponding to the received voice signal segments, captioning resources that are not currently captioning a voice signal segment being in a standby mode;   (ii) a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and that provides the voice signal segments in a first in, first out order to the captioning resources in a round robin fashion without regard to which captioning resource captioned prior voice signal segments generated during any of the calls, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments; and   wherein the number of captioning resources is less than the number of ongoing calls.   
     
     
         19 . A captioning relay for captioning hearing user (HU) voice signals of HU's that use HU communication devices to participate in voice communication calls with assisted users (AUs) that use AU communication devices, each call including at least one HU generating voice signals that are transmitted from an HU communication device to at least one AU communication device, the relay comprising:
 (i) a plurality of separate captioning resources, each captioning resource configured to receive voice signal segments and generate captions corresponding to the received voice signal segments; and   (ii) a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and that provides the voice signal segments to the captioning resources to be captioned, the administrator module increasing and decreasing the number of captioning resources available to caption voice signal segments and maintaining the number of captioning resources below the number of ongoing calls.   
     
     
         20 . The captioning relay of  claim 19  wherein the number of captioning resources available is maintained within a range between 40% and 80% of the total number of ongoing calls.