Semiautomated relay method and apparatus

Abstract

A communication method to facilitate communication between first and second users that use first and second devices during a communication session, respectively, includes the steps of: obtaining a first voice signal from the first device during the communication session, using a first transcription process to generate a first caption text associated with the first voice signal, and automatically assessing an accuracy value indicating accuracy of the first caption text. When the accuracy value is below a threshold level, the method also includes obtaining a second voice signal during the communication session, using a second transcription process to generate a second caption text associated with the second voice signal, and presenting the second caption text via a display on a device that is used by the second user.

What is claimed is: 
     
         1 . A communication method to facilitate communication between first and second users that use first and second devices during a communication session, respectively, the method comprising the steps of:
 obtaining a first voice signal from the first device during the communication session;   using a first transcription process to generate a first caption text associated with the first voice signal;   automatically assessing an accuracy value indicating accuracy of the first caption text;   when the accuracy value is below a threshold level:   obtaining a second voice signal during the communication session;   (ii) using a second transcription process to generate a second caption text associated with the second voice signal; and   (iii) presenting the second caption text via a display on a device that is used by the second user.   
     
     
         2 . The communication method of  claim 1  wherein the accuracy value includes a percent of the first caption text that is accurate over a period of time. 
     
     
         3 . The communication method of  claim 1  wherein the step of using a second transcription process to generate a second caption text includes presenting the second voice signal to an automated speech recognition (ASR) engine. 
     
     
         4 . The communication method of  claim 3  wherein the step of using a second transcription process to generate a second caption text further includes receiving call assistant (CA) corrections to the captions generated by the ASR engine to generate the second caption text. 
     
     
         5 . The communication method of  claim 3  wherein the step of using a first transcription process to generate a first caption text associated with the first voice signal includes presenting the first voice signal to an ASR engine. 
     
     
         6 . The communication method of  claim 5  further including presenting the first caption text via the display during at least a first part of the communication session. 
     
     
         7 . The communication method of  claim 5  wherein the step of using a first transcription process to generate a first caption text associated with the first voice signal continues during the communication session while the accuracy value is below the threshold level. 
     
     
         8 . The communication method of  claim 5  further including the step of obtaining a third voice signal via the second device wherein the third voice signal includes speech, and using a third transcription process to generate a third caption text associated with the third voice signal. 
     
     
         9 . The communication method of  claim 8  further including the step of presenting the third caption text via the display. 
     
     
         10 . The communication method of  claim 9  wherein only one or the other of the first and second caption texts is presented via the display at any time during the communication session. 
     
     
         11 . The communication method of  claim 1  wherein the second voice signal is a revoicing of the first voice signal by a call assistant (CA), the second transcription process including using an automated speech recognition (ASR) engine that is trained to the CA's voice to generate ASR caption text. 
     
     
         12 . The communication method of  claim 1  wherein the second device includes a device processor and wherein at least one of the first and second transcription processes is performed by the device processor. 
     
     
         13 . The communication method of  claim 12  wherein the first transcription process is performed by the device processor. 
     
     
         14 . The communication method of  claim 12  for use with a remote captioning relay system that performs at least a portion of the second transcription process. 
     
     
         15 . The communication method of  claim 1  for use with a remote captioning relay system that performs at least a portion of at least one of the first and second transcription processes. 
     
     
         16 . A communication method to facilitate communication between first and second users that use first and second devices during a communication session, the devices for use with a system including a captioning relay that is remote from each of the first and second devices, the method comprising the steps of:
 obtaining a first voice signal from the first device during the communication session wherein the first voice signal includes speech;   providing the first voice signal to the relay;   receiving a first caption text from the relay at the second device, the first caption text corresponding to the first voice signal;   presenting the first caption text via a display on a device used by the second user;   receiving a second voice signal at the second device wherein the second voice signal includes speech;   using an automated speech recognition (ASR) engine at the second device to generate a second caption text corresponding to the second voice signal; and   presenting the second caption text via the display.   
     
     
         17 . The communication method of  claim 16  wherein the step of providing the first voice signal to the relay includes receiving the first voice signal at the second device and transmitting the first voice signal to the relay from the second device. 
     
     
         18 . The communication method of  claim 16  further including the relay using a second ASR engine to generate ASR caption text associated with the first voice signal and receiving corrections to the ASR caption text from a device operated by a call assistant (CA) to generate the second caption text. 
     
     
         19 . The communication method of  claim 18  further including receiving the ASR caption text from the relay and presenting the ASR caption text via the display screen, the step of receiving a first caption text including receiving corrections to the ASR caption text from the relay, the step of presenting the first caption text including making corrections to the presented ASR caption text on the display. 
     
     
         20 . The communication method of  claim 19  wherein the step of making corrections includes making in line corrections to the ASR text on the display. 
     
     
         21 . The communication method of  claim 16  wherein the first caption text includes corrections to a caption text associated with the first voice signal. 
     
     
         22 . The communication method of  claim 16  wherein the second device includes the display. 
     
     
         23 . The communication method of  claim 16  wherein first text captions are presented in a first column on the display and second captions are presented in a second column on the display. 
     
     
         24 . The communication method of  claim 23  wherein first and second user indicators are presented on the display at locations spatially associated with the first and second caption texts. 
     
     
         25 . The communication method of  claim 23  wherein each caption text is provided within a caption field and wherein first caption text and second caption text caption fields are presented progressively from the top of the display toward the bottom with most recent caption text presented lower on the display than more recent captions. 
     
     
         26 . The communication method of  claim 25  wherein each caption field includes a border to distinguish the caption field from other information presented on the display. 
     
     
         27 . A communication method to facilitate communication between first and second users that use first and second devices during a communication session, the method comprising the steps of:
 obtaining a first voice signal from the first device during the communication session wherein the first voice signal includes speech;   obtaining a second voice signal from the second device during the communication session wherein the second voice signal includes speech;   using an automated speech recognition (ASR) engine to generate a first caption text corresponding to the first voice signal without any assistance from a call assistant;   generating a second caption text corresponding to the second voice signal using input from a human call assistant;   presenting the first caption text via a display on a device used by the second user; and   presenting the second caption text via the display.   
     
     
         28 . A communication method to facilitate communication between first and second users that use first and second devices during a communication session, respectively, the method comprising the steps of:
 obtaining a first voice signal from the first device during the communication session;   using a first transcription process to generate a first caption text associated with the first voice signal;   automatically assessing an accuracy value indicating accuracy of the first caption text;   presenting the first caption text via a display on a device that is used by the second user; and   presenting the accuracy value on the display.   
     
     
         29 . A communication method to facilitate communication between first and second users that use first and second devices during a communication session, respectively, the method comprising the steps of:
 obtaining a first voice signal during the communication session;   using a first transcription process to generate a first caption text associated with the first voice signal wherein the first transcription process includes presenting the first voice signal to a first automated speech recognition (ASR) engine at a remote relay and receiving CA input via an interface device to generate the first caption text;   receiving a command via the second device from the second user regarding captioning service;   upon receiving the command via the second device:   (i) using a second transcription process to generate a second caption text associated with a second voice signal, the second transcription process including presenting the second voice signal to a second ASR engine to generate the second caption text; and   (ii) presenting the second caption text via a display on a device that is used by the second user.   
     
     
         30 . The communication method of  claim 29  wherein the second device obtains the second voice signal. 
     
     
         31 . The communication system of  claim 30  wherein the second voice signal is the first user's voice signal which is initially generated by the first device and wherein the first voice signal is a revoicing of the second voice signal by the CA.