Caption assisted calling to maintain connection in challenging network conditions

Abstract

Systems are provided for managing and coordinating STT/TTS systems and the communications between these systems when they are connected in online meetings and for mitigating connectivity issues that may arise during the online meetings to provide a seamless and reliable meeting experience with either live captions and/or rendered audio. Initially, online meeting communications are transmitted over a lossy connectionless type protocol/channel. Then, in response to detected connectivity problems with one or more systems involved in the online meeting, which can cause jitter or packet loss, for example, an instruction is dynamically generated and processed for causing one or more of the connected systems to transmit and/or process the online meeting content with a more reliable connection/protocol, such as a connection-oriented protocol. Codecs at the systems are used, when needed to convert speech to text with related speech attribute information and to convert text to speech.

What is claimed is: 
     
       1. A server comprising:
 one or more processors; and 
 one or more hardware storage devices storing computer-readable instructions that are executable by the one or more processors to configure the server to: 
 receive first online meeting content over a first channel from a transmitting client system that comprises audio data obtained by the transmitting client system and that is encoded and transmitted by the transmitting client system to the server using a first protocol and in an audio data format; 
 transmit the first online meeting content received from the transmitting client system over the first channel to a receiving client system; and 
 generate and send the transmitting client system an instruction for activating one or more codecs at the transmitting client system, subsequent to receiving the first online meeting content, to initiate transforming and encoding of audio obtained from the online meeting by the transmitting client system into text data and associated speech attribute information, the one of more codecs having speech-to-text functionality. 
 
     
     
       2. The server of  claim 1 , wherein the computer-readable instructions are further executable by the one or more processors to configure the server to further receive new online meeting content from the transmitting client as text data with the associated speech attribute information over a second channel that has a lower bitrate and a different protocol than used by the first channel, the new meeting content being received after sending the instruction, the text data having been converted by the one or more codecs at the transmitting client system from audio into the text data in response to the instruction. 
     
     
       3. The server of  claim 1 , wherein the computer-readable instructions are further executable by the one or more processors to configure the server to transcode the first online meeting content into text at the server. 
     
     
       4. The server of  claim 1 , wherein the computer-readable instructions are further executable by the one or more processors to configure the server to transmit the first online meeting content received from the transmitting client to the receiving client system using the first protocol and without transcoding the first online meeting content from the audio data format into a text data format. 
     
     
       5. The server of  claim 1 , wherein the computer-readable instructions are further executable by the one or more processors to configure the server to transmit the first online meeting content received from the transmitting client to the receiving client system using a second protocol that is less lossy than the first protocol and after first transcoding the first online meeting content from the audio data format into a text data format. 
     
     
       6. The server of  claim 1 , wherein the computer-readable instructions are further executable by the one or more processors to configure the server to generate and send the receiving client system a different instruction for activating the one or more codecs at the receiving client system to initiate decoding of online meeting content with a codec having text-to-speech functionality. 
     
     
       7. The server of  claim 1 , wherein the generating of the instruction is automatically triggered in response to the server detecting one or more connectivity issues in the electronic communications between the server and the transmitting and/or receiving client systems that negatively impact quality for rendering content of the online meeting at the second client system. 
     
     
       8. The server of  claim 7 , wherein the detected connectivity issues include packet loss. 
     
     
       9. The server of  claim 7 , wherein detected connectivity issues include jitter. 
     
     
       10. The server of  claim 7 , wherein detection of the connectivity issues comprises the server detecting user input entered by a user participating in the online meeting that identifies a connectivity problem. 
     
     
       11. The server of  claim 10 , wherein the user input is received from the receiving client system. 
     
     
       12. The server of  claim 7 , wherein the detection of the connectivity issues comprises the server detecting a change in bandwidth availability for transmitting communications between the server and the transmitting client system. 
     
     
       13. The server of  claim 7 , wherein the detection of the connectivity issues comprises the server detecting a change in bandwidth availability for transmitting communications between the server and the receiving client system. 
     
     
       14. A computing system configured as a receiving client system that receives content during an online meeting, the computing system comprising:
 one or more processors; and 
 one or more hardware storage devices storing computer-readable instructions that are executable by the one or more processors to configure the computing system to: 
 receive and decode first online meeting content during an online meeting between the computing system and a remote transmitting computing system, the first online meeting content being received from a server interposed between the computing system and the remote transmitting computing system in an audio data format; 
 identifying an instruction, subsequent to receiving the first online meeting content during the online meeting, for activating one or more codecs having text to speech functionality capable of decoding online meeting content received in a text format; 
 activate the one or more codecs in response to the instruction; and 
 receive and decode new online meeting content for the online meeting with the one or more codecs having the text to speech functionality, the new online meeting content being received in a text format with speech attribute information associated with the text. 
 
     
     
       15. The computing system of  claim 14 , wherein the computer-executable instructions are further executable by the one or more processors to cause the computing system to render of the new online meeting content after it is decoded with the one or more codecs in the text format on a display device. 
     
     
       16. The computing system of  claim 14 , wherein the instruction is received from the server. 
     
     
       17. The computing system of  claim 14 , wherein the instruction is generated by the computing system. 
     
     
       18. The computing system of  claim 14 , wherein the computer-readable instructions are further executable by the one or more processors to configure the computing system to generate a notification for triggering the creation of the instructions in response to detecting connectivity issues that negatively affect transmission or rendering of the first online meeting content. 
     
     
       19. A computing system configured as a transmitting client system that transmits content during an online meeting, the computing system comprising:
 one or more processors; and 
 one or more hardware storage devices storing one or more computer-readable instructions that are executable by the one or more processors to configure the computing system to mitigate connectivity issues during an online meeting, and by at least configuring the computing system to: 
 generate and transmit first online meeting content for the online meeting in an audio data format; 
 identify an instruction, subsequent to transmitting the first online meeting content of the online meeting, for activating one or more codecs to initiate transcoding of new online meeting content for the online meeting from the audio format into a text format, the one or more codecs having speech-to-text functionality; 
 activate the one or more codecs in response to the instruction; 
 identify new online meeting content for the online meeting comprising audio and using the one or more codecs to convert the new online meeting content into text and corresponding speech attribute information; and 
 transmit the new online meeting content for the online meeting to the server. 
 
     
     
       20. The computing system of  claim 19 , wherein the instruction is received from the server.