Synchronization and Mixing of Audio and Video Streams in Network-Based Video Conferencing Call Systems

Abstract

In one aspect, audio streams are added to a mix until the mix is either complete (i.e., all audio streams have been added) or the mix is closed early (i.e., before the mix is complete). In another aspect, audio and video streams are synchronized by playing back the audio stream and then synchronizing display of the video frames to the playback of the audio stream.

1 . A computer-implemented method for mixing audio streams from N senders, N at least two, the method comprising:
 receiving the N audio streams over a network, the audio streams divided into audio chunks;   buffering the audio chunks;   opening a mix;   cycling through the N senders, for each sender on each cycle:
 if the sender is not yet in the mix, adding the sender's audio chunk to the mix if the missing audio chunk is available from the buffer; and 
   closing the mix if either audio chunks from all N senders are in the mix or if a predetermined early close condition is met.   
     
     
         2 . The computer-implemented method of  claim 1  wherein:
 the step of opening a mix comprises resetting a counter for the number of senders in the mix; 
 the step of cycling through the N senders further comprises:
 if the sender's audio chunk is added to the mix in a cycle, incrementing the counter; and 
 
 the step of closing the mix further comprises closing the mix if the counter indicates N senders are in the mix. 
 
     
     
         3 . The computer-implemented method of  claim 1  wherein the step of cycling through the N senders further comprises:
 on each cycle, if the sender's audio chunk is already in the mix and the sender has at least one additional audio chunk buffered, incrementing a wait counter for that sender; 
 wherein the predetermined early close condition is based on the value of the wait counter(s). 
 
     
     
         4 . The computer-implemented method of  claim 3  wherein the predetermined early close condition occurs if any wait counter reaches a predetermined maximum value. 
     
     
         5 . The computer-implemented method of  claim 4  wherein the predetermined maximum value is three. 
     
     
         6 . The computer-implemented method of  claim 3  wherein:
 the step of cycling through the N senders further comprises:
 if the sender's audio chunk is added to the mix in a cycle, decrementing the wait counter for that sender if the wait counter is greater than 0. 
 
 
     
     
         7 . The computer-implemented method of  claim 3  wherein:
 the step of opening a mix comprises resetting the wait counters for all senders. 
 
     
     
         8 . The computer-implemented method of  claim 1  wherein each audio chunk contains audio of 40 ms duration. 
     
     
         9 . The computer-implemented method of  claim 1  wherein:
 the step of receiving the N audio streams over a network comprises a central server receiving the N audio streams over a network from sender clients; and 
 the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the central server. 
 
     
     
         10 . The computer-implemented method of  claim 1  wherein:
 the step of receiving the N audio streams over a network comprises a receiver client receiving the N audio streams over a network from a central server, the central server having received the N audio streams from sender clients; and 
 the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the receiver client. 
 
     
     
         11 . The computer-implemented method of  claim 1  wherein:
 the step of receiving the N audio streams over a network comprises a receiver client receiving the N audio streams over a network from sender clients; and 
 the steps of buffering, opening a mix, cycling through the senders and closing the mix occur at the receiver client. 
 
     
     
         12 . A computer-implemented method for mixing audio streams from N senders, N at least two, the method comprising:
 opening a mix;   receiving the N audio streams over a network, the audio streams divided into audio chunks;   as each sender's audio chunk is received:
 if the sender is not yet in the mix and the received audio chunk is the correct audio chunk for the mix, adding the sender's audio chunk to the mix; 
 otherwise, buffering the sender's audio chunk for a future mix; and 
   closing the mix if either audio chunks from all N senders are in the mix or if a predetermined early close condition is met.   
     
     
         13 . The computer-implemented method of  claim 12  wherein:
 the step of opening a mix comprises resetting a counter for the number of senders in the mix; 
 the step of adding a sender's audio chunk to the mix further comprises incrementing the counter; and 
 the step of closing the mix further comprises closing the mix if the counter indicates N senders are in the mix. 
 
     
     
         14 . The computer-implemented method of  claim 12  wherein the step of buffering the sender's audio chunk for a future mix further comprises:
 incrementing a queue counter for that sender; wherein the predetermined early close condition is based on the value of the queue counter(s). 
 
     
     
         15 . The computer-implemented method of  claim 14  wherein the predetermined early close condition occurs if any queue counter reaches a predetermined maximum value. 
     
     
         16 . The computer-implemented method of  claim 14  further comprising:
 upon closing the mix, opening a new mix and determining whether any of the previously buffered audio chunks should be added to the new mix. 
 
     
     
         17 - 22 . (canceled)