Synchronizing secondary audiovisual content based on frame transitions in streaming content

Abstract

According some aspects, a secondary device may display secondary audiovisual content along with playback of audiovisual content on a primary device. For example, the secondary device may display an augmented reality application synchronized with the video. Aspects may predetermine a set of frame transition ranges for the video, where each respective frame transition is determined based on frames of the video that are determined to be substantially identical by a frame reference function and frames that are determined to be different. Two frames may be substantially identical even if they are different in the source video. This may be due to shortcomings in the frame reference function, or encoding/compression losses in transmission and playback of the video. Playback may be synchronized based on a first detected frame, but synchronization may be refined upon detecting a frame transition to a second frame that is no longer substantially identical to prior frames.

1 . (canceled) 
     
     
         2 . (canceled) 
     
     
         3 . (canceled) 
     
     
         4 . A method of determining a set of frame transition ranges for a video, wherein the set of frame transition ranges are used by a user device to control synchronized display of secondary audiovisual content associated with the video, the method comprising:
 processing frames of the video to determine a plurality of series of contiguous frames that are deemed substantially identical by a predefined frame reference function;   determining the set of frame transition ranges for the video based on the plurality of series of contiguous frames, wherein a given frame transition range corresponds to a respective series of contiguous frames that are deemed substantially identical by the predefined frame reference function and comprises:
 a respective starting frame identifier of the frame transition range, corresponding to the first frame of the series of contiguous frames that differs from the frames of a prior transition range based on the predetermined frame reference function, and 
 a respective ending frame identifier of the frame transition range, corresponding to the last frame of the series of contiguous frames prior to a different frame of a next transition range based on the predetermined frame reference function; and 
   transmitting the set of frame transition ranges in association with the video.   
     
     
         5 . The method of  claim 4 , wherein the video is a streaming video. 
     
     
         6 . The method of  claim 5 , further comprising:
 generating the streaming video, wherein the streaming video comprises a compressed version of an original video.   
     
     
         7 . The method of  claim 5 , further comprising:
 generating the streaming video, wherein the streaming video comprises a lower resolution version of an original video.   
     
     
         8 . The method of  claim 5 , wherein the predefined frame reference function determines the contiguous series of frames to be substantially identical based on at least one difference, between the streaming video and a corresponding original video, that was caused by generating the streaming video. 
     
     
         9 . The method of  claim 5 , wherein the predefined frame reference function determines the contiguous series of frames to be substantially identical based on compression artifacts associated with generating the streaming video. 
     
     
         10 . The method of  claim 5 , wherein the predefined frame reference function determines the contiguous series of frames to be substantially identical based on compression artifacts associated with transmitting the streaming video. 
     
     
         11 . The method of  claim 5 , wherein:
 a first frame and a second frame of the streaming video are deemed substantially identical by the predefined frame reference function, and   a corresponding first frame and a corresponding second frame of a corresponding original video are not substantially identical.   
     
     
         12 . The method of  claim 4 , wherein processing the frames of the video to determine a given of series of contiguous frames that are deemed substantially identical by a predefined frame reference function comprises:
 processing a first frame of the video and a subsequent second frame of the video using a predefined frame reference function to determine whether the first frame and the second frame are deemed substantially identical by the predefined frame reference function;   based on determining that the first frame and the second frame are deemed substantially identical by the predefined frame reference function, continuing to process one or more next frames subsequent to the second frame to determine whether each next frame is deemed substantially identical to a prior frame by the predefined frame reference function; and   determining a last frame of the series of contiguous frames based on determining that a next frame is not deemed substantially identical to a prior frame by the predefined frame reference function,   wherein the series of contiguous frames includes each frame deemed substantially identical by the predefined frame reference function.   
     
     
         13 . The method of  claim 4 , wherein the predefined frame reference function is provided by an augmented reality toolkit and is configured to compare a portion of a captured image to a reference image. 
     
     
         14 . The method of  claim 4 , wherein each frame in a respective frame transition range is determined to be substantially identical to each other frame in the frame transition range based on the predefined frame reference function determining that the frames are identical. 
     
     
         15 . The method of  claim 4 , wherein the secondary audiovisual content comprises an augmented reality application corresponding to the video. 
     
     
         16 . A method of determining a set of frame transitions for a video, wherein the set of frame transitions are used by a user device to control synchronized display of secondary audiovisual content associated with the video, the method comprising:
 processing frames of the video to determine a plurality transition frames that are deemed to be different from a prior frame based on the predefined frame reference function;   determining the set of frame transitions for the video based on the plurality of transition frames, wherein a given frame transition corresponds to a respective series of contiguous frames that are deemed substantially identical by the predefined frame reference function and comprises:
 a respective starting frame identifier of the frame transition, corresponding to the first frame of the series of contiguous frames that differs from the frames of a prior series of contiguous frames based on the predetermined frame reference function; and 
   transmitting the set of frame transition ranges in association with the video.   
     
     
         17 . The method of  claim 16 , wherein the video is a streaming video, wherein the streaming video comprises a compressed version or a lower resolution version of an original video. 
     
     
         18 . The method of  claim 16 , wherein the predefined frame reference function determines the contiguous series of frames to be substantially identical based on at least one difference, between the streaming video and a corresponding original video, that was caused by generating the streaming video. 
     
     
         19 . The method of  claim 16 , wherein the predefined frame reference function determines the contiguous series of frames to be substantially identical based on compression artifacts associated with generating the streaming video or associated with transmitting the streaming video. 
     
     
         20 . The method of  claim 16 , wherein:
 the secondary audiovisual content comprises an augmented reality application corresponding to the video, and   the predefined frame reference function is provided by an augmented reality toolkit, associated with the augmented reality application, and is configured to compare a portion of a captured image to a reference image.   
     
     
         21 . A method of determining a set of frame transition ranges for an audio track, wherein the set of frame transition ranges are used by a user device to control synchronized display of secondary audiovisual content associated with the audio track, the method comprising:
 processing audio frames of the audio track to determine a plurality of series of contiguous frames that are deemed substantially identical by a predefined frame reference function, wherein processing a given audio frame of the audio track comprises generating, as the given audio frame, a spectrogram based on a first portion of the audio track;   determining the set of frame transition ranges for the audio track based on the plurality of series of contiguous frames, wherein a given frame transition range corresponds to a respective series of contiguous frames that are deemed substantially identical by the predefined frame reference function and comprises:
 a respective starting frame identifier of the frame transition range, corresponding to the first frame of the series of contiguous frames that differs from the frames of a prior transition range based on the predetermined frame reference function, and 
 a respective ending frame identifier of the frame transition range, corresponding to the last frame of the series of contiguous frames prior to a different frame of a next transition range based on the predetermined frame reference function; and 
   transmitting the set of frame transition ranges in association with the audio track.   
     
     
         22 . The method of  claim 21 , wherein the audio track is a streaming audio track, wherein the streaming audio track comprises a compressed version or a lower resolution version of an original audio track. 
     
     
         23 . The method of  claim 21 , wherein the predefined frame reference function determines the contiguous series of frames to be substantially identical based on at least one difference, between the streaming audio track and a corresponding original audio track, that was caused by generating the streaming audio track.