Automated captioning of augmented reality effects in videos

Abstract

Described herein are techniques for generating captions for augmented reality (AR) effects in videos using an adapted multimodal large language model (MLLM). The technique involves sampling frames from base and AR-applied videos, combining them into concatenated frames, and processing them with a hybrid vision encoder. Visual tokens are projected into a language model token space, reshaped, downsampled, and interleaved with text tokens. A fine-tuned large language model processes this input sequence to generate AR effect captions. Optical character recognition extracts text from AR frames, which is combined with generated captions and metadata to produce merged captions and content tags. This approach enables accurate description of temporal AR effects and facilitates downstream applications like search and ranking.

What is claimed is: 
     
         1 . A method for generating a caption for a video with an augmented reality (AR) effect, the method comprising:
 sampling N frames from a base video and N corresponding frames from the base video with the AR effect applied to form N pairs of frames, wherein each pair of frames represents the same point in time;   combining each pair of frames in the N pairs of frames into a concatenated frame to form N concatenated frames, each concatenated frame formed by positioning, side by side, a base video frame and a corresponding frame from the base video with the AR effect applied;   processing the N concatenated frames with a vision encoder to produce visual tokens;   projecting the visual tokens into a language model token space to derive projected tokens;   reshaping the projected tokens from a one-dimensional representation to a two-dimensional representation;   spatially downsampling the reshaped projected tokens to derive downsampled tokens;   interleaving the downsampled tokens with text tokens to form an input sequence;   processing the input sequence using a large language model to generate a caption describing the AR effect applied to the base video.   
     
     
         2 . The method of  claim 1 , wherein processing the N concatenated frames with a vision encoder comprises:
 processing the N concatenated frames using a hybrid vision encoder having a first component trained on diverse text-image data that accepts low resolution inputs and a second component trained on high-quality segmentation datasets that accepts high resolution inputs;   wherein the hybrid vision encoder captures both global and local semantic understanding of the AR effect applied to the base video.   
     
     
         3 . The method of  claim 1 , wherein processing the input sequence using the large language model comprises:
 processing the input sequence using a causal attention mask with a prefix;   generating the caption by predicting tokens only for a portion of the input sequence representing the caption; and   predicting an end-of-sentence token to determine when to stop generating the caption.   
     
     
         4 . The method of  claim 1 , wherein interleaving the downsampled tokens with text tokens to form an input sequence comprises:
 including a text instruction that directs the large language model to:
 compare the base video frame and the corresponding frame with the AR effect applied in each concatenated frame; 
 focus on describing the AR effect visible in the frame with the AR effect applied; 
 provide a detailed description of any animations or dynamic elements of the AR effect; and 
 exclude descriptions of content present in the base video frame unless it relates to a location where the AR effect is applied. 
   
     
     
         5 . The method of  claim 1 , further comprising:
 extracting text from at least one of the N frames from the base video with the AR effect applied using optical character recognition (OCR); and   combining the generated caption, the OCR text, and metadata related to the AR effect, using a generative language model to produce a merged caption and content tags for the AR effect.   
     
     
         6 . The method of  claim 1 , further comprising:
 generating word embeddings from the generated caption using a text encoder;   storing the word embeddings in a database; and   using the word embeddings for at least one of:
 retrieving similar augmented reality effects based on caption similarity, 
 improving augmented reality effect search indexing, or 
 enhancing augmented reality effect ranking in recommendation systems. 
   
     
     
         7 . The method of  claim 1 , further comprising training the large language model, wherein the training of the large language model comprises:
 providing training data comprising pairs of concatenated frames and corresponding human-labeled captions;   processing the concatenated frames with the vision encoder to produce visual tokens;   projecting the visual tokens into the language model token space to derive projected tokens;   interleaving the projected tokens with text tokens representing instructions and captions to form input sequences;   processing the input sequences using the large language model;   computing a loss function for the large language model, wherein the loss function:
 uses a causal attention mask with a prefix for the visual tokens and instruction tokens; and 
 computes loss only for predicted caption tokens and not for the instruction or visual tokens; and 
 updating parameters of the large language model based on the computed loss to improve caption generation performance. 
   
     
     
         8 . A system for generating a caption for a video with an augmented reality (AR) effect, the system comprising:
 at least one processor; and   at least one memory storage decice storing instructions thereon, which, when executed by the at least one processor, cause the system to perform operations comprising:   sampling N frames from a base video and N corresponding frames from the base video with the AR effect applied to form N pairs of frames, wherein each pair of frames represents the same point in time;   combining each pair of frames in the N pairs of frames into a concatenated frame to form N concatenated frames, each concatenated frame formed by positioning, side by side, a base video frame and a corresponding frame from the base video with the AR effect applied;   processing the N concatenated frames with a vision encoder to produce visual tokens;   projecting the visual tokens into a language model token space to derive projected tokens;   reshaping the projected tokens from a one-dimensional representation to a two-dimensional representation;   spatially downsampling the reshaped projected tokens to derive downsampled tokens;   interleaving the downsampled tokens with text tokens to form an input sequence;   processing the input sequence using a large language model to generate a caption describing the AR effect applied to the base video.   
     
     
         9 . The system of  claim 8 , wherein processing the N concatenated frames with a vision encoder comprises:
 processing the N concatenated frames using a hybrid vision encoder having a first component trained on diverse text-image data that accepts low resolution inputs and a second component trained on high-quality segmentation datasets that accepts high resolution inputs;   wherein the hybrid vision encoder captures both global and local semantic understanding of the AR effect applied to the base video.   
     
     
         10 . The system of  claim 8 , wherein processing the input sequence using the large language model comprises:
 processing the input sequence using a causal attention mask with a prefix;   generating the caption by predicting tokens only for a portion of the input sequence representing the caption; and   predicting an end-of-sentence token to determine when to stop generating the caption.   
     
     
         11 . The system of  claim 8 , wherein interleaving the downsampled tokens with text tokens to form an input sequence comprises:
 including a text instruction that directs the large language model to:
 compare the base video frame and the corresponding frame with the AR effect applied in each concatenated frame; 
 focus on describing the AR effect visible in the frame with the AR effect applied; 
 provide a detailed description of any animations or dynamic elements of the AR effect; and 
 exclude descriptions of content present in the base video frame unless it relates to a location where the AR effect is applied. 
   
     
     
         12 . The system of  claim 8 , further comprising:
 extracting text from at least one of the N frames from the base video with the AR effect applied using optical character recognition (OCR); and   combining the generated caption, the OCR text, and metadata related to the AR effect, using a generative language model to produce a merged caption and content tags for the AR effect.   
     
     
         13 . The system of  claim 8 , further comprising:
 generating word embeddings from the generated caption using a text encoder;   storing the word embeddings in a database; and   using the word embeddings for at least one of:
 retrieving similar augmented reality effects based on caption similarity, 
 improving augmented reality effect search indexing, or 
 enhancing augmented reality effect ranking in recommendation systems. 
   
     
     
         14 . The system of  claim 8 , further comprising training the large language model, wherein the training of the large language model comprises:
 providing training data comprising pairs of concatenated frames and corresponding human-labeled captions;   processing the concatenated frames with the vision encoder to produce visual tokens;   projecting the visual tokens into the language model token space to derive projected tokens;   interleaving the projected tokens with text tokens representing instructions and captions to form input sequences;   processing the input sequences using the large language model;   computing a loss function for the large language model, wherein the loss function:
 uses a causal attention mask with a prefix for the visual tokens and instruction tokens; and 
 computes loss only for predicted caption tokens and not for the instruction or visual tokens; and 
 updating parameters of the large language model based on the computed loss to improve caption generation performance. 
   
     
     
         15 . A syste for generating a caption for a video with an augmented reality (AR) effect, the system comprising:
 means for sampling N frames from a base video and N corresponding frames from the base video with the AR effect applied to form N pairs of frames, wherein each pair of frames represents the same point in time;   means for combining each pair of frames in the N pairs of frames into a concatenated frame to form N concatenated frames, each concatenated frame formed by positioning, side by side, a base video frame and a corresponding frame from the base video with the AR effect applied;   means for processing the N concatenated frames with a vision encoder to produce visual tokens;   means for projecting the visual tokens into a language model token space to derive projected tokens;   means for reshaping the projected tokens from a one-dimensional representation to a two-dimensional representation;   means for spatially downsampling the reshaped projected tokens to derive downsampled tokens;   measns for interleaving the downsampled tokens with text tokens to form an input sequence;   means for processing the input sequence using a large language model to generate a caption describing the AR effect applied to the base video.   
     
     
         16 . The system of  claim 15 , wherein the means for processing the N concatenated frames with a vision encoder comprises:
 means for processing the N concatenated frames using a hybrid vision encoder having a first component trained on diverse text-image data that accepts low resolution inputs and a second component trained on high-quality segmentation datasets that accepts high resolution inputs;   wherein the hybrid vision encoder captures both global and local semantic understanding of the AR effect applied to the base video.   
     
     
         17 . The system of  claim 15 , wherein the means for processing the input sequence using the large language model comprises:
 means for processing the input sequence using a causal attention mask with a prefix;   means for generating the caption by predicting tokens only for a portion of the input sequence representing the caption; and   means fo predicting an end-of-sentence token to determine when to stop generating the caption.   
     
     
         18 . The system of  claim 1 , wherein the means for interleaving the downsampled tokens with text tokens to form an input sequence comprises:
 means for including a text instruction that directs the large language model to:
 compare the base video frame and the corresponding frame with the AR effect applied in each concatenated frame; 
 focus on describing the AR effect visible in the frame with the AR effect applied; 
 provide a detailed description of any animations or dynamic elements of the AR effect; and 
 exclude descriptions of content present in the base video frame unless it relates to a location where the AR effect is applied. 
   
     
     
         19 . The system of  claim 15 , the system further comprising:
 means for extracting text from at least one of the N frames from the base video with the AR effect applied using optical character recognition (OCR); and   means for combining the generated caption, the OCR text, and metadata related to the AR effect, using a generative language model to produce a merged caption and content tags for the AR effect.   
     
     
         20 . The system of  claim 15 , the system further comprising:
 means for generating word embeddings from the generated caption using a text encoder;   means for storing the word embeddings in a database; and   means for using the word embeddings for at least one of:
 retrieving similar augmented reality effects based on caption similarity, 
 improving augmented reality effect search indexing, or 
 enhancing augmented reality effect ranking in recommendation systems.