Video matching with a messaging application

Abstract

Aspects of the present disclosure involve a system and a method for performing operations comprising: identifying a plurality of features for frames of a video received by a messaging application server; assigning a first sequence of a first subset of the plurality of features and a second sequence of a second subset of the plurality of features respectively to a first nearest visual codebook cluster and a second nearest visual codebook cluster; applying the first and second nearest visual codebook clusters to a visual search database to identify a plurality of candidate matching videos; selecting a given matching video from the plurality of candidate matching videos based on a rank representing similarity of the given matching video to the video received by the messaging application server; and accessing an augmented reality experience corresponding to the given matching video.

What is claimed is: 
     
       1. A method comprising:
 identifying a plurality of features for frames of a video received by an application server; 
 assigning a first sequence of a first subset of the plurality of features and a second sequence of a second subset of the plurality of features respectively to a first visual codebook cluster and a second visual codebook cluster; 
 applying the first and second visual codebook clusters to a visual search database to identify a plurality of candidate matching videos, the plurality of candidate matching videos being identified in response to computing cosine similarity between term frequency information associated with the video and term frequency information associated with a plurality of marker videos, the plurality of candidate matching videos being a first subset of the plurality of marker videos having a greater cosine similarity than a second subset of the plurality of marker videos; 
 selecting a given matching video from the plurality of candidate matching videos based on a rank representing similarity of the given matching video to the video received by the application server; and 
 accessing an augmented reality experience corresponding to the given matching video. 
 
     
     
       2. The method of  claim 1 , further comprising computing a root scale-invariant feature transform (SIFT) based on the identified plurality of features of the video. 
     
     
       3. The method of  claim 1 , further comprising:
 identifying, based on sequences of different subsets of the plurality of features, a plurality of descriptors and corresponding key points for the video; and 
 applying the plurality of descriptors and the corresponding key points to a visual word codebook and document frequency index, generated based on training videos, to compute a visual word histogram and the term frequency information comprising a term frequency-inverse document frequency (TF-IDF) vector for the video. 
 
     
     
       4. The method of  claim 3 , wherein the first sequence of the first subset of the plurality of features is assigned to the first visual codebook cluster based on the visual word histogram. 
     
     
       5. The method of  claim 3 , wherein applying the first and second visual codebook clusters to the visual search database comprises:
 accessing a visual search database that represents a TF-IDF vector of visual words for a plurality of marker videos, wherein the term frequency information associated with the video comprises the TF-IDF vector of the video and the term frequency information associated with the plurality of marker videos comprises the TF-IDF vector of the plurality of marker videos. 
 
     
     
       6. The method of  claim 1 , wherein a first marker video of the plurality of candidate matching videos comprises a first sequence of markers across a subset of frames of the first marker video, and wherein a second marker video of the plurality of marker videos comprises a second sequence of markers across a subset of frames of the second marker video. 
     
     
       7. The method of  claim 1 , further comprising:
 obtain a set of training videos comprising marker videos and distractor videos; and 
 calculating a root scale-invariant feature transform (SIFT) vector comprising descriptors and key points for the set of training videos. 
 
     
     
       8. The method of  claim 7 , wherein the marker videos comprise videos having a sequence of predefined objects that is associated with a respective augmented reality experience, and wherein the distractor videos comprise random videos that are publicly available. 
     
     
       9. The method of  claim 7 , further comprising resizing and cropping a duration of the set of training videos to correspond to a common size and common duration. 
     
     
       10. The method of  claim 7 , further comprising:
 training a k-means index based on the root SIFT vector, wherein a plurality of centroids of the k-means index represent respectively a plurality of visual words. 
 
     
     
       11. The method of  claim 10 , further comprising:
 identifying a first closest centroid of the plurality of centroids for a first sequence of key points of a given training video; 
 assigning a first descriptor corresponding to the first sequence of key points to the first closest centroid representing a first visual word; 
 identifying a second closest centroid of the plurality of centroids for a second sequence of key points of the given training video; and 
 assigning a second descriptor corresponding to the second sequence of key points to the second closest centroid representing a second visual word. 
 
     
     
       12. The method of  claim 11 , further comprising generating a visual word codebook comprising the first visual codebook cluster and the second visual codebook cluster using the plurality of centroids. 
     
     
       13. The method of  claim 11 , further comprising:
 computing count values for each of the plurality of visual words for each of the set of training videos, each of the count values represents a total number of descriptors in a training video of the set of training videos that is assigned to a given one of the plurality of visual words. 
 
     
     
       14. The method of  claim 13 , further comprising:
 computing a document frequency of each of the plurality of visual words that represents frequency of the plurality of visual words across the set of training videos. 
 
     
     
       15. The method of  claim 14 , further comprising:
 obtaining the marker videos from the set of training videos; and 
 generating, based on the computed document frequency, a sparse matrix representing a term frequency-inverse document frequency (TF-IDF) vector of each of the plurality of visual words associated with the descriptors of the marker videos. 
 
     
     
       16. The method of  claim 15 , wherein a first video of the set of training videos comprises a first sequence of a given set of features, and wherein a second video of the set of training videos comprises a second sequence of the given set of features. 
     
     
       17. The method of  claim 16 , wherein the first video is associated with a first augmented reality experience and the second video is associated with a second augmented reality experience that is related to the first augmented reality experience. 
     
     
       18. The method of  claim 1 , wherein each of the plurality of candidate matching videos is associated with a different type of augmented reality experience. 
     
     
       19. A system comprising:
 a processor configured to perform operations comprising: 
 identifying a plurality of features for frames of a video received by an application server; 
 assigning a first sequence of a first subset of the plurality of features and a second sequence of a second subset of the plurality of features respectively to a first visual codebook cluster and a second visual codebook cluster; 
 applying the first and second visual codebook clusters to a visual search database to identify a plurality of candidate matching videos, the plurality of candidate matching videos being identified in response to computing cosine similarity between term frequency information associated with the video and term frequency information associated with a plurality of marker videos, the plurality of candidate matching videos being a first subset of the plurality of marker videos having a greater cosine similarity than a second subset of the plurality of marker videos; 
 selecting a given matching video from the plurality of candidate matching videos based on a rank representing similarity of the given matching video to the video received by the application server; and 
 accessing an augmented reality experience corresponding to the given matching video. 
 
     
     
       20. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
 identifying a plurality of features for frames of a video received by an application server; 
 assigning a first sequence of a first subset of the plurality of features and a second sequence of a second subset of the plurality of features respectively to a first visual codebook cluster and a second visual codebook cluster; 
 applying the first and second visual codebook clusters to a visual search database to identify a plurality of candidate matching videos, the plurality of candidate matching videos being identified in response to computing cosine similarity between term frequency information associated with the video and term frequency information associated with a plurality of marker videos, the plurality of candidate matching videos being a first subset of the plurality of marker videos having a greater cosine similarity than a second subset of the plurality of marker videos; 
 selecting a given matching video from the plurality of candidate matching videos based on a rank representing similarity of the given matching video to the video received by the application server; and 
 accessing an augmented reality experience corresponding to the given matching video.