Method of processing multimodal tasks, and an apparatus for the same

Abstract

An electronic device may obtain a query from a user input; obtain a sequence of frames of one or more input videos; select frames from the sequence of frames of the one or more input videos, via a sampler neural network configured to extract features from the sequence of frames that are input to the sampler neural network, determine temporal dependencies between the extracted features, and determine an action of selecting or skipping for each of the sequence of frames; and identify a video that matches the query via a multimodal neural network configured to receive the selected frames and the query, and output the video that matches the query, among the one or more input videos, wherein the sampler neural network and the multimodal neural network are jointly trained based on an aggregated loss that combines an accuracy loss that represents an accuracy of determining the video that matches the query, and an efficiency loss that reflects a proportion of frames being passed to the multimodal neural network.

What is claimed is: 
     
         1 . An electronic device comprising:
 at least one memory storing one or more instructions; and   at least one processor configured to execute the one or more instructions to:
 obtain a query from a user input; 
 obtain a sequence of frames of one or more input videos; 
 select frames from the sequence of frames of the one or more input videos, via a sampler neural network configured to extract features from the sequence of frames that are input to the sampler neural network, determine temporal dependencies between the extracted features, and determine an action of selecting or skipping for each of the sequence of frames; and 
 identify a video that matches the query via a multimodal neural network configured to receive the selected frames and the query, and output the video that matches the query, among the one or more input videos, 
   wherein the sampler neural network and the multimodal neural network are jointly trained based on an aggregated loss that combines an accuracy loss that represents an accuracy of determining the video that matches the query, and an efficiency loss that reflects a proportion of frames being passed to the multimodal neural network.   
     
     
         2 . The electronic device of  claim 1 , wherein the multimodal neural network is stored in the electronic device or a server, and
 wherein, when the multimodal neural network is stored in the server, the at least one processor is further configured to execute the one or more instructions to:
 transmit the query and the selected frames to the server; and 
 identify the video that matches the query based on a response received from the server. 
   
     
     
         3 . The electronic device of  claim 1 , further comprises:
 a communication interface configured to receive the user input including the query from an external device, and transmit the video that matches the query to the external device; and   a camera,   wherein the at least one processor is further configured to execute the one or more instructions to obtain the one or more input videos via the communication interface or the camera.   
     
     
         4 . The electronic device of  claim 1 , wherein the at least one processor is further configured to execute the one or more instructions to:
 determine a computational resource of the electronic device;   obtain the sequence of frames by sampling the one or more input videos at an interval corresponding to the computational resource of the electronic device; and   input the sequence of frames to the sampler neural network to determine the action of selecting or skipping for each of the sequence of frames.   
     
     
         5 . The electronic device of  claim 1 , wherein the at least one processor is further configured to execute the one or more instructions to:
 determine a computational resource of the electronic device;   identify the sampler neural network corresponding to the computational resource of the electronic device, among a plurality of sampler neural networks that are trained using difference weights for combining the accuracy loss and the efficiency loss; and   select the frames using the sampler neural network that is identified from the plurality of sampler neural networks based on the computational resource of the electronic device.   
     
     
         6 . The electronic device of  claim 1 , wherein the sampler model is configured to stop providing the selected frames to the multimodal neural network and start providing the selected frames to another multimodal neural network, when the multimodal neural network stops operating and the other multimodal neural network starts running to perform a multimodal task. 
     
     
         7 . The electronic device of  claim 1 , wherein the sampler neural network is further configured to:
 determine the temporal dependencies based on hidden state values that are obtained from the extracted features;   determine the action of selecting or skipping for each of the sequence of frames by inputting the hidden state values to fully connected layers; and   determine the video that matches the query, based on similarity scores between the selected frames and the query.   
     
     
         8 . The electronic device of  claim 1 , wherein the sampler neural network comprises:
 a two-dimensional (2D) convolutional neural network (CNN) configured to extract the features; and   a long short-term memory (LSTM) configured to identify the temporal dependencies between the extracted features.   
     
     
         9 . The electronic device of  claim 1 , wherein the sampler neural network comprises:
 a two-dimensional (2D) convolutional neural network (CNN) configured to extract the features; and   a transformer configured to process the sequence of frames in parallel, and identify the temporal dependencies between the extracted feature through a self-attention mechanism.   
     
     
         10 . The electronic device of  claim 1 , wherein the sampler neural network is further configured to:
 obtain continuous approximation vectors that approximate a distribution of selecting or skipping for each of the frames, as back-propagation data for the sampler neural network and the multimodal neural network.   
     
     
         11 . The electronic device of  claim 1 , wherein the multimodal neural network is further configured to:
 extract the features from the selected frames using a three-dimensional (3D) convolutional neural network (CNN); and   process the features extracted by the 3D CNN via a projection layer and a pooling layer.   
     
     
         12 . A method of processing image data, the method comprising:
 obtaining a query from a user input;   obtaining a sequence of frames of one or more input videos;   selecting frames from the sequence of frames of the one or more input videos, via a sampler neural network configured to extract features from the sequence of frames that are input to the sampler neural network, determine temporal dependencies between the extracted features, and determine an action of selecting or skipping for each of the sequence of frames; and   identifying a video that matches the query via a multimodal neural network configured to receive the selected frames and the query, and output the video that matches the query, among the one or more input videos,   wherein the sampler neural network and the multimodal neural network are jointly trained based on an aggregated loss that combines an accuracy loss that represents an accuracy of determining the video that matches the query, and an efficiency loss that reflects a proportion of frames being passed to the multimodal neural network.   
     
     
         13 . The method of  claim 12 , further comprising:
 determining a computational resource of the electronic device;   obtaining the sequence of frames by sampling the one or more input videos at an interval corresponding to the computational resource of the electronic device; and   inputting the sequence of frames to the sampler neural network to determine the action of selecting or skipping for each of the sequence of frames.   
     
     
         14 . The method of  claim 12 , further comprising:
 determining a computational resource of the electronic device;   identifying the sampler neural network corresponding to the computational resource of the electronic device, among a plurality of sampler neural networks that are trained using difference weights for combining the accuracy loss and the efficiency loss; and   selecting the frames using the sampler neural network that is identified from the plurality of sampler neural networks based on the computational resource of the electronic device.   
     
     
         15 . The method of  claim 12 , further comprising:
 determining the temporal dependencies based on hidden state values that are obtained from the extracted features;   determining the action of selecting or skipping for each of the sequence of frames by inputting the hidden state values to fully connected layers; and   determining the video that matches the query, based on similarity scores between the selected frames and the query.   
     
     
         16 . The method of  claim 12 , further comprising:
 extracting the features via a two-dimensional (2D) convolutional neural network (CNN) included in the sampler neural network; and   identifying the temporal dependencies between the extracted features via a long short-term memory (LSTM) included in the sampler neural network.   
     
     
         17 . The method of  claim 12 , further comprising:
 extracting the features via a two-dimensional (2D) convolutional neural network (CNN) included in the sampler neural network; and   identifying the temporal dependencies between the extracted features via a transformer configured to process the sequence of frames in parallel, and identify the temporal dependencies between the extracted feature through a self-attention mechanism.   
     
     
         18 . The method of  claim 12 , further comprising:
 obtaining continuous approximation vectors that approximate a distribution of selecting or skipping for each of the frames, as back-propagation data for the sampler neural network and the multimodal neural network.   
     
     
         19 . The method of  claim 12 , further comprising:
 extracting the features from the selected frames using a three-dimensional (3D) convolutional neural network (CNN); and   processing the features extracted by the 3D CNN via a projection layer and a pooling layer.   
     
     
         20 . A non-transitory computer readable storage medium which is configured to, when executed by at least one processor, perform a method of processing image data, the method comprising:
 obtaining a query from a user input;   obtaining a sequence of frames of one or more input videos;   selecting frames from the sequence of frames of the one or more input videos, via a sampler neural network configured to extract features from the sequence of frames that are input to the sampler neural network, determine temporal dependencies between the extracted features, and determine an action of selecting or skipping for each of the sequence of frames; and   identifying a video that matches the query via a multimodal neural network configured to receive the selected frames and the query, and output the video that matches the query, among the one or more input videos,   wherein the sampler neural network and the multimodal neural network are jointly trained based on an aggregated loss that combines an accuracy loss that represents an accuracy of determining the video that matches the query, and an efficiency loss that reflects the proportion of frames being passed to the multimodal model.