Method and apparatus that controls augmented reality (AR) apparatus based on action prediction

Abstract

A method and apparatus for controlling an augmented reality (AR) apparatus are provided. The method includes acquiring a video, detecting a human body from the acquired video, performing an action prediction with regard to the detected human body, and controlling the AR apparatus based on a result of the action prediction and a mapping relationship between human body actions and AR functions.

What is claimed is: 
     
       1. A method of controlling an augmented reality (AR) apparatus, the method comprising:
 acquiring a video; 
 detecting a human body from the acquired video; 
 performing an action prediction with regard to the detected human body; and 
 controlling the AR apparatus based on a result of the action prediction and a mapping relationship between human body actions and AR functions, 
 wherein the performing the action prediction further includes: 
 generating a human body part parsing result based on a convolution of an approximate semantic segmentation result and a direction prediction result; and 
 performing an action classification with regard to the human body actions based on a frame-based local feature image, a human body pose feature, and the human body part parsing result, 
 wherein the performing of the action classification comprises: 
 determining an image area of a human body instance in an image frame of the acquired video; 
 performing an approximate partial semantic segmentation on a human body image area candidate extracted from the image area, and generating an approximate semantic segmentation result corresponding to detailed parts of the human body; 
 predicting a relative direction of each pixel in the image frame with respect to a central portion of the human body and generating the direction prediction result; 
 performing the convolution based on the approximate semantic segmentation result and the direction prediction result and generating the human body part parsing result; and 
 performing the action classification with the first action classification scheme based on the frame-based local feature image, the human body pose feature and the human body part parsing result. 
 
     
     
       2. The method of  claim 1 , wherein the acquired video is one or more of a video of the AR apparatus and a user of the AR apparatus captured by a camera distinguished from the AR apparatus, a video captured from a viewpoint of the user of the AR apparatus with a camera connected to the AR apparatus, and a video generated and stored in advance in the AR apparatus. 
     
     
       3. The method of  claim 1 , wherein in response to the acquired video being a video of the AR apparatus and a user of the AR apparatus captured by a camera distinguished from the AR apparatus,
 the detecting of the human body comprises:
 recognizing the user corresponding to the detected human body; 
 associating the user with the AR apparatus based on an AR user database (DB); and 
 establishing a communication with the AR apparatus. 
 
 
     
     
       4. The method of  claim 1 , wherein the performing of the action prediction comprises performing the action prediction by fusing classification results based on a plurality of action classification schemes. 
     
     
       5. The method of  claim 1 , wherein the performing of the action prediction comprises:
 acquiring the frame-based local feature image, a video-based local feature image and a video-based global feature image from an image frame of the acquired video; 
 acquiring action classification results by performing the action classification with regard to the human body actions based on any two or any combination of a first action classification scheme of using the frame-based local feature image and the human body pose feature, a second action classification scheme of using the video-based local feature image, and a third action classification scheme of using the video-based global feature image and the video-based local feature image; and 
 fusing the action classification results and performing the action prediction. 
 
     
     
       6. The method of  claim 5 , wherein the acquiring of the frame-based local feature image, the video-based local feature image and the video-based global feature image from the image frame of the video comprises:
 generating a frame-based global feature image by extracting a feature from an image sequence of a temporal-spatial domain of the image frame using a three-dimensional (3D) convolutional neural network (CNN); 
 determining a region of interest (ROI) candidate from the frame-based global feature image using a regional recommendation network (RPN); 
 performing a ROI pooling with the ROI candidate with respect to the image frame and tearing the frame-based local feature image from the frame-based global feature image; 
 generating the video-based local feature image by performing a ROI sorting on the frame-based local feature image; and 
 generating the video-based global feature image based on the frame-based global feature image and the video-based local feature image. 
 
     
     
       7. The method of  claim 5 , wherein the acquiring of the frame-based local feature image, the video-based local feature image and the video-based global feature image from the image frame of the video comprises performing an action localization on a plurality of frame-based local feature images including the frame-based local feature image and selecting the video-based local feature image from a plurality of video-based local feature images. 
     
     
       8. The method of  claim 7 , wherein the performing of the action localization comprises performing the action localization with a fully connected (FC) network that comprises a first FC branch that determines which ROI candidate includes the human body and a second FC branch that determines a position of a box including the human body. 
     
     
       9. The method of  claim 1 , wherein the performing of the action prediction comprises:
 acquiring a video-based local feature image from an image frame of the video; 
 extracting a first feature associated with a human body pose action and a second feature associated with an interactive action from the video-based local feature image with a first 3D CNN having a human body pose action as a classification label and a second 3D CNN having an interactive action as a classification label; and 
 fusing the first feature and the second feature and acquiring an action classification result. 
 
     
     
       10. The method of  claim 9 , wherein
 the first 3D CNN is trained in advance with a loss function that classifies pose actions with a plurality of labels in mutually exclusive categories, and 
 the second 3D CNN is trained in advance with a loss function that classifies interactive actions with a plurality of labels in mutually non-exclusive categories. 
 
     
     
       11. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of  claim 1 . 
     
     
       12. A server that controls an augmented reality (AR) apparatus, the server comprising:
 a processor; and 
 a memory comprising instructions executable in the processor, 
 wherein in response to the instructions being executed in the processor, the processor is configured to: 
 acquire a video; 
 detect a human body from the acquired video; 
 perform an action prediction with regard to the detected human body; and 
 control the AR apparatus based on a result of the action prediction and a mapping relationship between human body actions and AR functions, 
 wherein, for the performing the action prediction, the processor is further configured to: 
 generate a human body part parsing result based on a convolution of an approximate semantic segmentation result and a direction prediction result; and 
 perform an action classification with regard to the human body actions based on a frame-based local feature image, a human body pose feature, and the human body part parsing result, 
 wherein the performing of the action classification comprises: 
 determining an image area of a human body instance in an image frame of the acquired video; 
 performing an approximate partial semantic segmentation on a human body image area candidate extracted from the image area, and generating an approximate semantic segmentation result corresponding to detailed parts of the human body; 
 predicting a relative direction of each pixel in the image frame with respect to a central portion of the human body and generating the direction prediction result; 
 performing the convolution based on the approximate semantic segmentation result and the direction prediction result and generating the human body part parsing result; and 
 performing the action classification with the first action classification scheme based on the frame-based local feature image, the human body pose feature and the human body part parsing result. 
 
     
     
       13. The server of  claim 12 , wherein the acquired video is a video of the AR apparatus and a user of the AR apparatus captured by a camera distinguished from the AR apparatus. 
     
     
       14. The server of  claim 12 , wherein the processor is configured to recognize a user corresponding to the detected human body, to associate the user with the AR apparatus, and to establish a communication with the AR apparatus. 
     
     
       15. The server of  claim 12 , wherein the processor is configured to perform the action prediction by fusing classification results based on a plurality of action classification schemes. 
     
     
       16. The server of  claim 12 , wherein the processor is configured to:
 acquire a video-based local feature image from an image frame of the acquired video; 
 extract a first feature associated with a human body pose action and a second feature associated with an interactive action from the video-based local feature image with a first three-dimensional (3D) convolutional neural network (CNN) having a human body pose action as a classification label and a second 3D CNN having an interactive action as a classification label; and 
 fuse the first feature and the second feature and acquire an action classification result. 
 
     
     
       17. An augmented reality (AR) apparatus comprising:
 a processor; and 
 a memory comprising instructions executable in the processor, 
 wherein in response to the instructions being executed in the processor, the processor is configured to: 
 acquire a video; 
 detect a human body from the acquired video; 
 perform an action prediction with regard to the detected human body; and 
 execute an AR function based on a result of the action prediction and a mapping relationship between human body actions and AR functions, 
 wherein, for the performing the action prediction, the processor is further configured to: 
 generate a human body part parsing result based on a convolution of an approximate semantic segmentation result and a direction prediction result; and 
 perform an action classification with regard to the human body actions based on a frame-based local feature image, a human body pose feature, and the human body part parsing result, 
 wherein the performing of the action classification comprises: 
 determining an image area of a human body instance in an image frame of the acquired video; 
 performing an approximate partial semantic segmentation on a human body image area candidate extracted from the image area, and generating an approximate semantic segmentation result corresponding to detailed parts of the human body; 
 predicting a relative direction of each pixel in the image frame with respect to a central portion of the human body and generating the direction prediction result; 
 performing the convolution based on the approximate semantic segmentation result and the direction prediction result and generating the human body part parsing result; and 
 performing the action classification with the first action classification scheme based on the frame-based local feature image, the human body pose feature and the human body part parsing result. 
 
     
     
       18. The AR apparatus of  claim 17 , wherein the acquired video is a video captured from a viewpoint of a user of the AR apparatus with a camera connected to the AR apparatus, or a video generated and stored in advance in the AR apparatus. 
     
     
       19. The AR apparatus of  claim 17 , wherein the processor is configured to perform the action prediction by fusing classification results based on a plurality of action classification schemes. 
     
     
       20. The AR apparatus of  claim 17 , wherein the processor is configured to:
 acquire a video-based local feature image from an image frame of the video; 
 extract a first feature associated with a human body pose action and a second feature associated with an interactive action from the video-based local feature image with a first three-dimensional (3D) convolutional neural network (CNN) having a human body pose action as a classification label and a second 3D CNN having an interactive action as a classification label; and 
 fuse the first feature and the second feature and acquire an action classification result. 
 
     
     
       21. A method comprising:
 acquiring a video with an augmented reality (AR) apparatus; 
 detecting an object in the acquired video; 
 performing an action prediction with regard to the detected object; and 
 displaying, on the AR apparatus, an AR function based on a result of the action prediction, 
 wherein the performing the action prediction further includes: 
 generating a parsing result based on a convolution of an approximate semantic segmentation result and a direction prediction result; and 
 performing an action classification with regard to object actions based on a frame-based local feature image, a object pose feature, and the parsing result, 
 wherein the performing of the action classification comprises: 
 determining an image area of a human body instance in an image frame of the acquired video; 
 performing an approximate partial semantic segmentation on a human body image area candidate extracted from the image area, and generating an approximate semantic segmentation result corresponding to detailed parts of the human body; 
 predicting a relative direction of each pixel in the image frame with respect to a central portion of the human body and generating the direction prediction result; 
 performing the convolution based on the approximate semantic segmentation result and the direction prediction result and generating the human body part parsing result; and 
 performing the action classification with the first action classification scheme based on the frame-based local feature image, the human body pose feature and the human body part parsing result.