Apparatus and method for user interest information generation

Abstract

The present invention relates to an artificial intelligence (AI), which emulates functions of a human brain, such as recognition and reasoning, by utilizing a machine learning algorithm such as deep learning, and relates to context awareness based artificial intelligence application technology for obtaining interest information of a user from an image displayed to the user. An electronic device, according to one embodiment of the present invention acquires context data related to the image, and uses the context data to select a node of interest corresponding to the context data from among nodes of an index tree for searching for sample images which correspond to candidate objects extracted from the image by using a model generated as a result of machine learning, wherein the node of interest is selected by using a result of comparison between a subject of each node of the index tree and the context data; and an object of interest is selected from among the candidate objects included in the image by using the node of interest.

What is claimed is: 
     
       1. An electronic device comprising:
 a processor; 
 a display for displaying an image; and 
 a memory storing a plurality of instructions for execution by the processor, 
 wherein the plurality of instructions comprise:
 one or more instructions for obtaining context data associated with an image displayed on the display; 
 one or more instructions for selecting, from among nodes of a semantic hierarchical index tree for searching for a sample image corresponding to a candidate object included in the image, a node of interest corresponding to the candidate object using the context data, wherein the node of interest is selected based on a comparison results between subjects of each of the nodes of the index tree and the context data; and 
 one or more instructions for selecting an object of interest among candidate objects included in the image by traversing the index tree in a top-down manner starting with the node of interest; 
 
 wherein the nodes of the semantic hierarchical index tree are hierarchically connected in accordance with semantic relationships among subjects of each of the nodes. 
 
     
     
       2. The electronic device of  claim 1 , wherein:
 the context data comprises a plurality of context elements, and the one or more instructions for selecting the node of interest comprise instructions for selecting, as the node of interest, a lowermost node among the plurality of nodes corresponding to the plurality of context elements. 
 
     
     
       3. The electronic device of  claim 2 , wherein the plurality of context elements comprise a first context element and a second context element and the index tree includes a first node corresponding to the first context element and a second node corresponding to the second context element, and
 wherein, based on the second node being, the lowermost node in the index tree as between the first node and the second node, 
 the one or more instructions for selecting the object of interest comprises:
 one or more instructions for searching for an object corresponding to the first node among the plurality of candidate objects based on failing to identify an object, among the candidate objects, corresponding to the second node; and 
 one or more instructions for selecting an object of interest among candidate objects included in the image using the first node. 
 
 
     
     
       4. The electronic device of  claim 3 , wherein:
 the first node has a depth of N, where N is a natural number greater than or equal to 1, on the index tree, and the second node has a depth of N+2, where N is a natural number greater than or equal to 1, on the index tree, 
 the one or more instructions for searching for an object corresponding to the first node among the plurality of candidate objects comprises, based on failing to identify an object, among candidate objects, corresponding to the second node, comprises an instructions for passing one or more nodes present between the second node and the first node and traveling to the first node. 
 
     
     
       5. The electronic device of  claim 1 , wherein the one or more instructions for selecting the object of interest comprises:
 one or more instructions for searching for an object corresponding to a node higher in the index tree than the node of interest among the plurality of candidate objects, based on failing to identify an object, among the candidate objects, corresponding to the node of interest; and 
 one or more instructions for selecting an object of interest among candidate objects included in the image using the higher node. 
 
     
     
       6. The electronic device of  claim 1 , wherein:
 the nodes of the index tree comprise leaf nodes and non-leaf nodes, 
 each of the non-leaf nodes comprises a classifier for receiving feature data and generating an output indicating a child node corresponding to the feature data, 
 the one or more instructions for selecting an object of interest among the candidate objects included in the image comprises:
 one or more instructions for, based on the node of interest being one of the non-leaf nodes, inputting feature data of the candidate object to the classifier of the node of interest and traveling to a child node indicated by output of the classifier; and 
 one or more instructions for repeating the instruction to travel to a child node until one of the leaf nodes is reached. 
 
 
     
     
       7. The electronic device of  claim 6 , wherein:
 each leaf node corresponds to one or more images matched to a subject allocated to the leaf node among each sample image of a plurality of sample images of an image archive, 
 the one or more instructions for selecting an object of interest among the candidate objects included in the image comprises:
 one or more instructions for, based on reaching one of the leaf nodes, identifying whether a sample image matched with the reached leaf node matches the candidate object; and 
 one or more instruction for selecting, as the object of interest, the candidate object that is identified as being matched with the sample image matched with the reached leaf node. 
 
 
     
     
       8. The electronic device of  claim 6 , wherein:
 the one or more instructions for selecting an object of interest among candidate objects included in the image comprises one or more instructions for traveling to a higher node in the index tree than a current node based on the output of the classifier of the current node indicating that a child node corresponding to the feature data is not present for any candidate object; 
 the higher node being a closest higher node, among higher nodes of the index tree, matched with one of a plurality of context elements included in the context data. 
 
     
     
       9. The electronic device of  claim 1 , wherein the one or more instruction for obtaining the context data comprises one or more instructions for obtaining context data matched with a subject of at least some nodes among the nodes of the index tree. 
     
     
       10. The electronic device of  claim 1 , wherein the one or more instructions for obtaining the context data comprises one or more instructions for obtaining, as the context data, position information of the electronic device. 
     
     
       11. The electronic device of  claim 1 , wherein the one or more instructions for obtaining the context data comprises one or more instructions for obtaining, as the context data, a text that is generated by converting voice data input through a microphone while the image is being displayed on the display. 
     
     
       12. The electronic device of  claim 1 , wherein the one or more instructions for obtaining the context data comprises one or more instructions for obtaining the context data based on sensing a trigger of an object of interest selection operation, and
 wherein the trigger is indicative of sensing an operation associated with an image of a user of the electronic device. 
 
     
     
       13. The electronic device of  claim 12 , wherein:
 the display comprises a touch display, 
 the one or more instructions for obtaining the context data comprises one or more instructions for, based on sensing the trigger of the object of interest selection operation, obtaining the context data, and 
 the trigger is indicative of displaying a keyboard on the display while the image is being displayed. 
 
     
     
       14. The electronic device of  claim 12 , wherein:
 the one or more instructions for obtaining the context data comprises one or more instructions for obtaining the context data, based on sensing the trigger of the object of interest selection operation, and 
 the trigger is indicative of sensing a screen capture manipulation. 
 
     
     
       15. A method executed by an electronic device, the method comprising:
 obtaining context data at a time of displaying of an image; 
 selecting, from among nodes of a semantic hierarchical index tree to search for a sample image corresponding to a candidate object recognized in the image, a node of interest corresponding to the candidate object using the context data, wherein the node of interest is selected based on comparison results between subject of each of the nodes of the index tree and the context data; and 
 selecting an object of interest among the plurality of candidate objects by traversing the index tree in a top-down manner starting with the node of interest, 
 wherein the nodes of the semantic hierarchical index tree are hierarchically connected in accordance with semantic relationships among subjects of each of the nodes. 
 
     
     
       16. The method of  claim 15 , wherein:
 the nodes of the index tree comprise leaf nodes and non-leaf nodes, 
 each of the non-leaf nodes comprises an inherent classifier, the inherent classifier configured to receive feature data of an image to generate an output indicating a child node corresponding to the feature data, and each leaf node corresponds to one or more images allocated to the leaf node among each sample image of a plurality of sample images of an archive, and 
 the selecting the object of interest comprises:
 based on the node of interest being one of the non-leaf nodes, inputting feature data of the candidate object to the classifier of the node of interest and traveling to a child node indicated by output of the classifier; 
 repeating the traveling to a child node until one of the leaf nodes is reached; 
 based on reaching one of the leaf node, identifying whether a sample image matched with the reached leaf node matches with the candidate object; and 
 selecting, as the object of interest, the candidate object that is identified as being matched with the sample image matched with the reached leaf node. 
 
 
     
     
       17. The method of  claim 16 , wherein:
 the plurality of candidate objects comprise a first object and a second object, and the traveling to the child node indicated by the output of the classifier comprises:
 based on an output of the classifier according to an input of feature data of the first object indicating traveling to one of child nodes of a current node and an output of the classifier according to input of feature data of the second object indicating that a child node corresponding to the second object is not present, splitting a tree search for the first object and a tree search for the second object. 
 
 
     
     
       18. The method of  claim 16 , wherein:
 the plurality of candidate objects comprise a first object and a second object, 
 the traveling to the child node indicated by the output of the classifier comprises, based on an output of the classifier according to an input of feature data of the first object indicating traveling to one of child nodes of a current node and an output of the classifier according to input of feature data of the second object indicating that a child node corresponding to the second object is not present, terminating a tree search for the second object.