Refrigerator and control method therefor

Abstract

A refrigerator includes: a body including a storage chamber; a door including a door bin; a camera, arrangeable in the body, to photograph an inside of the body and an inside of the door; a memory; and a processor which obtains an image of at least a portion of the inside of the body and at least a portion of the door that is captured through the camera based on a trigger signal, detects a food object in the image, tracks the food object and identifies whether the food object is put in or taken out of the refrigerator, acquires feature information corresponding to the food object by inputting the image to a trained neural network model based on the food object being identified as being put in, and matches the image and the acquired feature information and stores the image and the acquired feature information in a food database.

1 . A refrigerator comprising:
 a body including a storage chamber;   a door, which is rotatably coupled to the body to open and close the storage chamber, including a door bin;   a camera, arrangeable in the body, configured to photograph an inside of the body and an inside of the door;   a memory to store at least one instruction; and   a processor configured to:
 based on detecting a trigger signal, obtain an image of at least a portion of the inside of the body and at least a portion of the door that is photographed through the camera while the camera is arranged in the body, 
 detect a food object included in the obtained image, 
 track the food object and identify whether the food object is put in or taken out of the refrigerator, 
 based on identifying that the food object is put in the refrigerator, obtain feature information corresponding to the food object by inputting the image into a trained neural network model, and 
 match the image and the obtained feature information, and store the image and the obtained feature information based on the match in a food database. 
   
     
     
         2 . The refrigerator of  claim 1 ,
 wherein the processor is configured to:
 obtain information associated with the food object by inputting the image into a trained second neural network model, and 
 store the information associated with the food object in the food database together with the image and the obtained feature information, and 
   the information associated with the food object comprises:
 at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object. 
   
     
     
         3 . The refrigerator of  claim 1 ,
 wherein the processor is configured to:
 crop an area including the food object from the image, and 
 obtain the feature information corresponding to the food object by inputting an area where the food object is included into the trained neural network model. 
   
     
     
         4 . The refrigerator of  claim 1 , wherein the obtained feature information is first feature information, and
 the processor is configured to:
 based on identifying that the food object is taken out, identify the second feature information which was obtained when the food object was put in in the food database, and 
 based on the identifying the second feature information, delete the second feature information from the food database. 
   
     
     
         5 . The refrigerator of  claim 4 , wherein the processor is configured to:
 based on the second feature information not being identified, identify a candidate list corresponding to the food object, and   based on identifying the candidate list, provide user interface to receive user's selection for deleting at least one candidate from the food database,   wherein the candidate list comprises:
 at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in. 
   
     
     
         6 . The refrigerator of  claim 4 , wherein the processor is configured to:
 based on identifying that the food object was taken out and not is put in within a predetermined time, delete the second feature information from the food database   
     
     
         7 . The refrigerator of  claim 1 , wherein the food object comprises a first food object and a second food object, and
 the processor is configured to:
 identify that the second food object is put in the refrigerator, 
 based on identifying that the second food object is put in, identify whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, and 
 based on identifying that the first food object and the second food object are matched, determine occurrence of an event where the first food object is put in again, and store information on the event in the food database. 
   
     
     
         8 . The refrigerator of  claim 7 ,
 wherein the processor is configured to:
 based on identifying that the first food object matching the second food object does not exist, store the image and the obtained feature information as new food object. 
   
     
     
         9 . The refrigerator of  claim 1 ,
 wherein the processor is configured to:
 based on identifying that the food object is put in, identify whether the feature information corresponding to the food object is obtainable, 
 based on identifying the feature information corresponding to the food object is unobtainable, obtain information associated with an area corresponding to the food object from the image, and 
 match the image and the information associated with the area corresponding to the food object, and store the image and the information with the area in the food database. 
   
     
     
         10 . The refrigerator of  claim 1 ,
 wherein the processor is configured to:
 detect a hand object from the image, 
 obtain identification information associated with the detected hand object based on a plurality of pre-stored hand objects, and 
 match the identification information associated with the hand object, the image, and the feature information, and store the identification information associated with the hand object, the image and the feature information in the food database. 
   
     
     
         11 . A control method for a refrigerator, the control method comprising:
 based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body;   detecting a food object included in the obtained image;   tracking the food object and identifying whether the food object is put in or taken out of the refrigerator;   based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; and   matching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.   
     
     
         12 . The control method of  claim 11 ,
 wherein the control method further comprises:
 obtaining information associated with the food object by inputting the image into a trained second neural network model, and 
   the storing comprises:
 storing the information associated with the food in the food database together with the image and the obtained feature information, and 
   the information associated with the food object comprises:
 at least one of a type of the food object, a product name of the food object, a manufacturer of the food object, or net contents of the food object. 
   
     
     
         13 . The control method of  claim 11 ,
 wherein the control method comprises:
 cropping an area including the food object from the image, and 
   the obtaining the feature information comprises:
 obtaining the feature information corresponding to the food object by inputting an area wherein the food object is included into the trained neural network model. 
   
     
     
         14 . The control method of  claim 11 , wherein the obtained feature information is first feature information, and
 wherein the control method comprises:
 based on identifying that the food object is taken out, identifying the second feature information which was obtained when the food object was put in in the food database, and 
 based on the identifying the second feature information, deleting the second feature information from the food database. 
   
     
     
         15 . The control method of  claim 14 , wherein the control method comprises:
 based on the second feature information not being identified, identifying a candidate list corresponding to the food object, and   based on identifying the candidate list, providing user interface to receive user's selection for deleting at least one candidate from the food database,   wherein the candidate list comprises:   at least one of food object corresponding to feature information having a similarity more than a predetermined value with feature information corresponding to the food object, unspecified packaged object, or food object that was recently put in.   
     
     
         16 . The control method of  claim 14 , wherein the control method comprises:
 based on identifying that the food object was taken out and not is put in within a predetermined time, deleting the second feature information from the food database.   
     
     
         17 . The control method of  claim 11 , wherein the food object comprises a first food object and a second food object, and
 wherein the control method comprises:
 identifying that the second food object is put in the refrigerator, 
 based on identifying that the second food object is put in, identifying whether the second food object matches the first food object was being identified as being taken out within a predetermined time before, and 
 based on identifying that the first food object and the second food object are matched, determining occurrence of an event where the first food object is put in again, and storing information on the event in the food database. 
   
     
     
         18 . The control method of  claim 17 , wherein the control method comprises:
 based on identifying that the first food object matching the second food object does not exist, storing the image and the obtained feature information as new food object.   
     
     
         19 . The control method of  claim 11 , wherein the control method comprises
 based on identifying that the food object is put in, identifying whether the feature information corresponding to the food object is obtainable,   based on identifying the feature information corresponding to the food object is unobtainable, obtaining information associated with an area corresponding to the food object from the image, and   matching the image and the information associated with the area corresponding to the food object, and storing the image and the information with the area in the food database.   
     
     
         20 . One or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of a refrigerator individually or collectively, cause the refrigerator to perform operations, the operations comprising:
 based on detecting a trigger signal, obtaining an image of at least a portion of an inside of a body including a storage chamber and at least a portion of a door, rotatably coupled to the body to open and close the storage chamber, including a door bin, the image being photographed through a camera located in the body;   detecting a food object included in the obtained image;   tracking the food object and identifying whether the food object is put in or taken out of the refrigerator;   based on identifying that the food object is put in the refrigerator, obtaining feature information corresponding to the food object by inputting the image into a trained neural network model; and   matching the image and the obtained feature information, and storing the image and the obtained feature information based on the match in a food database.