Refrigerator and controlling method thereof

Abstract

A refrigerator includes: a body comprising a storage chamber; a door coupled to the body; a camera located on the body; and at least one processor operatively coupled to a memory and configured to execute one or more instructions stored in the memory, in which the one or more instructions cause the refrigerator to: based on detecting opening of the door, control the camera to capture the image of at least a part of the inside of the body and the door, identify, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition, and based on identifying that the predetermined condition is satisfied, store the image as an optimal door bin image.

What is claimed is: 
     
         1 . A refrigerator comprising:
 a body comprising a storage chamber;   a door rotatably coupled to the body to open or close the storage chamber, the door comprising a door bin;   a camera on the body, the camera being configured to capture an image of an inside of the body and an inside of the door;   a memory configured to store one or more instructions; and   at least one processor operatively coupled to the memory and configured to execute the one or more instructions stored in the memory,   wherein the one or more instructions, when executed by the at least one processor, cause the refrigerator to:
 based on detecting opening of the door, control the camera to capture the image of at least a part of the inside of the body and the door, 
 identify, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition, and 
 based on identifying that the predetermined condition is satisfied, store the image as an optimal door bin image. 
   
     
     
         2 . The refrigerator of  claim 1 , wherein the camera is on an upper end of the body,
 wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and   wherein the first line is on an upper end of the door.   
     
     
         3 . The refrigerator of  claim 1 , wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: obtain a template image while the door is open and is halted during a specific time,
 compare the template image with the image captured by the camera, and   identify, based on a result of the comparison of the template image with the image captured by the camera, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.   
     
     
         4 . The refrigerator of  claim 3 , wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: detect a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera,
 determine, based on the plurality of images, whether the door is in a moving state,   based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determine that the door is in a state of being open and halted, and   based on determining that the door is in the state of being open and halted during the specific time, obtain, as the template image, an image at a time point in the state in which the door is open and halted.   
     
     
         5 . The refrigerator of  claim 3 , wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to:
 based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, identify that the predetermined condition is satisfied.   
     
     
         6 . The refrigerator of  claim 1 , further comprising:
 an output device,
 wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: based on receiving a user input to output the optimal door bin image, control the output device to output the optimal door bin image. 
   
     
     
         7 . The refrigerator of  claim 6 , wherein the door further comprises a first door and a second door, and
 wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to:
 based on detecting one of the first door or the second door is opened, control the camera to obtain an optimal door bin image corresponding to the one of the first door or the second door, and 
 control the output device to output the optimal door bin image corresponding to the one of the first door or the second door. 
   
     
     
         8 . The refrigerator of  claim 1 , wherein the camera is a wide angle camera, and
 wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to:
 correct an image captured by the wide angle camera, and 
 identify whether the first area of the inside of the body included in the corrected image and the first line included in the door included in the corrected image satisfy the predetermined condition. 
   
     
     
         9 . The refrigerator of  claim 1 , further comprising:
 an output device,   wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to:
 rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline, 
 obtain an image of an area of interest by extracting the area of interest in the rotated optimal door bin image and cropping the area of interest, 
 correct the image of the area of interest by applying a perspective transformation, and 
 based on receiving a user input to output the optimal door bin image, control the output device to output the corrected image of the area of interest. 
   
     
     
         10 . The refrigerator of  claim 9 , wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to:
 obtain, based on the image of the area of interest, information on incoming groceries or outgoing groceries, and   control the output device to provide information on the incoming groceries or the outgoing groceries.   
     
     
         11 . A controlling method of a refrigerator comprising a body comprising a storage chamber, a door comprising a door bin and rotatably coupled to the body to open or close the storage chamber, and a camera on the body, the controlling method comprising:
 based on detecting opening of the door, controlling the camera to capture an image of at least a part of an inside of the body and the door;   identifying, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition; and   based on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image.   
     
     
         12 . The controlling method of  claim 11 , wherein the camera is located on an upper end of the body,
 wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and   wherein the first line is on an upper end of the door.   
     
     
         13 . The controlling method of  claim 11 , wherein the identifying whether the image satisfies the predetermined condition comprises:
 obtaining a template image while the door is open and is halted during a specific time;   comparing the template image with the image captured by the camera; and   identifying, based on a result of the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.   
     
     
         14 . The controlling method of  claim 13 , wherein the obtaining the template image comprises:
 detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera;   determining, based on the plurality of images, whether the door is in a moving state;   based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; and   based on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted.   
     
     
         15 . The controlling method of  claim 13 , wherein the identifying whether the predetermined condition is satisfied comprises:
 based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, identifying that the predetermined condition is satisfied.   
     
     
         16 . A non-transitory computer readable medium having instructions stored therein which when executed by a processor of a refrigerator comprising a body comprising a storage chamber, a door comprising a door bin and rotatably coupled to the body to open or close the storage chamber, and a camera located on the body, cause the processor to execute a method comprising:
 based on detecting opening of the door, controlling the camera to capture an image of at least a part of an inside of the body and the door;   identifying, based on the image, whether at least one of a first area of the inside of the body included in the image or a first line included in the door included in the image satisfies a predetermined condition; and   based on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image.   
     
     
         17 . The non-transitory computer readable medium of  claim 16 , wherein the camera is on an upper end of the body,
 wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and   wherein the first line is on an upper end of the door.   
     
     
         18 . The non-transitory computer readable medium of  claim 16 , wherein the identifying whether the image satisfies the predetermined condition comprises:
 obtaining a template image while the door is open and is halted during a specific time;   comparing the template image with the image captured by the camera; and   identifying, based on a result the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition.   
     
     
         19 . The non-transitory computer readable medium of  claim 18 , wherein the obtaining the template image comprises:
 detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera;   determining, based on the plurality of images, whether the door is in a moving state;   based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; and   based on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted.   
     
     
         20 . The non-transitory computer readable medium of  claim 18 , wherein the identifying whether the predetermined condition is satisfied comprises:
 based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, determining that the predetermined condition is satisfied.