Cooking motion estimation device, cooking motion estimation method, and cooking motion estimation program

Abstract

Coordinates of a joint point are identified, and a hand region that is a coordinate region of a hand is estimated, for each of video frames that constitute a cooking behavior video, based on posture recognition technology. A cooking utensil region that is a coordinate region of a cooking utensil is identified for each of the video frames that constitute the cooking behavior video, based on object recognition technology. When the hand region and the cooking utensil region overlap, a cooking motion for each of the video frames is estimated from a type of the cooking utensil.

1 . A cooking motion estimation device comprising a storage unit and a control unit, wherein
 the storage unit includes:   a video storage unit that stores a cooking behavior video of each of users, and   the control unit includes:   a hand estimating unit that identifies coordinates of a joint point and estimates a hand region that is a coordinate region of a hand, for each of video frames that constitute the cooking behavior video, based on posture recognition technology;   a cooking utensil identifying unit that identifies a cooking utensil region that is a coordinate region of a cooking utensil, for each of the video frames that constitute the cooking behavior video, based on object recognition technology; and   a cooking motion estimating unit that estimates a cooking motion for each of the video frames from a type of the cooking utensil when the hand region and the cooking utensil region overlap.   
     
     
         2 . The cooking motion estimation device according to  claim 1 , wherein
 the control unit further includes:   a time setting unit that sets the video frames in connection with an elapsed time; and   a classification calculating unit that calculates a cooking time and a workload for each cooking motion classification that distinguishes a feature of the cooking motion, based on the cooking motion for each of the video frames.   
     
     
         3 . The cooking motion estimation device according to  claim 2 , wherein
 the control unit further includes:   a representative value obtaining unit that obtains a cooking time representative value and a workload representative value for the each cooking motion classification, based on the cooking time and the workload for the each cooking motion classification of all of the users.   
     
     
         4 . The cooking motion estimation device according to  claim 3 , wherein
 the control unit further includes:   an outlier identifying unit that identifies the cooking behavior video in which the cooking motion with any one or both of the cooking time and the workload that are outliers is recorded, based on any one or both of the cooking time representative value and the workload representative value.   
     
     
         5 . The cooking motion estimation device according to  claim 2 , wherein
 the cooking behavior video is set in connection with attribute data indicating an attribute of the user, and   the classification calculating unit calculates the cooking time and the workload for each attribute and for the each cooking motion classification, based on the attribute data and the cooking motion for each of the video frames.   
     
     
         6 . The cooking motion estimation device according to  claim 2 , wherein
 the time setting unit further obtains order data of the cooking motion, based on the cooking motion for each of the video frames, and   the control unit further includes:   a cooking behavior obtaining unit that obtains cooking behavior data of the user, based on the cooking time and the workload for the each cooking motion classification, and the order data.   
     
     
         7 . The cooking motion estimation device according to  claim 1 , wherein
 the storage unit further includes:   a model storage unit that stores a posture recognition model in which hand video frames in which a plurality of hand movements during cooking are recorded are training data, the video frames that constitute the cooking behavior video are input, and the hand region is output, and   the hand estimating unit identifies the coordinates of a joint point and estimates the hand region that is the coordinate region of a hand, for each of the video frames that constitute the cooking behavior video, using the posture recognition model.   
     
     
         8 . The cooking motion estimation device according to  claim 1 , wherein
 the storage unit further includes:   a model storage unit that stores an object recognition model in which cooking utensil video frames in which a plurality of the cooking utensils are recorded are training data, the video frames that constitute the cooking behavior video are input, and the cooking utensil region is output, and   the cooking utensil identifying unit identifies the cooking utensil region that is the coordinate region of the cooking utensil, for each of the video frames that constitute the cooking behavior video, using the object recognition model.   
     
     
         9 . The cooking motion estimation device according to  claim 5 , wherein the attribute is a cooking proficiency level for distinguishing between being good at cooking and being poor at cooking. 
     
     
         10 . The cooking motion estimation device according to  claim 1 , wherein the cooking behavior video is a video that records cooking of each of the users from a side in any kitchen including a home kitchen of the user. 
     
     
         11 . The cooking motion estimation device according to  claim 1 , wherein
 the storage unit further includes:   a model storage unit that stores a cooking motion estimation model that is a machine learning model in which a cooking video labeled with the hand region is training data, the hand region and the cooking utensil region are explanatory variables, and the cooking motion is a response variable, and   when the hand region and the cooking utensil region overlap, the cooking motion estimating unit estimates the cooking motion for each of the video frames from a type of the cooking utensil, using the cooking motion estimation model.   
     
     
         12 . The cooking motion estimation device according to  claim 1 , wherein the control unit further includes:
 a food ingredient identifying unit that identifies a food ingredient region that is a coordinate region of a food ingredient, for each of the video frames that constitute the cooking behavior video, based on the object recognition technology or image segmentation technology for the video frames.   
     
     
         13 . The cooking motion estimation device according to  claim 12 , wherein the food ingredient identifying unit further estimates intake nutrients from the food ingredient. 
     
     
         14 . The cooking motion estimation device according to  claim 12 , wherein when the hand region and the food ingredient region overlap, or when the cooking utensil region and the food ingredient region overlap, the cooking motion estimating unit further estimates the cooking motion for each of the video frames from a type of the food ingredient. 
     
     
         15 . The cooking motion estimation device according to  claim 1 , wherein the control unit further includes:
 a seasoning identifying unit that identifies a seasoning region that is a coordinate region of a seasoning, for each of the video frames that constitute the cooking behavior video, based on the object recognition technology.   
     
     
         16 . The cooking motion estimation device according to  claim 15 , wherein when the hand region and the seasoning region overlap, the cooking motion estimating unit further estimates the cooking motion for each of the video frames from a type of the seasoning. 
     
     
         17 . A cooking motion estimation method executed by a cooking motion estimation device including a storage unit and a control unit, wherein
 the storage unit includes:   a video storage unit that stores a cooking behavior video of each of users,   the cooking motion estimation method executed by the control unit comprising:   a hand estimating step of identifying coordinates of a joint point and estimating a hand region that is a coordinate region of a hand, for each of video frames that constitute the cooking behavior video, based on posture recognition technology;   a cooking utensil identifying step of identifying a cooking utensil region that is a coordinate region of a cooking utensil, for each of the video frames that constitute the cooking behavior video, based on object recognition technology; and   a cooking motion estimating step of, when the hand region and the cooking utensil region overlap, estimating a cooking motion for each of the video frames from a type of the cooking utensil.   
     
     
         18 . A cooking motion estimation program executed by a cooking motion estimation device including a storage unit and a control unit, wherein
 the storage unit includes:   a video storage unit that stores a cooking behavior video of each of users,   the cooking motion estimation program causing the control unit to execute:   a hand estimating step of identifying coordinates of a joint point and estimating a hand region that is a coordinate region of a hand, for each of video frames that constitute the cooking behavior video, based on posture recognition technology;   a cooking utensil identifying step of identifying a cooking utensil region that is a coordinate region of a cooking utensil, for each of the video frames that constitute the cooking behavior video, based on object recognition technology; and   a cooking motion estimating step of, when the hand region and the cooking utensil region overlap, estimating a cooking motion for each of the video frames from a type of the cooking utensil.