Approaches to independently detecting presence and estimating pose of body parts in digital images and systems for implementing the same

Abstract

Introduced here are computer-implemented platforms (also referred to as “pose monitoring platforms”) that are designed to improve adherence to, and success of, programs requiring performance of physical activities. As part of a program, a participant may be requested to engage with a pose monitoring platform to perform a single physical activity, multiple repetitions of a single physical activity, or multiple repetitions of multiple physical activities. The pose monitoring platform can determine, for example, using a neural network that has parallel branches, whether digital images of the participant's environment includes certain body parts and then estimate poses of those body parts. The pose monitoring platform can use the estimated poses to guide the participant through the program, such as by providing instructions for performing the physical activities.

What is claimed is: 
     
         1 . A method performed by a computer program executed on a computing device, the method comprising:
 receiving a digital image of a scene that includes an individual performing a physical activity in an environment;   extracting multiple feature maps for the digital image, wherein each feature map represents content in a corresponding one of multiple segments of the digital image;   for each of the multiple feature maps, applying a neural network so as to:
 determine, via a first branch of the neural network, a likelihood that the corresponding one of the multiple segments includes a given body part, 
 determine, via a second branch of the neural network, an estimated pose of the given body part in the corresponding one of the multiple segments, 
 compare the likelihood to a threshold value programmed in memory of the computing device, and 
 responsive to a determination that the likelihood exceeds the threshold value, store, in a data structure, an indication that the given body part in the corresponding one of the multiple segments is in the estimated pose. 
   
     
     
         2 . The method of  claim 1 , wherein each segment is representative of a contiguous region of pixels that is associated with a portion of the scene. 
     
     
         3 . The method of  claim 1 , wherein the data structure is in memory of the computing device. 
     
     
         4 . The method of  claim 1 , wherein the first branch is part of a set of branches, each of which is associated with a different body part. 
     
     
         5 . The method of  claim 1 , wherein the second branch is part of a set of branches, each of which is associated with a different pose. 
     
     
         6 . A non-transitory medium storing instructions that, when executed by a processor of a computing device, cause the computing device to perform operations comprising:
 receiving image data that is representative of a digital image of a scene that includes an individual performing a physical activity;   extracting feature maps from segments of the image data, wherein each segment is representative of a contiguous region of pixels in the digital image;   for each of the feature maps, applying a neural network so as to:
 determine a likelihood that a corresponding one of the segments includes a given body part, 
 determine an estimated pose of the given body part in the corresponding one of the segments, and 
 responsive to a determination that the likelihood exceeds a threshold value, store an indication that the given body part is in the estimated pose in a data structure. 
   
     
     
         7 . The non-transitory medium of  claim 6 , wherein the operations further comprise:
 receiving multiple digital images;   determining locations of one or more anatomical landmarks in each of the multiple digital images;   determining, based on the locations, spatial positions of one or more body parts in each of the multiple digital images;   for each type of body part in the multiple digital images:
 placing a bounding box around that body part, and 
 iteratively displacing the bounding box until the bounding box does not include the spatial positions associated with that body part; and 
   for each displaced bounding box, adding a portion of the multiple digital images that is associated with the displaced bounding box to a training dataset; and   training the neural network on the training dataset.   
     
     
         8 . The non-transitory medium of  claim 7 , wherein the locations are two-dimensional locations. 
     
     
         9 . The non-transitory medium of  claim 7 , wherein the locations are three-dimensional locations. 
     
     
         10 . The non-transitory medium of  claim 6 , wherein said storing comprises:
 programmatically associating the estimated pose of the given body part with a portion of the scene.   
     
     
         11 . The non-transitory medium of  claim 6 , wherein the operations further comprise:
 determining, based on the estimated pose, a therapeutic activity being performed by the individual; and   displaying, via a graphic user interface on the computing device, an indication of the therapeutic exercise.   
     
     
         12 . The non-transitory medium of  claim 6 , wherein the operations further comprise:
 determining, based on the estimated pose, an instruction for improving a technique associated with the estimated pose; and   displaying, via a graphic user interface on the computing device, the instruction.   
     
     
         13 . The non-transitory medium of  claim 6 , wherein the neural network comprises a series of convolutional layers and a series of connected layers of decreasing size. 
     
     
         14 . The non-transitory medium of  claim 13 , wherein a last layer of the neural network is a sigmoid activation function. 
     
     
         15 . A computing device comprising:
 a processor; and   a memory with instructions stored therein that, when executed by the processor, cause the computing device to perform operations comprising:
 obtaining a digital image of a scene that includes an individual performing a physical activity in an environment; and 
 applying a neural network to multiple segments of the digital image in an independent manner, so as to:
 determine a likelihood that a corresponding one of the multiple segments includes a given body part; 
 determine an estimated pose of the given body part in the corresponding one of the multiple segments; and 
 in response to a determination that the likelihood exceeds a threshold value, store an indication that the given body part in the corresponding one of the multiple segments is in the estimated pose. 
 
   
     
     
         16 . The computing device of  claim 15 , wherein said storing comprises:
 programmatically associating the estimated pose of the given body part with a time at which the digital image is generated.   
     
     
         17 . The computing device of  claim 15 , further comprising:
 an image sensor that is configured to generate the digital image in response to receiving input that indicates the individual is performing the physical activity.