Systems and methods for determining and using three-dimensional (3d) meshes of a subject

Abstract

Illustrative systems and methods for determining and using three-dimensional (3D) meshes of a subject are described herein. An illustrative system obtains 3D meshes of a subject at a frame rate, the 3D meshes including a current-frame 3D mesh corresponding to a current frame and a previous-frame 3D mesh corresponding to a previous frame. The system deforms the previous-frame 3D mesh based on the current-frame 3D mesh and determines, based on the deformed previous-frame 3D mesh, a 3D mesh of the subject for use with the current frame. In certain examples, the system uses the 3D mesh of the subject to facilitate rendering of a viewpoint-adaptive 3D representation of the subject.

What is claimed is: 
     
         1 . A method comprising:
 obtaining three-dimensional (3D) meshes of a subject at a frame rate, the 3D meshes including a current-frame 3D mesh corresponding to a current frame and a previous-frame 3D mesh corresponding to a previous frame;   deforming the previous-frame 3D mesh based on the current-frame 3D mesh; and   determining, based on the deformed previous-frame 3D mesh, a 3D mesh of the subject for use with the current frame.   
     
     
         2 . The method of  claim 1 , wherein deforming the previous-frame 3D mesh based on the current-frame 3D mesh comprises moving vertices of the previous-frame 3D mesh closer to corresponding vertices of the current-frame 3D mesh. 
     
     
         3 . The method of  claim 2 , wherein moving the vertices of the previous-frame 3D mesh closer to the corresponding vertices of the current-frame 3D mesh is constrained to a predefined amount of motion of the subject. 
     
     
         4 . The method of  claim 2 , wherein moving the vertices of the previous-frame 3D mesh closer to the corresponding vertices of the current-frame 3D mesh is constrained to a predefined type of motion of the subject. 
     
     
         5 . The method of  claim 1 , further comprising:
 receiving depth-data frames of the subject at the frame rate;   wherein obtaining the 3D meshes of the subject at the frame rate includes generating the 3D meshes of the subject based on the received depth-data frames.   
     
     
         6 . The method of  claim 1 , further comprising:
 receiving a plurality of depth-data streams captured of the subject by a plurality of depth cameras that each have a known vantage point, each depth-data stream including depth-data frames at the frame rate, each depth-data frame including a plurality of depth-data points;   wherein obtaining the 3D meshes of the subject at the frame rate includes, in each frame-rate period:
 combining the depth-data points from the depth-data streams into a collective 3D point cloud in a coordinate system; and 
 generating a respective one of the 3D meshes based on the collective 3D point cloud. 
   
     
     
         7 . The method of  claim 1 , further comprising:
 using the determined 3D mesh of the subject for use with the current frame to determine a set of visible vertices of the 3D mesh at the current frame; and   using the set of visible vertices of the 3D mesh at the current to generate a set of data streams configured to be used to render a viewpoint-adaptive 3D representation of the subject.   
     
     
         8 . The method of  claim 1 , embodied as computer-executable instructions on at least one non-transitory computer-readable medium. 
     
     
         9 . A system comprising:
 a processor; and   memory storing instructions executable by the processor to:
 obtain three-dimensional (3D) meshes of a subject at a frame rate, the 3D meshes including a current-frame 3D mesh corresponding to a current frame and a previous-frame 3D mesh corresponding to a previous frame; 
 deform the previous-frame 3D mesh based on the current-frame 3D mesh; and 
 determine, based on the deformed previous-frame 3D mesh, a 3D mesh of the subject for use with the current frame. 
   
     
     
         10 . The system of  claim 9 , wherein deforming the previous-frame 3D mesh based on the current-frame 3D mesh comprises moving vertices of the previous-frame 3D mesh closer to corresponding vertices of the current-frame 3D mesh. 
     
     
         11 . The system of  claim 10 , wherein moving the vertices of the previous-frame 3D mesh closer to the corresponding vertices of the current-frame 3D mesh is constrained to a predefined amount of motion of the subject. 
     
     
         12 . The system of  claim 10 , wherein moving the vertices of the previous-frame 3D mesh closer to the corresponding vertices of the current-frame 3D mesh is constrained to a predefined type of motion of the subject. 
     
     
         13 . The system of  claim 9 , the instructions further executable by the processor to:
 receive depth-data frames of the subject at the frame rate;   wherein obtaining the 3D meshes of the subject at the frame rate includes generating the 3D meshes of the subject based on the received depth-data frames.   
     
     
         14 . The system of  claim 9 , the instructions further executable by the processor to:
 receive a plurality of depth-data streams captured of the subject by a plurality of depth cameras that each have a known vantage point, each depth-data stream including depth-data frames at the frame rate, each depth-data frame including a plurality of depth-data points;   wherein obtaining the 3D meshes of the subject at the frame rate includes, in each frame-rate period:
 combining the depth-data points from the depth-data streams into a collective 3D point cloud in a coordinate system; and 
 generating a respective one of the 3D meshes based on the collective 3D point cloud. 
   
     
     
         15 . The system of  claim 9 , the instructions further executable by the processor to:
 use the determined 3D mesh of the subject for use with the current frame to determine a set of visible vertices of the 3D mesh at the current frame; and   use the set of visible vertices of the 3D mesh at the current to generate a set of data streams configured to be used to render a viewpoint-adaptive 3D representation of the subject.   
     
     
         16 . A system comprising:
 a processor; and   memory storing instructions executable by the processor to:
 receive a set of video streams captured, of a subject, by a set of video cameras having known vantage points in a coordinate system, the video streams including time-synchronized video frames corresponding to a current frame period; 
 determine a three-dimensional (3D) mesh of the subject, for the current frame period, based on a current-frame 3D mesh of the subject corresponding to the current frame period and a previous-frame 3D mesh of the subject corresponding to a previous frame period; and 
 generate one or more time-synchronized data streams based on the set of video streams and the 3D mesh of the subject, wherein the one or more time-synchronized data streams are operable to cause a viewpoint-adaptive 3D representation of the subject to be rendered. 
   
     
     
         17 . The system of  claim 16 , wherein determining the 3D mesh of the subject comprises:
 deforming the previous-frame 3D mesh based on the current-frame 3D mesh; and   determining, based on the deformed previous-frame 3D mesh, the 3D mesh of the subject for use with the current frame period.   
     
     
         18 . The system of  claim 17 , wherein deforming the previous-frame 3D mesh based on the current-frame 3D mesh comprises moving vertices of the previous-frame 3D mesh closer to corresponding vertices of the current-frame 3D mesh. 
     
     
         19 . The system of  claim 18 , wherein moving the vertices of the previous-frame 3D mesh closer to the corresponding vertices of the current-frame 3D mesh is constrained to a predefined amount of motion of the subject. 
     
     
         20 . The system of  claim 18 , wherein moving the vertices of the previous-frame 3D mesh closer to the corresponding vertices of the current-frame 3D mesh is constrained to a predefined type of motion of the subject.