Reduction of Time Lag Between Positions and Orientations Being Measured and Display Corresponding to the Measurements

Abstract

A system to extrapolate from motion states measured for past time instances during a user movement to predict motion states at a subsequent time instance at a display of a virtual object corresponding to the user movement. The prediction can be used to render the display and reduce or eliminate the lag between user action and corresponding action of the virtual object. An artificial neural network can be trained to improve the prediction accuracy based on patterns of user movements in the use of the application displaying virtual reality, augmented reality, mixed reality, and/or extended reality.

What is claimed is: 
     
         1 . A method, comprising:
 receiving, in a computing device, first measurements of at least one sensor module attached respectively to at least one part of a user, wherein the first measurements are representative of first motion states of the at least one part of the user as measured, via the at least one sensor module, at one or more first time instances prior to the receiving of the first measurements;   predicting, by the computing device using an artificial neural network, second measurements representative of second motion states of the at least one part of the user at a second time instance that is after the predicting of the second measurements;   providing, prior to the second time instance, the second measurements as an input to an application running in the computing device;   rendering, by the application, a display of a virtual object to have third motion states corresponding to the second measurements predicted for the second time instance; and   presenting, by the computing device, the display of the virtual object to have the third motion states.   
     
     
         2 . The method of  claim 1 , wherein the virtual object is presented in a virtual reality, an augmented reality, a mixed reality, or an extended reality, or any combination thereof. 
     
     
         3 . The method of  claim 2 , wherein the virtual object includes an avatar of the user rendered according to a skeleton model configured to represent motions of the at least one part of the user. 
     
     
         4 . The method of  claim 3 , wherein the first motion states include position, orientation, velocity, acceleration, or rotation, or any combination thereof. 
     
     
         5 . The method of  claim 4 , wherein the at least one sensor module includes a plurality of sensor modules attached to a subset of a plurality of parts of the user forming a kinematic chain on the user; and the second measurements include motion states of at least one part on the kinematic chain that has no sensor module being attached to the user. 
     
     
         6 . The method of  claim 5 , wherein the artificial neural network is trained to generate the second measurements based on third measurements from a separate tracking system generated in tracking sample user movements in using the application; and the separate tracking system is not used during performance of the method. 
     
     
         7 . The method of  claim 6 , wherein the separate tracking system include a camera configured to capture images of sensor modules during the sample user movements at a predetermined interval; and the method further comprises:
 identifying first motion parameters as measured by the camera at first times of displaying of motion according to second motion parameters measured using sensor modules at second times prior to the first times;   training the artificial neural network to predict the first motion parameters based on the second motion parameters.   
     
     
         8 . The method of  claim 6 , wherein the first measurements include a first portion generated using at least one inertial measurement unit, and a second portion generated using a camera configured on a head mounted display. 
     
     
         9 . The method of  claim 6 , further comprising:
 receiving, in the computing device, first data representative of measurements generated by at least one inertial measurement unit configured in the at least one sensor module;   receiving, in the computing device, second data representative of images of the at least one sensor module captured by a camera configured on a head mounted display;   generating, by the computing device, the first measurements by combining the first data and the second data using a filter.   
     
     
         10 . The method of  claim 6 , further comprising:
 predicting, using the artificial neural network, a time delay of the rendering of the display of the virtual object, wherein the second measurements are representative of an extrapolation over the time delay according to the first measurements.   
     
     
         11 . The method of  claim 6 , wherein the artificial neural network includes a first portion having a convolution neural network to process an optical measurement sequence and a second portion having a recurrent neural network to process an inertial measurement unit measurement sequence. 
     
     
         12 . The method of  claim 11 , wherein the artificial neural network further includes a third portion having a recurrent neural network to combine inputs from the first portion and the second portion. 
     
     
         13 . A computing device, comprising:
 memory storing instructions for a motion processor and an application of virtual reality, augmented reality, mixed reality, or extended reality, or any combination thereof;   a communication device configured to receive input from at least one sensor module;   at least one processor configured via the instructions to:
 receive first measurements of the at least one sensor module attached respectively to at least one part of a user, wherein the first measurements are representative of first motion states of the at least one part of the user as measured, via the at least one sensor module, at one or more first time instances prior to reception of the first measurements; 
 predict, using an artificial neural network, second measurements representative of second motion states of the at least one part of the user at a second time instance that is after prediction of the second measurements; 
 provide, prior to the second time instance, the second measurements as an input to the application running in the computing device; 
 render, by the application, a display of a virtual object to have third motion states corresponding to the second measurements predicted for the second time instance; and 
 present the display of the virtual object to have the third motion states. 
   
     
     
         14 . The computing device of  claim 13 , wherein the virtual object includes an avatar of the user rendered according to a skeleton model configured to represent motions of the at least one part of the user; and the first motion states include position, orientation, velocity, acceleration, or rotation, or any combination thereof. 
     
     
         15 . The computing device of  claim 14 , wherein the at least one sensor module includes a plurality of sensor modules attached to a subset of a plurality of parts of the user forming a kinematic chain on the user; and the second measurements include motion states of at least one part on the kinematic chain that has no sensor module being attached to the user. 
     
     
         16 . The computing device of  claim 14 , wherein the artificial neural network is trained to generate the second measurements based on third measurements from a separate tracking system generated in tracking sample user movements in using the application; and the separate tracking system is not used during performance of the method. 
     
     
         17 . The computing device of  claim 16 , wherein the processor is further configured via the instructions to:
 receive first data representative of measurements generated by at least one inertial measurement unit configured in the at least one sensor module;   receive second data representative of images of the at least one sensor module captured by a camera configured on a head mounted display;   generate the first measurement by combining the first data and the second data using a filter.   
     
     
         18 . The computing device of  claim 16 , wherein the artificial neural network includes:
 a first portion having a convolution neural network to process an optical measurement sequence;   a second portion having a recurrent neural network to process an inertial measurement unit measurement sequence; and   a third portion having a recurrent neural network to combine inputs from the first portion and the second portion.   
     
     
         19 . A non-transitory computer storage medium storing instructions which, when executed on a computing device, cause the device to perform a method, comprising:
 receiving, in the computing device, first measurements of at least one sensor module attached respectively to at least one part of a user, wherein the first measurements are representative of first motion states of the at least one part of the user as measured, via the at least one sensor module, at one or more first time instances prior to the receiving of the first measurements;   predicting, by the computing device using an artificial neural network, second measurements representative of second motion states of the at least one part of the user at a second time instance that is after the predicting of the second measurements;   providing, prior to the second time instance, the second measurements as an input to an application running in the computing device;   rendering, by the application, a display of a virtual object to have third motion states corresponding to the second measurements predicted for the second time instance; and   presenting, by the computing device, the display of the virtual object to have the third motion states.   
     
     
         20 . The non-transitory computer storage medium of  claim 19 , wherein the first measurements include a portion generated using at least one inertial measurement unit, and a second portion generated using a camera configured on a head mounted display; and the artificial neural network includes:
 a first portion having a convolution neural network to process an optical measurement sequence;   a second portion having a recurrent neural network to process an inertial measurement unit measurement sequence; and   a third portion having a recurrent neural network to combine inputs from the first portion and the second portion.