Depth data adjustment based on non-visual pose data

Abstract

An HMD adjusts adjusting depth information based on detected motion of the system. The HMD includes a depth camera that collects depth data for objects in the local environment of the HMD. The HMD further includes an inertial measurement unit (IMU) including non-visual motion sensors such as one or more accelerometers, gyroscopes, and the like. The HMD adjusts the received depth information based on motion data provided by the IMU, thereby improving the accuracy of the depth information, and in turn reducing visual artifacts that can result from inaccuracies in the depth information.

1 .- 20 . (canceled) 
     
     
         21 . A method comprising:
 identifying a first motion of a head mounted display (HMD);   adjusting a first raw image received from a depth camera of the HMD based on the first motion to generate a first adjusted image;   generating a first ray based on a center of projection and a focal length of the depth camera;   transforming the first ray based on the first motion to generate a second ray;   generating adjusted depth data based on a plurality of images including the first adjusted image, wherein the adjusted depth data is generated using a point-to-pixel representation of the second ray, the point-to-pixel representation being based on a first index representation of the second ray, a second index representation of the second ray, and a third index representation of the second ray; and   providing content at the HMD based on the adjusted depth data.   
     
     
         22 . The method of  claim 21 , wherein identifying the first motion comprises:
 identifying the first motion based on non-visual sensor data received from sensors of the HMD.   
     
     
         23 . The method of  claim 22 , wherein the sensors of the HMD comprise an inertial measurement unit (IMU) of the HMD. 
     
     
         24 . The method of  claim 21 , wherein adjusting the first raw image comprises:
 adjusting the first raw image based on the center of projection and the focal length.   
     
     
         25 . The method of  claim 24 , wherein adjusting the first image further comprises:
 adjusting the first raw image based on the second ray.   
     
     
         26 . The method of  claim 25 , wherein adjusting the first raw image further comprises:
 generating a set of adjusted pixel values based on the second ray; and   storing the set of adjusted pixel values as a frame of the first adjusted image.   
     
     
         27 . The method of  claim 21 , wherein adjusting the first raw image comprises:
 adjusting a first pixel of a frame of the first raw image by a first amount; and   adjusting a second pixel of the frame of the first raw image by a second amount different from the first amount.   
     
     
         28 . The method of  claim 21 , wherein the content comprises augmented reality (AR) content. 
     
     
         29 . A head mounted display (HMD) comprising:
 a motion detection module to identify a first motion of the HMD;   a depth camera to generate a first raw image;   a depth adjustment module to adjust the first raw image based on the first motion to generate a first adjusted image and to generate first adjusted depth data based on the first adjusted image by:
 generating a first ray based on a center of projection and a focal length of the depth camera; 
 transforming the first ray based on the first motion to generate a second ray, wherein the first adjusted depth data is generated using a point-to-pixel representation of a second ray, the point-to-pixel representation being based on a first index representation of the second ray, a second index representation of the second ray, and a third index representation of the second ray; and 
   a content generator to generate augmented reality (AR) content based on the first adjusted depth data.   
     
     
         30 . The HMD of  claim 29 , wherein the motion detection module indicates the first motion based on non-visual sensor data. 
     
     
         31 . The HMD of  claim 30 , wherein the motion detection module an inertial measurement unit (IMU). 
     
     
         32 . The HMD of  claim 29 , wherein the depth adjustment module adjusts the first raw image by:
 adjusting the first raw image based on the center of projection and the focal length.   
     
     
         33 . The HMD of  claim 32 , wherein the depth adjustment module adjusts the first raw image by:
 adjusting the first raw image based on the second ray.   
     
     
         34 . The HMD of  claim 33 , wherein the depth adjustment module adjusts the first raw image by:
 generating a set of adjusted pixel values based on the second ray; and   storing the set of adjusted pixel values as a frame of the adjusted depth data.   
     
     
         35 . The HMD of  claim 29 , wherein the depth adjustment module adjusts the first raw image by:
 adjusting a first pixel of a frame of the first raw image by a first amount; and   adjusting a second pixel of the frame of the first raw image by a second amount different from the first amount.   
     
     
         36 . A non-transitory computer readable medium embodying instructions to manipulate a processor to execute a method comprising:
 identifying a first motion of a head mounted display (HMD);   adjusting a first raw image received from a depth camera of the HMD based on the first motion to generate a first adjusted image;   generating a first ray based on a center of projection and a focal length of the depth camera;   transforming the first ray based on the first motion to generate a second ray;   generating adjusted depth data based on the first adjusted image, wherein the adjusted depth data is generated using a point-to-pixel representation of a second ray, the point-to-pixel representation being based on a first index representation of the second ray, a second index representation of the second ray, and a third index representation of the second ray; and   providing augmented reality (AR) content at the HMD based on the adjusted depth data.   
     
     
         37 . The non-transitory computer readable medium of  claim 36 , wherein identifying the first motion comprises:
 identifying the first motion based on non-visual sensor data received from sensors of the HMD.   
     
     
         38 . The non-transitory computer readable medium of  claim 37 , wherein the sensors of the HMD comprise an inertial measurement unit (IMU) of the HMD. 
     
     
         39 . The non-transitory computer readable medium of  claim 38 , wherein adjusting the first raw image comprises:
 adjusting the first raw image based on the center of projection and the focal length.   
     
     
         40 . The non-transitory computer readable medium of  claim 39 , wherein adjusting the first raw image further comprises:
 adjusting the first raw image based on the second ray.