Using energy model to enhance depth estimation with brightness image

Abstract

A depth estimation system can use an image energy model to enhance depth estimation using brightness image. Light is projected onto an object. The object reflects at least a portion of the projected light. The reflected light is at least partially captured by an image sensor. The depth estimation system may generate a depth image based on a phase shift between the captured light and the projected light and generate a brightness image based on brightness of the captured light. The depth estimation system may determine a fusion energy based on the depth image and the brightness image and minimize the fusion energy to determine a new depth value of a pixel. The depth estimation system can assign the new depth value to the pixel and generates an enhanced depth image. The enhanced depth image includes better depth estimation than the original depth image.

1 . A method, comprising:
 converting a depth image comprising a plurality of depth pixels into a disparity image comprising a plurality of disparity pixels, wherein a disparity value of each of the plurality of disparity pixel is determined based on a depth value of a different depth pixel of the plurality of depth pixels;   determining a boundary weight for a target depth pixel of the plurality of depth pixels based on a gradient magnitude of the target depth pixel in the depth image and a gradient magnitude of a brightness pixel in a brightness image;   determining an energy for the target depth pixel based on the boundary weight;   determining a new depth value of the target depth pixel by optimizing the energy; and   updating the depth image by assigning the new depth value to the target depth pixel.   
     
     
         2 . The method of  claim 1 , wherein:
 the brightness image and the depth image capture a same object,   the brightness image comprises a plurality of brightness pixels that includes the brightness pixel, and   each respective brightness pixel of the plurality of brightness pixels correspond to a respective depth pixel of the plurality of depth pixels.   
     
     
         3 . The method of  claim 1 , wherein the target depth pixel represents a same locus of an object as the brightness pixel. 
     
     
         4 . The method of  claim 1 , wherein determining the energy for the target depth pixel based on the boundary weight comprises:
 determining a spatial error energy for the target depth pixel based on the boundary weight,   wherein optimizing the energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel that is adjacent to the target depth pixel in the depth image.   
     
     
         5 . The method of  claim 4 , wherein determining the energy for the target depth pixel based on the boundary weight further comprises:
 determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image,   wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel.   
     
     
         6 . The method of  claim 1 , wherein the disparity value is proportional to a reciprocal of the depth value. 
     
     
         7 . The method of  claim 1 , wherein the depth image and the brightness image are generated based on image data from a same image sensor. 
     
     
         8 . The method of  claim 1 , further comprising:
 instructing an illuminator assembly to project modulated light into a local area including an object;   instructing a camera assembly to capture reflected light from at least a portion of the object; and   generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area.   
     
     
         9 . The method of  claim 8 , further comprising:
 generating the brightness image based on brightness of the reflected light.   
     
     
         10 . The method of  claim 8 , wherein the reflected light is first reflected light, and the method further comprises:
 instructing the camera assembly to capture second reflected light from at least the portion of the object; and   generating the brightness image based on brightness of the second reflected light,   wherein the second reflected light has a different wavelength from the first reflected light.   
     
     
         11 . A system, comprising:
 an illuminator assembly configured to project modulated light into a local area including an object;   a camera assembly configured to capture reflected light from at least a portion of the object; and   a controller configured to:
 generate a depth image from the reflected light, the depth image comprising a plurality of depth pixels and capturing at least a portion of the object, 
 generate a brightness image comprising a plurality of brightness pixels and capturing at least the portion of the object, each brightness pixel corresponding to a different depth pixel, 
 for each respective depth pixel of the plurality of depth pixels, determine a respective energy based on a gradient magnitude of the respective depth pixel in the depth image and a gradient magnitude of a brightness pixel in the brightness image, and 
 generate an enhanced depth image by fusing the depth image with the brightness image based on respective energies of the plurality of depth pixels. 
   
     
     
         12 . The system of  claim 11 , wherein fusing the depth image with the brightness image based on respective energies of the plurality of depth pixels comprises:
 for each respective depth pixel of the plurality of depth pixels, optimizing the respective energy.   
     
     
         13 . The system of  claim 12 , wherein the controller is configured to determine the respective energy based on the gradient magnitude of the respective depth pixel in the depth image and the gradient magnitude of the brightness pixel in the brightness image by:
 determining a spatial error energy for the respective depth pixel based on the gradient magnitude of the respective depth pixel in the depth image and the gradient magnitude of the brightness pixel in the brightness image,   wherein optimizing the respective energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the respective depth pixel and a depth value of another depth pixel that is adjacent to the respective depth pixel in the depth image.   
     
     
         14 . The system of  claim 11 , wherein the controller is configured to determine the respective energy based on the gradient magnitude of the respective depth pixel in the depth image and the gradient magnitude of the brightness pixel in the brightness image further by:
 determining a conditional error energy for the respective depth pixel based on a depth value of the respective depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image,   wherein the conditional error energy indicates a measure of uncertainty in the depth value of the respective depth pixel given the brightness value of the brightness pixel.   
     
     
         15 . The system of  claim 11 , wherein the controller is configured to generate the depth image and the brightness image based on the reflected light by:
 generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and   generating the brightness image based on brightness of the reflected light.   
     
     
         16 . One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:
 converting a depth image comprising a plurality of depth pixels into a disparity image comprising a plurality of disparity pixels, wherein a disparity value of each of the plurality of disparity pixel is determined based on a depth value of a different depth pixel of the plurality of depth pixels;   determining a boundary weight for a target depth pixel of the plurality of depth pixels based on a gradient magnitude of the target depth pixel in the depth image and a gradient magnitude of a brightness pixel in a brightness image;   determining an energy for the target depth pixel based on the boundary weight;   determining a new depth value of the target depth pixel by optimizing the energy; and   updating the depth image by assigning the new depth value to the target depth pixel.   
     
     
         17 . The one or more non-transitory computer-readable media of  claim 16 , wherein the operations further comprise:
 the brightness image and the depth image capture a same object,   the brightness image comprises a plurality of brightness pixels that includes the brightness pixel, and   each respective brightness pixel of the plurality of brightness pixels correspond to a respective depth pixel of the plurality of depth pixels.   
     
     
         18 . The one or more non-transitory computer-readable media of  claim 16 , wherein determining the energy for the target depth pixel based on the boundary weight comprises:
 determining a spatial error energy for the target depth pixel based on the boundary weight,   wherein optimizing the energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel that is adjacent to the target depth pixel in the depth image.   
     
     
         19 . The one or more non-transitory computer-readable media of  claim 18 , wherein determining the energy for the target depth pixel based on the boundary weight further comprises:
 determining a conditional error energy for the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image,   wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the brightness value of the brightness pixel.   
     
     
         20 . The one or more non-transitory computer-readable media of  claim 16 , wherein the depth image and the brightness image are generated based on image data from a same image sensor.