Depth map with structured and flood light

Abstract

A method including receiving an image of a scene illuminated by both a predetermined structured light pattern and a flood fill illumination, generating an active brightness image of the scene based on the received image of the scene including detecting a plurality of dots of the predetermined structured light pattern, and removing the plurality of dots of the predetermined structured light pattern from the active brightness image, and generating a depth map of the scene based on the received image and the active brightness image.

1 . A method comprising:
 receiving an image of a scene illuminated by both a predetermined structured light pattern and a flood fill illumination;   generating an active brightness image of the scene based on the received image of the scene, wherein generating the active brightness image includes:
 generating a data representation of the received image; 
 detecting a plurality of dots of the predetermined structured light pattern within the data representation of the received image; and 
 inpainting pixels of the data representation of the received image corresponding to one or more of the plurality of dots of the predetermined structured light pattern with an intensity value; and 
   generating a depth map of the scene based on the received image, a prerecorded image of the predetermined structured light pattern, and the active brightness image.   
     
     
         2 . The method of  claim 1 , wherein inpainting pixels corresponding to a target dot of the plurality of dots includes sampling the intensity value from a pixel in the received image outside of the target dot of the plurality of dots. 
     
     
         3 . The method of  claim 2 , wherein the intensity value is sampled from a pixel in the received image that is within a threshold pixel distance of the target dot of the plurality of dots. 
     
     
         4 . The method of  claim 2 , wherein the intensity value is sampled from a next closest pixel outside of the target dot of the plurality of dots. 
     
     
         5 . The method of  claim 2 , wherein the intensity value is sampled from a pixel in the received image that is on a substantially similar depth plane as the target dot of the plurality of dots. 
     
     
         6 . The method of  claim 1 , wherein generating the depth map of the scene includes performing stereo matching between the received image and the prerecorded image of the predetermined structured light pattern. 
     
     
         7 . The method of  claim 6 , further comprising filtering the generated depth map of the scene based on the generated active brightness image. 
     
     
         8 . The method of  claim 1 , wherein generating the depth map of the scene includes:
 for each pixel in the received image, calculating support weights for each other pixel in a region of interest surrounding that pixel based on the generated active brightness image; and   performing stereo matching between the received image and the prerecorded image of the predetermined structured light pattern based on the regions of interest and calculated support weights for each pixel.   
     
     
         9 . The method of  claim 8 , wherein the support weights are calculated based on a similarity in intensity value in the active brightness image. 
     
     
         10 . The method of  claim 1 , wherein generating the depth map of the scene includes:
 calculating one or more feature vectors for each of the plurality of dots detected in the received image;   performing stereo matching between each of the plurality of dots in the received image and a corresponding plurality of dots in the prerecorded image of the predetermined structured light pattern based on the calculated feature vectors;   calculating depth values for pixels covered by each of the plurality of dots in the received image based on the stereo matching;   for each pixel in the received image outside of the plurality of dots:
 determining a region of interest surrounding that pixel; 
 calculating support weights for each pixel in that region of interest based on the generated active brightness image; 
 calculating a weighted depth median for that pixel based on calculated depth values of pixels covered by one or more of the plurality of dots within the region of interest and the calculated support weights; and 
 inpainting that pixel with the weighted depth median. 
   
     
     
         11 . The method of  claim 10 , wherein the one or more feature vectors for each of the plurality of dots are calculated based on a pixel distance to another of the plurality of dots. 
     
     
         12 . A computing device comprising:
 an illuminator configured to emit both a predetermined structured light pattern and a flood fill illumination on a scene;   a camera configured to capture an image of the scene illuminated by the predetermined structured light pattern and the flood fill illumination; and   a processor configured to:
 generate an active brightness image of the scene based on the captured image of the scene, wherein to generate the active brightness image the processor is configured to:
 generate a data representation of the captured image; 
 detect a plurality of dots of the predetermined structured light pattern within the data representation of the captured image; and 
 inpaint pixels of the data representation of the captured image corresponding to one or more of the plurality of dots of the predetermined structured light pattern with an intensity value; and 
 
 generate a depth map of the scene based on the captured image, a prerecorded image of the predetermined structured light pattern, and the active brightness image. 
   
     
     
         13 . The computing device of  claim 12 , wherein inpainting pixels corresponding to a target dot of the plurality of dots includes sampling the intensity value from a pixel in the received image outside of the target dot of the plurality of dots. 
     
     
         14 . The computing device of  claim 13 , wherein the intensity value is sampled from a pixel in the received image that is within a threshold pixel distance of the target dot of the plurality of dots. 
     
     
         15 . The computing device of  claim 13 , wherein the intensity value is sampled from a next closest pixel outside of the target dot of the plurality of dots. 
     
     
         16 . The computing device of  claim 13 , wherein the intensity value is sampled from a pixel in the received image that is on a substantially similar depth plane as the target dot of the plurality of dots. 
     
     
         17 . The computing device of  claim 12 , wherein to generate the depth map of the scene, the processor is further configured to perform stereo matching between the captured image and the prerecorded image of the predetermined structured light pattern. 
     
     
         18 . The computing device of  claim 17 , wherein the processor is further configured to filter the generated depth map of the scene based on the generated active brightness image. 
     
     
         19 . The computing device of  claim 12 , wherein to generate the depth map of the scene, the processor is further configured to:
 for each pixel in the captured image, calculate support weights for each other pixel in a region of interest surrounding that pixel based on the generated active brightness image; and   perform stereo matching between the captured image and the prerecorded image of the predetermined structured light pattern based on the regions of interest and calculated support weights for each pixel.   
     
     
         20 . A method comprising:
 concurrently emitting both a predetermined structured light pattern and a flood fill illumination in a predetermined band of the infrared light spectrum;   capturing an image of a scene concurrently illuminated by both the predetermined structured light pattern and the flood fill illumination;   generating an active brightness image of the scene based on the captured image of the scene, wherein generating the active brightness image includes:
 generating a data representation of the captured image; 
 detecting a plurality of dots of the predetermined structured light pattern within the data representation of the captured image; and 
 inpainting pixels of the data representation of the received image corresponding to one or more of the plurality of dots of the predetermined structured light pattern with an intensity value; and 
   generating a depth map of the scene based on the captured image, a prerecorded image of the structured light pattern, and the active brightness image.