Method and electronic device for determining depth information of objects in a scene

Abstract

A method for determining depth information of objects in a scene, includes: generating RGB images having a first pixel density using an RGB camera; generating a depth image having a second pixel density using a depth camera, the second pixel density being lower than the first pixel density; generating an attention map using the RGB images with a high pixel density; merging the RGB images and the depth image to generate a merged image; determining an uncertainty score of each pixel of a plurality of pixels in the merged image; determining a threshold for the uncertainty score based on a level of uncertainty in the plurality of pixels; determining a depth value for each pixel of the plurality of pixels exceeding the threshold of the uncertainty score; applying the depth value of each pixel of the plurality of pixels in the merged image; and generating the scene with the depth information based on the depth value.

What is claimed is: 
     
         1 . A method for determining depth information of objects in a scene, the method comprising:
 generating RGB images having a first pixel density using an RGB camera;   generating a depth image having a second pixel density using a depth camera, the second pixel density being lower than the first pixel density;   generating an attention map using the RGB images with a high pixel density;   merging the RGB images and the depth image to generate a merged image;   determining an uncertainty score of each pixel of a plurality of pixels in the merged image;   determining a threshold for the uncertainty score based on a level of uncertainty in the plurality of pixels;   determining a depth value for each pixel of the plurality of pixels exceeding the threshold of the uncertainty score;   applying the depth value of each pixel of the plurality of pixels in the merged image; and   generating the scene with the depth information based on the depth value.   
     
     
         2 . The method of  claim 1 , wherein the attention map comprises semantically related pixels of the RGB images. 
     
     
         3 . The method of  claim 1 , wherein the generating the depth image comprises:
 illuminating the scene with a modulated light source using the RGB camera;   capturing a reflected light on a sensor associated with the depth camera at different phases with respect to the illuminated scene;   determining a phase shift between the illuminated scene and the reflected light at the different phases; and   collocating the illuminated scene with the reflected light on the sensor to generate the depth Image.   
     
     
         4 . The method of  claim 1 , wherein the determining the threshold for the uncertainty score comprises:
 determining each pixel value of the RGB images based on uncertainty;   assigning weights to each pixel of the plurality of pixels based on the level of the uncertainty; and   determining the level of the uncertainty in the plurality of pixels based on the assigned weights.   
     
     
         5 . The method of  claim 1 , wherein the generating the attention map comprises determining a region of interest having poor depth values. 
     
     
         6 . The method of  claim 1 , wherein the determining the depth value for each pixel exceeding the threshold of the uncertainty score comprises:
 determining a unified ordinal vector (UOV) to each pixel of the scene, wherein the UOV identifies missing depth pixel value through classification scene semantics; and   applying a regression method on the determined UOV to determine the depth value.   
     
     
         7 . The method of  claim 1 , wherein the determining the uncertainty score of each pixel of the plurality of pixels is based on Neutrosophic sets. 
     
     
         8 . An electronic device for determining depth information of objects in a scene, the electronic device comprising:
 a memory storing instructions;   an RGB camera;   a depth camera;   at least one processor operatively coupled with the memory, the RGB camera and the depth camera, wherein the least one processor is configured to execute the instructions to:   generate RGB images having a first pixel density using the RGB camera;   generate a depth image having a second pixel density using the depth camera, the second pixel density being lower than the first pixel density;   generate an attention map using the RGB images with a high pixel density;   merge the RGB images and the depth image to generate a merged image;   determine an uncertainty score of each pixel of a plurality of pixels in the merged image;   determine a threshold for the uncertainty score based on a level of uncertainty in the plurality of pixels;   determine a depth value for each pixel of the plurality of pixels exceeding the threshold of the uncertainty score;   apply the depth value for each pixel of the plurality of pixels in the merged image; and   generate the scene with the depth information based on the depth value.   
     
     
         9 . The electronic device of  claim 8 , wherein the attention map comprises semantically related pixels of the RGB images. 
     
     
         10 . The electronic device of  claim 8 , wherein the at least one processor is further configured to execute the instructions to generate the depth image having the second pixel density by:
 illuminating the scene with a modulated light source using the RGB camera;   capturing a reflected light on a sensor associated with the depth camera at different phases with respect to the illuminated scene;   determining a phase shift between the illuminated scene and the reflected light at the different phases; and   collocate the illuminated scene with the reflected light on the sensor to generate the depth image.   
     
     
         11 . The electronic device of  claim 8 , wherein the at least one processor is further configured to execute the instructions to determine the threshold for the uncertainty score by:
 determining each pixel value of the RGB images based on uncertainty;   assigning weights to each pixel of the plurality of pixels based on the level of the uncertainty; and   determining the level of the uncertainty in the plurality of pixels based on the assigned weights.   
     
     
         12 . The electronic device of  claim 8 , wherein the at least one processor is further configured to execute the instructions to generate the attention map by determining a region of interest having poor depth values. 
     
     
         13 . The electronic device of  claim 8 , wherein the at least one processor is further configured to execute the instructions to determine the depth value for each pixel of the plurality of pixels exceeding the threshold of the uncertainty score by:
 determining a unified ordinal vector (UOV) to each pixels of the scene, wherein the UOV identifies missing depth pixel value through classification scene semantics; and   applying a regression method on the determined UOV to determine the depth value.   
     
     
         14 . The electronic device of  claim 8 , wherein the at least one processor is further configured to execute the instructions to determine the uncertainty score of each pixel of the plurality of pixels based on a Neutrosophic sets.