Pixelwise Filterable Depth Maps for Robots

Abstract

A method includes receiving sensor data from a plurality of robot sensors on a robot. The method includes generating a depth map that includes a plurality of pixel depths. The method includes determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment. The method includes generating a pixelwise filterable depth map for a control system of the robot. The pixelwise filterable depth map is filterable to produce a robot operation specific depth map. The robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.

What is claimed is: 
     
         1 . A method comprising:
 receiving sensor data from a plurality of robot sensors on a robot, wherein the sensor data comprises a plurality of pixels that represent an environment of the robot;   generating a depth map comprising a plurality of pixel depths, wherein each pixel depth is determined based on the sensor data, and wherein each pixel depth is associated with at least one robot sensor of the plurality of robot sensors;   determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment; and   generating a pixelwise filterable depth map for a control system of the robot, wherein the pixelwise filterable depth map is filterable to produce a robot operation specific depth map, wherein the robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.   
     
     
         2 . The method of  claim 1 , further comprising determining a set of confidence levels for the pixelwise confidence levels based on the plurality of robot sensors, wherein a first confidence level in the set of confidence levels corresponds to a first sensor and wherein a second confidence level in the set of confidence levels corresponds to a second sensor. 
     
     
         3 . The method of  claim 2 , wherein the first confidence level is above the confidence threshold and the second confidence level is below the confidence threshold, the method further comprising filtering out pixels from the pixelwise filterable depth map that correspond to the second sensor. 
     
     
         4 . The method of  claim 3 , wherein the first sensor comprises a stereoscopic image capture device and wherein the second sensor comprises a monoscopic image capture device. 
     
     
         5 . The method of  claim 3 , wherein the first sensor comprises a light ranging and detection (LIDAR) device and wherein the second sensor comprises an image capture device. 
     
     
         6 . The method of  claim 1 , wherein the depth map comprises a first region associated with a first sensor and a second region associated with a second sensor, wherein the second region is adjacent to the first region such that the first region and the second region adjoin at a plurality of edge pixel depths, and wherein determining the pixelwise confidence level for each pixel depth comprises:
 determining a confidence level for each pixel depth in the second region based on a distance between the pixel and the plurality of edge pixel depths.   
     
     
         7 . The method of  claim 1 , further comprising:
 receiving, via a user interface, a user-selected confidence threshold associated with the at least one operation of the robot, wherein the user-selected confidence threshold corresponds to the confidence threshold; and   filtering the pixelwise filterable depth map to produce a robot operation specific depth map associated with the user-selected confidence threshold.   
     
     
         8 . The method of  claim 1 , further comprising:
 initiating an operation of the robot;   filtering the pixelwise filterable depth map based on a confidence threshold associated with the operation of the robot to produce a robot operation specific depth map, wherein the robot operation specific depth map comprises a used portion comprising unfiltered pixel depths and an unused portion comprising filtered pixel depths; and   performing the operation of the robot using the unfiltered pixel depths in the used portion of the operation specific depth map.   
     
     
         9 . The method of  claim 8 , further comprising:
 determining a proportion of unfiltered pixel depths relative to a total number of pixel depths in the pixelwise filterable depth map; and   determining that the proportion of unfiltered pixel depths exceeds a threshold proportion of unfiltered pixel depths associated with the operation of the robot,   wherein performing the operation of the robot comprises, performing the operation of the robot responsive to determining that the proportion of unfiltered pixel depths exceeds the threshold proportion of unfiltered pixel depths.   
     
     
         10 . The method of  claim 8 , further comprising:
 determining that the used portion of the robot operation specific depth map comprises an uninterrupted region of unfiltered pixel depths that exceeds a threshold region size,   wherein performing the operation of the robot comprises, performing the operation of the robot responsive to determining that the used portion of the robot operation specific depth map comprises the uninterrupted region of unfiltered pixel depths that exceeds the threshold region size.   
     
     
         11 . The method of  claim 8 , wherein the operation of the robot comprises navigating the robot within the environment, the method further comprising:
 selecting a path for navigating the robot within the environment that corresponds to the used portion of the robot operation specific depth map.   
     
     
         12 . The method of  claim 11 , further comprising:
 navigating the robot within the environment;   changing the robot operation specific depth map while navigating the robot within the environment based on additional sensor data from the at least one sensor; and   updating the path for navigating the robot within the environment based on a change in the used portion of robot operation specific depth map.   
     
     
         13 . The method of  claim 1 , wherein a plurality of robot operations each correspond to a different confidence threshold, the method further comprising:
 receiving an instruction to perform a robot operation;   selecting a confidence threshold based on the robot operation; and   filtering the pixelwise filterable depth map based on the selected confidence threshold associated with the operation of the robot to produce a robot operation specific depth map.   
     
     
         14 . The method of  claim 1 , wherein determining, for each respective pixel depth, the pixelwise confidence level comprises:
 comparing pixel depths from two or more sensors;   determining a degree of similarity between the compared pixel depths for each respective pixel depth in the depth map; and   determining, for each respective pixel depth, the pixelwise confidence level based at least in part on determining the degree of similarity between the compared pixel depths for each respective pixel depth in the depth map.   
     
     
         15 . A robot comprising:
 a first sensor;   a second sensor;   a computing having one or more processors;   a non-transitory computer readable medium; and   program instructions stored on the non-transitory computer readable medium and executable by the one or more processors to:
 receive sensor data from the first sensor and the second sensor, wherein the sensor data comprises a plurality of pixels that represent an environment of the robot; 
 generate a depth map comprising a plurality of pixel depths, wherein each pixel depth is determined based on the sensor data, and wherein each pixel depth is associated with at least one robot sensor of the plurality of robot sensors; 
 determine, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment; and 
 generate a pixelwise filterable depth map for a control system of the robot, wherein the pixelwise filterable depth map is filterable to produce a robot operation specific depth map, wherein the robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot. 
   
     
     
         16 . The system of  claim 15 , the program instructions being further executable to:
 initiate an operation of the robot;   filter the pixelwise filterable depth map based on a confidence threshold associated with the operation of the robot to produce a robot operation specific depth map, wherein the robot operation specific depth map comprises a used portion comprising unfiltered pixel depths and an unused portion comprising filtered pixel depths; and   perform the operation of the robot using the unfiltered pixel depths in the used portion of the operation specific depth map.   
     
     
         17 . The system of  claim 16 , the program instructions being further executable to:
 determine a proportion of unfiltered pixel depths relative to a total number of pixel depths in the pixelwise filterable depth map; and   determine that the proportion of unfiltered pixel depths exceeds a threshold proportion of unfiltered pixel depths associated with the operation of the robot,   wherein performing the operation of the robot comprises performing the operation of the robot responsive to determining that the proportion of unfiltered pixel depths exceeds the threshold proportion of unfiltered pixel depths.   
     
     
         18 . The system of  claim 16 , the program instructions being further executable to:
 determine that the used portion of the robot operation specific depth map comprises an uninterrupted region of unfiltered pixel depths that exceeds a threshold region size,   wherein performing the operation of the robot comprises, performing the operation of the robot responsive to determining that the used portion of the robot operation specific depth map comprises the uninterrupted region of unfiltered pixel depths that exceeds the threshold region size.   
     
     
         19 . The system of  claim 16 , wherein the operation of the robot comprises navigating the robot within the environment, the program instructions being further executable to:
 select a path for navigating the robot within the environment that corresponds to the used portion of the robot operation specific depth map;   navigate the robot within the environment;   change the robot operation specific depth map while navigating the robot within the environment based on additional sensor data from the first senor and the second sensor; and   update the path for navigating the robot within the environment based on a change in the used portion of robot operation specific depth map.   
     
     
         20 . A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a computing system to perform functions comprising:
 receiving sensor data from a plurality of robot sensors on a robot, wherein the sensor data comprises a plurality of pixels that represent an environment of the robot;   generating a depth map comprising a plurality of pixel depths, wherein each pixel depth is determined based on the sensor data, and wherein each pixel depth is associated with at least one robot sensor of the plurality of robot sensors;   determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment; and   generating a pixelwise filterable depth map for a control system of the robot, wherein the pixelwise filterable depth map is filterable to produce a robot operation specific depth map, wherein the robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.