Systems and Methods for Stereo Depth Sensing

Abstract

A method includes determining, based on a similarity measure, correlation scores between a target pixel and each of a candidate matching pixel, a left neighboring pixel of the candidate matching pixel, and a right neighboring pixel of the candidate matching pixel. When the correlation scores indicate that a neighboring pixel of the candidate matching pixel is better correlated to the target pixel than the candidate matching pixel, (i) a correlation score is determined between the target pixel and a second neighboring pixel of the neighboring pixel that is opposite the candidate matching pixel in the second image and (ii) an interpolation is fit to the correlation scores of the neighboring pixel, the second neighboring pixel, and the candidate matching pixel.

What is claimed is: 
     
         1 . A method comprising:
 receiving a first image from a first imager and a second image from a second imager;   providing the first image and the second image to stereo matching hardware;   receiving, from the stereo matching hardware, a candidate matching pixel to a target pixel from a given row of the second image;   determining, based on a similarity measure, correlation scores between the target pixel and each of the candidate matching pixel, a left neighboring pixel of the candidate matching pixel, and a right neighboring pixel of the candidate matching pixel;   determining a location between neighboring pixels in the second image at which an interpolation has a local minimum or maximum value,   wherein, when the correlation scores indicate that the candidate matching pixel is better correlated to the target pixel than each of the left neighboring pixel and the right neighboring pixel of the candidate matching pixel, the interpolation is fit to the correlation scores of the candidate matching pixel, the left neighboring pixel, and the right neighboring pixel,   and wherein, when the correlation scores indicate that a neighboring pixel of the candidate matching pixel is better correlated to the target pixel than the candidate matching pixel, (i) a correlation score is determined between the target pixel and a second neighboring pixel of the neighboring pixel that is opposite the candidate matching pixel in the second image and (ii) the interpolation is fit to the correlation scores of the neighboring pixel, the second neighboring pixel, and the candidate matching pixel; and   based on the determined location between neighboring pixels in the second image, determining a disparity value indicating how far apart a portion of an environment represented by the target pixel appears between the first image and the second image.   
     
     
         2 . The method of  claim 1 , wherein the stereo matching hardware uses a first block size to determine a first census feature, wherein determining the correlation scores based on the similarity measure comprises using a second block size to determine a second census feature, wherein the second block size is larger than the first block size. 
     
     
         3 . The method of  claim 2 , wherein the first block size comprises a 5×5 window, and wherein the second block size comprises a 7×7 window. 
     
     
         4 . The method of  claim 1 , wherein the stereo matching hardware uses a first census feature having a first feature size, and wherein the similarity measure comprises a second census feature having a second feature size that is greater than the first feature size. 
     
     
         5 . The method of  claim 1 , wherein the interpolation is a linear V-interpolation. 
     
     
         6 . The method of  claim 1 , wherein the interpolation is a quadratic interpolation. 
     
     
         7 . The method of  claim 1 , further comprising:
 determining a disparity map between the first image and the second image based on the disparity value; and   providing the disparity map between the first image and the second image.   
     
     
         8 . The method of  claim 1 , further comprising:
 determining a depth map of a portion of the environment of the first imager and the second imager based on the disparity value; and   providing the depth map of the portion of the environment.   
     
     
         9 . The method of  claim 1 , further comprising:
 determining a distance to an object in the environment of the first imager and the second imager based on the disparity value; and   providing the distance to the object in the environment.   
     
     
         10 . The method of  claim 1 , further comprising controlling operation of a robotic device based on the determined disparity value. 
     
     
         11 . The method of  claim 1 , wherein the stereo matching hardware uses a first similarity measure, and wherein the similarity measure involves sampling the first image and the second image with a greater pixel block size than the first similarity measure. 
     
     
         12 . The method of  claim 1 , wherein the stereo matching hardware uses a first similarity measure, wherein the first similarity measure is more efficient to compute than the similarity measure in terms of time or space requirements. 
     
     
         13 . A system comprising:
 a first imager;   a second imager; and   a control system configured to:   receive a first image from the first imager and a second image from the second imager;   provide the first image and the second image to stereo matching hardware;   receive, from the stereo matching hardware, a candidate matching pixel to a target pixel from a given row of the second image;   determine, based on a similarity measure, correlation scores between the target pixel and each of the candidate matching pixel, a left neighboring pixel of the candidate matching pixel, and a right neighboring pixel of the candidate matching pixel;   determine a location between neighboring pixels in the second image at which an interpolation has a local minimum or maximum value,   wherein, when the correlation scores indicate that the candidate matching pixel is better correlated to the target pixel than each of the left neighboring pixel and the right neighboring pixel of the candidate matching pixel, the interpolation is fit to the correlation scores of the candidate matching pixel, the left neighboring pixel, and the right neighboring pixel,   and wherein, when the correlation scores indicate that a neighboring pixel of the candidate matching pixel is better correlated to the target pixel than the candidate matching pixel, (i) a correlation score is determined between the target pixel and a second neighboring pixel of the neighboring pixel that is opposite the candidate matching pixel in the second image and (ii) the interpolation is fit to the correlation scores of the neighboring pixel, the second neighboring pixel, and the candidate matching pixel; and   based on the determined location between neighboring pixels in the second image, determine a disparity value indicating how far apart a portion of an environment represented by the target pixel appears between the first image and the second image.   
     
     
         14 . The system of  claim 13 , wherein the first imager and the second imager are aligned at a same height. 
     
     
         15 . The system of  claim 13 , wherein the first imager and the second imager are coupled to a robotic device, and wherein the control system is configured to control the robotic device based on the determined disparity value. 
     
     
         16 . The system of  claim 13 , wherein the stereo matching hardware uses a first block size to determine a first census feature, wherein determining the correlation scores based on the similarity measure comprises using a second block size to determine a second census feature, wherein the second block size is larger than the first block size. 
     
     
         17 . The system of  claim 13 , wherein the stereo matching hardware uses a first census feature having a first feature size, and wherein the similarity measure comprises a second census feature having a second feature size that is greater than the first feature size. 
     
     
         18 . The system of  claim 13 , wherein the stereo matching hardware uses a first similarity measure, and wherein the similarity measure involves sampling the first image and the second image with a greater pixel block size than the first similarity measure. 
     
     
         19 . The system of  claim 13 , wherein the stereo matching hardware uses a first similarity measure, wherein the first similarity measure is more efficient to compute than the similarity measure in terms of time or space requirements. 
     
     
         20 . A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a computing system to perform functions comprising:
 receiving a first image from a first imager and a second image from a second imager;   providing the first image and the second image to stereo matching hardware;   receiving, from the stereo matching hardware, a candidate matching pixel to a target pixel from a given row of the second image;   determining, based on a similarity measure, correlation scores between the target pixel and each of the candidate matching pixel, a left neighboring pixel of the candidate matching pixel, and a right neighboring pixel of the candidate matching pixel;   determining a location between neighboring pixels in the second image at which an interpolation has a local minimum or maximum value,   wherein, when the correlation scores indicate that the candidate matching pixel is better correlated to the target pixel than each of the left neighboring pixel and the right neighboring pixel of the candidate matching pixel, the interpolation is fit to the correlation scores of the candidate matching pixel, the left neighboring pixel, and the right neighboring pixel,   and wherein, when the correlation scores indicate that a neighboring pixel of the candidate matching pixel is better correlated to the target pixel than the candidate matching pixel, (i) a correlation score is determined between the target pixel and a second neighboring pixel of the neighboring pixel that is opposite the candidate matching pixel in the second image and (ii) the interpolation is fit to the correlation scores of the neighboring pixel, the second neighboring pixel, and the candidate matching pixel; and   based on the determined location between neighboring pixels in the second image, determining a disparity value indicating how far apart a portion of an environment represented by the target pixel appears between the first image and the second image.