Contour completion for augmenting surface reconstructions

Abstract

Surface reconstruction contour completion embodiments are described which provide dense reconstruction of a scene from images captured from one or more viewpoints. Both a room layout and the full extent of partially occluded objects in a room can be inferred using a Contour Completion Random Field model to augment a reconstruction volume. The augmented reconstruction volume can then be used by any surface reconstruction pipeline to show previously occluded objects and surfaces.

What is claimed is: 
     
         1 . A computer-implemented process for completing a partially reconstructed three dimensional scene (3D), comprising:
 receiving a dense partial reconstruction volume of a three dimensional scene;   detecting surfaces in the partial reconstruction volume;   classifying the surfaces into different types of surfaces;   determining the scene boundaries of the partial reconstruction volume of the scene using the classified surfaces;   determining boundaries of partially occluded surfaces of objects in the partial reconstruction volume using the classified surfaces; and   updating the partial reconstruction volume to show the determined boundaries of partially occluded surfaces of objects in the scene and the scene boundaries.   
     
     
         2 . The computer-implemented process of  claim 1  wherein the partial reconstruction is received from a surface reconstruction pipeline that maintains the partial reconstruction volume of the scene that is updated with new depth observations of the scene. 
     
     
         3 . The computer-implemented process of  claim 1 , further comprising using a contour completion process to update the contours of scene boundaries and the boundaries of the partially occluded surfaces of objects in the partial reconstruction volume. 
     
     
         4 . The computer-implemented process of  claim 3  wherein the contours of surfaces in the scene and objects in the scene are completed by using lines and parabolas fitted to visible parts of the contours of the surfaces. 
     
     
         5 . The computer-implemented process of  claim 4 , further comprising fitting parabolas to visible parts of contours of surfaces by:
 projecting the visible parts of the contours of the surfaces onto a two dimensional image;   identifying contact points in the image which fall at an intersection of free space, occupied and unknown pixels;   finding contour points in the image that are on a contour of a planar surface and bordering free space; and   enclosing an occluded contour by fitting a parabola so that it either connects to a set of contact points or continues from a contact point until it reaches an end of the image.   
     
     
         6 . The computer-implemented process of  claim 3 , wherein the contour completion process minimizes an energy function that determines whether a point lies on a contour of a surface or not and labels the point as being on the contour of the surface or not; and
 wherein two neighboring points that are assigned different labels as to belonging to a contour of a surface or not are assigned a penalty when determining whether the points belong to the contour of the surface.   
     
     
         7 . The computer-implemented process of  claim 1  wherein the dense partial reconstruction volume of the scene is expressed as a volume of voxels. 
     
     
         8 . The computer-implemented process of  claim 7  wherein every voxel is labeled as, occupied when having an observed surface at that voxel, free state when there is no observed surface at that voxel, or unknown. 
     
     
         9 . The computer-implemented process of  claim 1  wherein the dense partial reconstruction volume of the scene is expressed as a truncated signed distance function grid of voxels with a surface normal. 
     
     
         10 . The computer-implemented process of  claim 1  wherein each surface is classified into a semantic class. 
     
     
         11 . The computer-implemented process of  claim 10  wherein each surface is classified into a semantic class of floor, wall, ceiling and internal, wherein surfaces classified as one of floor, wall and ceiling are used to infer scene boundaries, and wherein internal surfaces are extended and filled. 
     
     
         12 . The computer-implemented process of  claim 10  wherein the planar surfaces are classified into the semantic classes using a trained classifier which predicts each surface's class using ground truth labels and 3D features and which captures attributes of each surface including its height, size and surface normal distribution. 
     
     
         13 . The computer-implemented process of  claim 13  wherein the classifier is a Random Forest classifier. 
     
     
         14 . A computer-implemented process for completing a dense partial 3D reconstruction of a scene, comprising:
 detecting surfaces in the dense partial 3D reconstruction volume of a scene;   classifying the detected surfaces into different types of surfaces;   augmenting the dense partial 3D reconstruction volume of the scene to show greater portions of partially occluded surfaces of objects in the scene and the scene boundaries using the classified surfaces and a contour completion procedure.   
     
     
         15 . The computer-implemented process of  claim 14 , further comprising continuously updating with new data the augmented 3D reconstruction volume as the scene changes. 
     
     
         16 . The computer-implemented process of  claim 14 , further comprising phasing out augmented portions of the augmented partial 3D reconstruction volume as the augmented portions of the 3D reconstruction volume become known. 
     
     
         17 . A system for completing a partial 3D reconstruction of a scene, comprising:
 one or more computing devices; and   a computer program having program modules executable by the one or more computing devices, the one or more computing devices being directed by the program modules of the computer program to,   classify planar surfaces detected in a partial 3D reconstruction of a scene; and   update the partial 3D reconstruction to show greater portions of partially occluded surfaces of objects in the scene and the scene boundaries by using the classified planar surfaces to complete visible contours of the planar surfaces.   
     
     
         18 . The system of  claim 17  wherein the detected planar surfaces are updated in real time. 
     
     
         19 . The system of  claim 18  wherein the 3D partial reconstruction of a scene is received via a network from an on-line surface reconstruction pipeline. 
     
     
         20 . The system of  claim 17  wherein the visible contours of the planar surfaces are completed by fitting one or more parabolas.