Semantic-based robotic navigation and manipulation in complex environments

Abstract

A method of and system for navigation and manipulation for a robot can include obtaining, by at least one camera and at least one depth sensor, a first visual data set and translating the first visual data set into a continuous three-dimensional map. The three-dimensional map can include semantic information and geometric information. The method and system may further include receiving instruction data and converting the instruction data into at least one task for the robot within the continuous three-dimensional map.

What is claimed is: 
     
         1 . A method of navigation for a robot, comprising:
 obtaining, by at least one camera and at least one depth sensor, a first visual data set;
 translating the first visual data set into a continuous three-dimensional map, wherein the continuous three-dimensional map comprises semantic information and geometric information; 
 receiving instruction data; and 
 converting the instruction data into at least one task for the robot within the continuous three-dimensional map. 
   
     
     
         2 . The method of  claim 1 , wherein the first visual data set comprises visual odometry data and red-green-blue-depth data. 
     
     
         3 . The method of  claim 1 , wherein translating the first visual data set includes generating, based on the first visual data set, an ellipsoid data set comprising a plurality of ellipsoids, wherein each ellipsoid in the ellipsoid data set comprises position data and covariance data. 
     
     
         4 . The method of  claim 3 , wherein translating the first visual data set includes projecting the ellipsoid data set onto a two-dimensional plane. 
     
     
         5 . The method of  claim 4 , wherein projecting the ellipsoid data set onto a two-dimensional plane includes color coding the semantic information and the geometric information into the three-dimensional map. 
     
     
         6 . The method of  claim 1 , wherein converting the instruction data includes classifying the continuous three-dimensional map into navigable and non-navigable spaces for the robot. 
     
     
         7 . The method of  claim 1 , wherein the converting includes identifying targets or locations within the three-dimensional map that the robot must reach to complete the at least one task. 
     
     
         8 . The method of  claim 1 , wherein the at least one task is selected based on a likelihood of success value. 
     
     
         9 . The method of  claim 1 , further comprising:
 moving the robot to perform the at least one task;   receiving, by the at least one camera or at least one depth sensor, a second visual data set; and   updating the three-dimensional map by incorporating the second visual data set into the first visual data set.   
     
     
         10 . A system comprising:
 a processor; and   a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the system to perform functions of:
 obtaining, by at least one camera and at least one depth sensor, a first visual data set; 
 translating the first visual data set into a continuous three-dimensional map, wherein the continuous three-dimensional map comprises semantic information and geometric information; 
 receiving instruction data; and 
 converting the instruction data into at least one task for a robot within the continuous three-dimensional map. 
   
     
     
         11 . The system of  claim 10 , wherein the first visual data set comprises visual odometry data and red-green-blue-depth data. 
     
     
         12 . The system of  claim 10 , wherein to translate the first visual data set, the memory further includes executable instruction that, when executed by the processor, cause the system to perform a function of generating, based on the first visual data set, an ellipsoid data set including a plurality of ellipsoids, wherein each ellipsoid in the ellipsoid data set comprises position data and covariance data. 
     
     
         13 . The system of  claim 12 , wherein to translate the first visual data set, the memory further includes executable instruction that, when executed by the processor, cause the system to perform a function of projecting the ellipsoid data set onto a two-dimensional plane. 
     
     
         14 . The system of  claim 13 , wherein to project the ellipsoid data set onto the two-dimensional plane, the memory further includes executable instruction that, when executed by the processor, cause the system to perform a function of color coding the semantic information and the geometric information into the three-dimensional map. 
     
     
         15 . The system of  claim 10 , wherein to convert the instruction data, the memory further includes executable instruction that, when executed by the processor, cause the system to perform a function of classifying the continuous three-dimensional map into navigable and non-navigable spaces for the robot. 
     
     
         16 . The system of  claim 10 , wherein the to convert the instruction data, the memory includes executable instruction that, when executed by the processor, cause the system to perform a function of identifying targets or locations within the three-dimensional map that the robot must reach to complete the at least one task. 
     
     
         17 . The system of  claim 10 , wherein the at least one task is selected based on a likelihood of success value. 
     
     
         18 . The system of  claim 10 , wherein the memory further comprises executable instructions that, when executed by the processor, cause the system to perform functions of:
 moving the robot to perform the at least one task;   receiving, by the at least one camera or at least one depth sensor, a second visual data set; and   updating the three-dimensional map by incorporating the second visual data set into the first visual data set.   
     
     
         19 . A non-transitory computer readable medium on which are stored instructions that when executed cause a programmable device to:
 obtain, by at least one camera and at least one depth sensor, a first visual data set;   translate the first visual data set into a continuous three-dimensional map, wherein the continuous three-dimensional map comprises semantic information and geometric information;   receive instruction data; and   convert the instruction data into at least one task for a robot within the continuous three-dimensional map.   
     
     
         20 . The non-transitory computer readable medium of  claim 19 , wherein the instructions when executed further cause the programmable device to:
 move the robot to perform the at least one task;   receive, by the at least one camera or at least one depth sensor, a second visual data set; and   update the three-dimensional map by incorporating the second visual data set into the first visual data set.