Unmanned aerial system (UAS) autonomous terrain mapping and landing site detection

Abstract

A method, system, and apparatus for an unmanned aerial vehicle (UAV) to autonomously reconstruct overflown terrain and detect safe landing sites. A UAV autonomously acquires on-board pose estimates from an on-board visual-inertial-range odometry method during flight. The on-board pose estimates are utilized as a pose prior and to regain metric scale during three-dimensional (3D) reconstruction. The on-board pose estimates are corrected based on a bundle adjustment approach using previously acquired images. 3D reconstruction is performed based on multiple captured images taken from an on-board camera. Range data from the multiple captured images is fused into a multi-resolution height map. A safe landing site on the terrain is detected based on the multi-resolution height map.

What is claimed is: 
     
       1. A method for autonomously detecting unmanned aerial vehicle (UAV) landing sites, comprising:
 (a) the UAV autonomously acquiring pose estimates from an on-board visual-inertial-range odometry method via an on-board pose estimator during flight, wherein the on-board pose estimates are utilized as a pose prior and to regain metric scale during three-dimensional (3D) reconstruction; 
 (b) performing, onboard the UAV during flight, 3D reconstruction based on multiple captured images taken from an on-board camera during flight, wherein the 3D reconstruction generates range data from the multiple captured images, wherein multiple 3D reconstructions are performed onboard the UAV during flight; 
 (c) fusing, onboard the UAV during flight, range data from the multiple captured images into a multi-resolution height map whenever new range data becomes available, wherein the multi-resolution height map comprises different resolution layers wherein range measurements are inserted based on a measurement resolution represented by a pixel footprint of the 3D reconstruction; and 
 (d) detecting, onboard the UAV during flight, a safe landing site on the terrain based on the multi-resolution height map and an evaluation that uses the different resolution layers for detecting hazards of different sizes, wherein the detecting selects one of the different resolution layers based on a hazard size. 
 
     
     
       2. The method of  claim 1 , wherein the on-board pose estimates from the visual-inertial-range odometry method are not accurate enough to perform image-based 3D reconstruction. 
     
     
       3. The method of  claim 1 , further comprising correcting the on-board pose estimates based on a bundle adjustment approach, wherein the bundle adjustment approach:
 is based on three or more previously acquired images and corresponding poses from the on-board pose estimator; and 
 uses feature matches and initial poses in a sliding window approach using past keyframes and the most current image. 
 
     
     
       4. The method of  claim 1 , wherein the performing 3D reconstruction comprises:
 selecting the multiple captured images from a keyframe buffer; 
 rectifying the selected multiple captured images; 
 calculating 3D data based on the rectified selected multiple captured images, wherein the 3D data comprises a disparity map and the range data; and 
 providing the range data as a range image, wherein each pixel of a most current image, of the rectified selected multiple captured images, is associated with a 3D point in the range data. 
 
     
     
       5. The method of  claim 4 , wherein:
 the 3D data is associated to a pixel footprint that corresponds to a footprint of a pixel on overflown terrain; and 
 the fusing comprises:
 (i) selecting a level-of-detail (LOD) of the multi-resolution height map using the pixel footprint; 
 (ii) fusing a height measurement with map cell data in the selected LOD; 
 (iii) repeating steps (i) and (ii) until all measurements from the most current image are incorporated into the multi-resolution height map; and 
 (iv) updating the multi-resolution map using a pooling operation, such that all LOD levels have a correct current height estimate. 
 
 
     
     
       6. The method of  claim 5 , wherein:
 the fusing enables a hierarchical approach to finding a landing hazard; 
 the fusing enables detection of a first landing hazard at a lower resolution level of the multi-resolution height map; 
 the UAV is unable to perceive a second landing hazard at a higher resolution of the multi-resolution height map; and 
 the first landing hazard is larger than the second landing hazard. 
 
     
     
       7. The method of  claim 1 , wherein:
 the performing 3D reconstruction comprises selecting more than two multiple captured images; and 
 the 3D reconstruction comprises multi-image 3D reconstruction. 
 
     
     
       8. The method of  claim 1 , wherein:
 the fusing comprises a mixture of Gaussian approach. 
 
     
     
       9. The method of  claim 8 , wherein:
 the mixture of Gaussian approach is based on an uncertainty model that is within a threshold level of accuracy; and 
 the mixture of Gaussian approach decouples a fusion of single-layer measurements and updating other layers of the multi-resolution height map by a pooling operation. 
 
     
     
       10. The method of  claim 1 , wherein:
 the multi-resolution height map is anchored within a global frame; 
 the multi-resolution height map travels with the UAV; and 
 image data from the selected multiple images flows into and out of the multi-resolution height map using a rolling buffer. 
 
     
     
       11. The method of  claim 1 , wherein:
 the multi-resolution height map is anchored and fixed within a global frame. 
 
     
     
       12. The method of  claim 1 , wherein the detecting is based on:
 surface roughness; 
 slope; 
 uncertainty; and 
 sufficient space for the UAV. 
 
     
     
       13. The method of  claim 1 , wherein the detecting:
 filters the multi-resolution height map to produce candidate safe landing areas using a filter that filters for different hazard categories; and 
 selects a landing point in a safe area by:
 producing a single resolution safe landing area map based on the candidate safe landing areas; 
 applying a distance transform to identify a center of the candidate safe landing areas, from the safe landing area map, as candidate landing sites; 
 selecting a final landing site from the candidate landing sites based on a size of the candidate safe landing area that surrounds each candidate landing site; 
 improving the final landing site by shifting a location of the final landing site based on an estimated roughness of terrain around the final landing site. 
 
 
     
     
       14. An unmanned aerial vehicle (UAV) comprising:
 (a) an on-board camera that captures multiple captured images; 
 (b) an on-board pose estimator that:
 (i) deploys a visual-inertial-range odometry method that utilizes images acquired during flight for flights in GPS-denied environments; 
 (ii) deploys a GPS-inertial odometry method that utilizes global positioning system positions for flights where GPS signals are available; 
 (iii) acquires on-board pose estimates, wherein the on-board pose estimates are utilized as a pose prior and to regain metric scale during three-dimensional (3D) reconstruction; 
 
 (e) a processor; 
 (f) software executed by the processor, wherein the software causes the UAV to autonomously:
 (i) perform, onboard the UAV during flight, 3D reconstruction based on the multiple captured images taken from the on-board camera during flight, wherein the 3D reconstruction generates range data from the multiple captured images, wherein multiple 3D reconstructions are performed onboard the UAV during flight; 
 (iii) fuse, onboard the UAV during flight, range data from the multiple captured images into a multi-resolution height map whenever new range data becomes available, wherein the multi-resolution height map comprises different resolution layers wherein range measurements are inserted based on a measurement resolution represented by a pixel footprint of the 3D reconstruction; and 
 (iv) detect, onboard the UAV during flight, a safe landing site on the terrain based on the multi-resolution height map and an evaluation that uses the different resolution layers for detecting hazards of different sizes, wherein the detecting selects one of the different resolution layers based on a hazard size. 
 
 
     
     
       15. The UAV of  claim 14 , wherein the on-board pose estimates from the visual-inertial-range odometry method are not accurate enough to perform image-based 3D reconstruction. 
     
     
       16. The UAV of  claim 14 , the UAV further autonomously corrects the on-board pose estimates based on a bundle adjustment approach, wherein the bundle adjustment approach:
 is based on three or more previously acquired images and corresponding poses from the on-board pose estimator; and 
 uses feature matches and initial poses in a sliding window approach using past keyframes and the most current image. 
 
     
     
       17. The UAV of  claim 14 , wherein the software performs the 3D reconstruction by:
 selecting the multiple captured images from a keyframe buffer; 
 rectifying the selected multiple captured images; 
 calculating 3D data based on the rectified selected multiple captured images, wherein the 3D data comprises a disparity map and the range data; and 
 providing the range data as a range image, wherein each pixel of a most current image, of the rectified selected multiple captured images, is associated with a 3D point in the range data. 
 
     
     
       18. The UAV of  claim 17 , wherein:
 the 3D data is associated to a pixel footprint that corresponds to footprint of a pixel on overflown terrain; and 
 the software fuses 3D data with the multi-resolution height map by:
 (i) selecting a level-of-detail (LOD) of the multi-resolution height map using the pixel footprint; 
 (ii) fusing a height measurement with map cell data in the selected LOD; 
 (iii) repeating steps (i) and (ii) until all measurements from the most current image are incorporated into the multi-resolution height map; and 
 (iv) updating the multi-resolution map using a pooling operation, such that all LOD levels have a correct current height estimate. 
 
 
     
     
       19. The UAV of  claim 18 , wherein:
 the software fusing enables a hierarchical approach to finding a landing hazard; 
 the software fusing enables detection of a first landing hazard at a lower resolution level of the multi-resolution height map; 
 the UAV is unable to perceive a second landing hazard at a higher resolution of the multi-resolution height map; and 
 the first landing hazard is larger than the second landing hazard. 
 
     
     
       20. The UAV of  claim 17 , wherein:
 the software performs the 3D reconstruction by selecting more than two multiple captured images; and 
 the 3D reconstruction comprises multi-image 3D reconstruction. 
 
     
     
       21. The UAV of  claim 14 , wherein:
 the software fusing comprises a mixture of Gaussian approach. 
 
     
     
       22. The UAV of  claim 21 , wherein:
 the mixture of Gaussian approach is based on an uncertainty model that is within a threshold level of accuracy; and 
 the mixture of Gaussian approach decouples a fusion of single-layer measurements and updating other layers of the multi-resolution height map by a pooling operation. 
 
     
     
       23. The UAV of  claim 14 , wherein:
 the multi-resolution height map is anchored within a global frame; 
 the multi-resolution height map travels with the UAV; and 
 image data from the selected multiple images flows into and out of the multi-resolution height map using a rolling buffer. 
 
     
     
       24. The UAV of  claim 14 , wherein:
 the multi-resolution height map is anchored and fixed within a global frame. 
 
     
     
       25. The UAV of  claim 14 , wherein the software detecting is based on:
 surface roughness; 
 slope; 
 uncertainty; and 
 sufficient space for the UAV. 
 
     
     
       26. The UAV of  claim 14 , wherein the software detecting:
 filters the multi-resolution height map to produce candidate safe landing areas using a filter that filters for different hazard categories; and 
 selects a landing point in a safe area by:
 producing a single resolution safe landing area map based on the candidate safe landing areas; 
 applying a distance transform to identify a center of the candidate safe landing areas, from the safe landing area map, as candidate landing sites; 
 selecting a final landing site from the candidate landing sites based on a size of the candidate safe landing area that surrounds each candidate landing site; 
 improving the final landing site by shifting a location of the final landing site based on an estimated roughness of terrain around the final landing site.