Analysis of objects of interest in sensor data using deep neural networks

Abstract

Sensor data captured by one or more sensors may be received at an analysis system. A neural network may be used to detect an object in the sensor data. A plurality of polygons surrounding the object may be generated in one or more subsets of the sensor data. A prediction of a future position of the object may be generated based at least in part on the polygons. One or more commands may be provided to a control system based on the prediction of the future position.

What is claimed is: 
     
         1 - 20 . (canceled) 
     
     
         21 . A movable device, comprising:
 one or more sensors configured to capture one or more image frames;   one or more processors and an associated memory, the memory storing a neural network configured to:
 receive the one or more image frames captured by the one or more sensors; 
 detect an object in the one or more image frames, including to:
 determine a probability that a portion of the object is positioned at a location in one of the one or more image frames; and 
 produce a post-processed image using one or more transformations of the image frame that removes one or more areas of the image frame that does not include the object; 
 
 generate a plurality of polygons surrounding the object in individual ones of the one or more image frames; 
 generate a prediction of a future position of the object based at least on the plurality of polygons; and 
 generate, based at least on the prediction of the future position of the object, a movement plan of the moveable device; and 
   a control system configured to perform one or more commands issued by the one or more processors to execute the movement plan.   
     
     
         22 . The movable device of  claim 21 , wherein the one or more processors is configured to:
 generate, based at least on the prediction of the future position of the object, a plurality of movement plans; and   select the movement plan from the plurality of movement plans based on application of a cost function to individual ones of the plurality of movement plans.   
     
     
         23 . The system as recited in  claim 21 , wherein to generate the prediction of the future position of the object, the neural network is configured to:
 obtain respective centroids and vertices for individual ones of the plurality of polygons; and   determine a position of a future polygon in a future image frame, based at least in part on the respective centroids and vertices.   
     
     
         24 . The movable device as recited in  claim 21 , wherein:
 the control system comprises a controller for a video camera, and   the one or more commands instruct the video camera to move or zoom to focus attention on the object.   
     
     
         25 . The system as recited in claim  1 , wherein:
 the control system comprises a motion control subsystem of a vehicle, and   the one or more commands comprise motion directives to the motion control subsystem to control movements of the vehicle.   
     
     
         26 . The system as recited in  claim 21 , wherein the one or more commands include a command to accelerate or decelerate the movable object. 
     
     
         72 . The system as recited in  claim 21 , wherein the one or more sensors include a Light Detection and Ranging (LIDAR) device. 
     
     
         28 . The system as recited in  claim 21 , wherein one or more processors implement an object tracker configured to detect and track objects of different types in the image frames. 
     
     
         29 . The system as recited in  claim 21 , wherein one or more processors is configured to generate a command to the control system in response to a detected state change in the object. 
     
     
         30 . The system as recited in  claim 21 , wherein:
 the object is another movable device, and   the movement plan is generated to avoid a collision between the movable device and the other movable device.   
     
     
         31 . A method, comprising:
 capturing, via one or more sensors of a movable device, one or more image frames;   performing, using a neural network implemented on one or more processors and an associated memory on the movable device:
 receiving the one or more image frames captured by the one or more sensors; 
 detecting an object in the one or more image frames, including:
 determining a probability that a portion of the object is positioned at a location in one of the one or more image frames; and 
 producing a post-processed image using one or more transformations of the image frame that removes one or more areas of the image frame that does not include the object; 
 
 generating a plurality of polygons surrounding the object in individual ones of the one or more image frames; 
 generating a prediction of a future position of the object based at least on the plurality of polygons; and 
 generating, based at least on the prediction of the future position of the object, a movement plan of the moveable device; and 
   performing, by a control system of the movable device, one or more commands issued by the one or more processors to execute the movement plan.   
     
     
         32 . The method as recited in  claim 31 , wherein:
 the control system comprises a controller for a video camera, and   the one or more commands instruct the video camera to move or zoom to focus attention on the object.   
     
     
         33 . The method as recited in  claim 31 , wherein:
 the control system comprises a motion control subsystem of a vehicle, and   the one or more commands comprise motion directives to the motion control subsystem to control movements of the vehicle.   
     
     
         34 . The method as recited in  claim 31 , wherein the one or more commands include a command to accelerate or decelerate the movable object. 
     
     
         35 . The method as recited in  claim 31 , wherein the one or more sensors include a Light Detection and Ranging (LIDAR) device. 
     
     
         36 . The method as recited in  claim 31 , further comprising tracking, via an object tracker implemented on the one or more processors, a plurality objects of different types in the image frames. 
     
     
         37 . The method as recited in  claim 31 , further comprising the one or more processors generating a command to the control system in response to a detected state change in the object. 
     
     
         38 . The method as recited in  claim 31 , wherein:
 the object is another movable device, and   the movement plan is generated to avoid a collision between the movable device and the other movable device.   
     
     
         39 . A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:
 receive one or more images via one or more sensors of a movable device;   use a neural network to implemented on a movable device to:
 detect an object in the one or more image frames, including to:
 determine a probability that a portion of the object is positioned at a location in one of the one or more image frames; and 
 produce a post-processed image using one or more transformations of the image frame that removes one or more areas of the image frame that does not include the object; 
 
 generate a plurality of polygons surrounding the object in individual ones of the one or more image frames; 
 generate a prediction of a future position of the object based at least on the plurality of polygons; and 
 generate, based at least on the prediction of the future position of the object, a movement plan of the moveable device; and 
   issue one or more commands to a control system of the movable device, wherein the one or more commands causes the control system to execute the movement plan.   
     
     
         40 . The non-transitory computer-accessible storage medium as recited in  claim 39 , wherein:
 the object is another movable device, and   the movement plan is generated to avoid a collision between the movable device and the other movable device.