Object Action Classification For Autonomous Vehicles

Abstract

Aspects of the disclosure relate to training and using a model for identifying actions of objects. For instance, LIDAR sensor data frames including an object bounding box corresponding to an object as well as an action label for the bounding box may be received. Each sensor frame is associated with a timestamp and is sequenced with respect to other sensor frames. Each given sensor data frame may be projected into a camera image of the object based on the timestamp associated with the given sensor data frame in order to provide fused data. The model may be trained using the fused data such that the model is configured to, in response to receiving fused data, the model outputs an action label for each object bounding box of the fused data. This output may then be used to control a vehicle in an autonomous driving mode.

1 . A method for controlling a vehicle having an autonomous driving mode, the method comprising:
 receiving, by one or more computing devices of the vehicle, a LIDAR sensor data frame including an object bounding box corresponding to an object;   projecting, by the one or more computing devices, the LIDAR sensor data frame into a projection image corresponding to a camera image of the object based on a timestamp associated with the LIDAR sensor data frame and a timestamp associated with the camera image in order to provide fused camera and LIDAR sensor data for the timestamp;   inputting, by the one or more computing devices, the fused camera and LIDAR sensor data into a model in order to determine an action label for the object; and   controlling, by the one or more computing devices, the vehicle in the autonomous driving mode based on the action label.   
     
     
         2 . The method of  claim 1 , wherein the object is a pedestrian. 
     
     
         3 . The method of  claim 2 , wherein the action label for the object include a label identifying that the pedestrian is walking. 
     
     
         4 . The method of  claim 2 , wherein the action label for the object include a label identifying that the pedestrian is running. 
     
     
         5 . The method of  claim 1 , wherein inputting the fused camera and LIDAR sensor data into the model further provides one or more historical action labels determined by the model for the pedestrian using fused camera and LIDAR sensor data from sensor data captured earlier in time from the LIDAR sensor data frame and the camera image. 
     
     
         6 . The method of  claim 5 , wherein the one or more historical action labels are also inputted into the model in order to determine the action label for the object. 
     
     
         7 . The method of  claim 1 , further comprising, using the action label in order to generate a trajectory of the vehicle, and wherein controlling the vehicle is further based on the trajectory. 
     
     
         8 . The method of  claim 1 , further comprising, providing the action label to a behavior model in order to predict a future behavior of the object, and wherein controlling the vehicle is further based on the predicted future behavior of the object. 
     
     
         9 . The method of  claim 1 , wherein the model is a deep neural network.