On-Vehicle Driving Behavior Modelling

Abstract

This application is directed to on-vehicle behavior modeling of vehicles. A vehicle has one or more processors, memory, a plurality of sensors, and a vehicle control system. The vehicle collects training data via the plurality of sensors, and the training data include data for one or more vehicles during a collection period. The vehicle locally applies machine learning to train a vehicle driving behavior model using the collected training data. The vehicle driving behavior model is configured to predict a behavior of one or more vehicles. The vehicle subsequently collecting sensor data from the plurality of sensors and drives the vehicle by applying the vehicle driving behavior model to predict vehicle behavior based on the collected sensor data. The vehicle driving behavior model is configured to predict behavior of an ego vehicle and/or a distinct vehicle that appears near the ego vehicle.

What is claimed is: 
     
         1 . A method for on-vehicle modeling, comprising:
 at a first vehicle having one or more processors, memory, and a plurality of sensors:
 collecting training data via the plurality of sensors; 
 locally at the first vehicle, training a vehicle driving model using the collected training data and computing resources that are not being used to at least partially autonomously drive the first vehicle, wherein computing resources are prioritized for driving of the first vehicle over training of the vehicle driving model; 
 subsequently collecting sensor data from the plurality of sensors; and 
 at least partially autonomously driving the first vehicle using the vehicle driving model based on the collected sensor data from the plurality of sensors. 
   
     
     
         2 . The method of  claim 1 , wherein:
 the one or more vehicles include the first vehicle, and the collected training data includes data for the first vehicle;   the vehicle driving model includes a vehicle driving behavior model configured to predict behavior of the first vehicle; and   the method comprises applying the vehicle driving behavior model to the collected sensor data to predict vehicle behavior of the first vehicle.   
     
     
         3 . The method of  claim 1 , wherein:
 the one or more vehicles include a plurality of second vehicles that are near the first vehicle, and the collected training data includes data for the plurality of second vehicles;   the vehicle driving model includes a vehicle driving behavior model configured to predict behavior of a third vehicle that appears near the first vehicle; and   the method comprises applying the vehicle driving behavior model to the collected sensor data to predict vehicle behavior of the third vehicle.   
     
     
         4 . The method of  claim 1 , wherein:
 the plurality of sensors include a camera, and the collected training data includes a plurality of images captured by the camera;   the one or more vehicles include a plurality of second vehicles that are near the first vehicle and correspond to a plurality of vehicle types; and   the vehicle driving model is trained using the plurality of images to predict behavior of the plurality of second vehicles based on the plurality of vehicle types.   
     
     
         5 . The method of  claim 1 , wherein the collected training data includes a first set of training data captured during a first duration of time and a second set of training data captured during a second duration of time following the first duration of time, and training the vehicle driving model further includes:
 determining behavior of the one or more vehicles from the first set of training data using the vehicle driving behavior model;   comparing the behavior of the one or more vehicles with the second set of training data; and   modifying one or more weights of a neural network of the vehicle driving model based on a comparison result.   
     
     
         6 . The method of  claim 1 , wherein:
 the one or more vehicles include one or more second vehicles near the first vehicle during a collection period;   the plurality of sensors includes at least a light detection and ranging (LiDAR) scanner and an inertial navigation system (INS) including accelerometers and gyroscopes; and   the training data includes relative positions that are measured between the first vehicle and the one or more second vehicles by the LiDAR scanner and relative motions measured between the first vehicle and the one or more second vehicles by the INS during the collection period.   
     
     
         7 . The method of  claim 1 , wherein at least partially autonomously driving the vehicle further includes applying the vehicle driving model to predict vehicle behavior based on the collected sensor data using a subset of the computing resources. 
     
     
         8 . The method of  claim 1 , wherein:
 the vehicle driving model is trained, before the sensor data are collected from the plurality of sensors and the vehicle driving model is used to predict vehicle behavior; and   the sensor data are added to the training data to iteratively train the vehicle driving model, after applying the vehicle driving model to predict the vehicle behavior.   
     
     
         9 . The method of  claim 1 , wherein the vehicle driving model includes a first vehicle driving model, and the first vehicle driving model is trained using the collected training data to generate a second vehicle driving model, concurrently when the first vehicle driving model is used to predict vehicle behavior from the collected sensor data, the method further comprising:
 adding the collected sensor data to the collected training data.   
     
     
         10 . The method of  claim 9 , the collected sensor data including first sensor data, the method further comprising:
 collecting second sensor data from the plurality of sensors after collecting the first sensor data; and   while continuing to train the second vehicle driving model with the training data added with the first sensor data, applying the second vehicle driving model to predict the vehicle behavior based on the second sensor data.   
     
     
         11 . A vehicle, comprising:
 a plurality of sensors;   one or more processors; and   memory storing one or more programs configured for execution by the one or more processors, the one or more programs comprising instructions for:
 collecting training data via the plurality of sensors; 
 locally at the vehicle, training a vehicle driving model using the collected training data and computing resources that are not being used to at least partially autonomously drive the vehicle, wherein computing resources are prioritized for driving of the vehicle over training of the vehicle driving model; 
 subsequently collecting sensor data from the plurality of sensors; and 
 at least partially autonomously driving the vehicle using the vehicle driving model based on the collected sensor data from the plurality of sensors. 
   
     
     
         12 . The vehicle of  claim 11 , wherein a vehicle control model is used together with the vehicle driving model to at least partially autonomously drive the vehicle. 
     
     
         13 . The vehicle of  claim 12 , the one or more programs further comprising instructions for:
 applying the vehicle driving model to predict vehicle behavior based on the collected sensor data; and   in accordance with a determination that the vehicle operates in an at least partial autonomous driving mode:
 providing the predicted vehicle behavior to the vehicle control model; and 
 based on the predicted vehicle behavior, applying the vehicle control model to generate a vehicle control instruction for controlling the vehicle. 
   
     
     
         14 . The vehicle of  claim 11 , the one or more programs further comprising instructions for:
 providing the vehicle driving model that is trained using the collected training data to a server, wherein the server is communicatively coupled to the vehicle via one or more communication networks.   
     
     
         15 . The vehicle of  claim 14 , wherein the server is configured to consolidate a plurality of driving models provided by a plurality of vehicles, generate a comprehensive driving model, and provide the comprehensive driving model to one or more target vehicles. 
     
     
         16 . A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a vehicle, the vehicle further including a plurality of sensors, the one or more programs comprising instructions for:
 collecting training data via the plurality of sensors;   locally at the vehicle, training a vehicle driving model using the collected training data and computing resources that are not being used to at least partially autonomously drive the vehicle, wherein computing resources are prioritized for driving of the vehicle over training of the vehicle driving model;   subsequently collecting sensor data from the plurality of sensors; and   at least partially autonomously driving the vehicle using the vehicle driving model based on the collected sensor data from the plurality of sensors   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 16 , wherein:
 the vehicle has a plurality of vehicle parameters, and each of the plurality of sensors and a vehicle control system operates according to a respective vehicle parameter; and   the vehicle driving model is coupled to a vehicle adjustment module that is configured to determine the plurality of vehicle parameters.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , the one or more programs further comprising instructions for:
 applying the vehicle driving model to predict vehicle behavior based on the collected sensor data;   providing the predicted vehicle behavior to the vehicle adjustment module;   updating by the vehicle adjustment module the plurality of vehicle parameters based on the predicted vehicle behavior; and   generating a vehicle adjustment instruction based on the plurality of vehicle parameters that are updated.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 16 , the one or more programs further comprising instructions for:
 collecting data for road objects that appear on a road and in measurement ranges of the plurality of sensors, the road objects including one or more of: lane lines, shoulder lines, road dividers, traffic lights, traffic signs, road signs, cones, a pedestrian, a bicycle, and a driver of the vehicle, wherein the data for road objects is applied to train one or more vehicle models.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 16 , wherein the plurality of sensors include one or more of: a global positioning system (GPS), a light detection and ranging (LiDAR) scanner, one or more cameras, a radio detection and ranging (RADAR) sensor, an infrared sensor, one or more ultrasonic sensors, a dedicated short-range communication (DSRC) module, an inertial navigation system (INS) including accelerometers and gyroscopes, and an odometry sensor.