Vehicle scenario mining for machine learning models

Abstract

Provided are methods for vehicle scenario mining for machine learning methods, which can include determining a set of attributes associated with an untested scenario for which a machine learning model of an autonomous vehicle is to make planned movements. The method includes searching a scenario database for the untested scenario based on the set of attributes. The scenario database includes a plurality of datasets representative of data received from an autonomous vehicle sensor system in which the plurality of datasets is marked with at least one attribute of the set of attributes. The method further includes obtaining the untested scenario from the scenario database for inputting into the machine learning model for training the machine learning model. The machine learning model is configured to make the planned movements for the autonomous vehicle. Systems and computer program products are also provided.

1 - 30 . (canceled) 
     
     
         31 . A system comprising:
 at least one data processor; and   at least one memory storing instructions, which when executed by the at least one data processor, result in operations comprising:
 determining a set of attributes associated with at least one of a vehicle, an environment in which the vehicle is operating, and an object in the environment, wherein at least one attribute among the set of attributes comprises at least one of a feature, a condition, or a characteristic; 
 executing a query at a scenario database to select one or more scenarios having the determined set of attributes, the one or more scenarios being selected based at least on an evaluation metric associated with a machine learning model trained to plan a movement of the at least one vehicle while the at least one vehicle encounters the one or more scenarios, wherein at least one scenario among the one or more scenarios comprises vehicle sensor data for a timeframe; 
 generating training data, wherein the training data comprises the one or more scenarios selected by the executed query as having the determined set of attributes; and 
 updating, based at least on the training data, the machine learning model trained to plan the movement of the at least one vehicle. 
   
     
     
         32 . The system of  claim 31 , wherein each scenario included in the scenario database includes a combination of attributes associated with the at least one of the vehicle, the environment in which the vehicle is operating, and the object in the environment. 
     
     
         33 . The system of  claim 32 , wherein the operations further comprise:
 extracting, from data logged by one or more vehicles, the combination of attributes associated with each scenario included in the scenario database, the data logged by the one or more vehicles including sensor data from the one or more vehicles.   
     
     
         34 . The system of  claim 33 , wherein the sensor data includes one or more of a LIDAR sensor data, camera sensor data, RADAR sensor data, and telemetrics sensor data. 
     
     
         35 . The system of  claim 32 , wherein each scenario in the scenario database further includes one or more frames, wherein each frame of the one or more frames is associated with at least one attribute, and wherein each frame of the one or more frames is further associated with a timestamp. 
     
     
         36 . The system of  claim 31 , wherein the one or more scenarios are selected based at least on the evaluation metric associated with the machine learning model satisfying one or more thresholds, at least one of the one or more thresholds comprising a threshold value for a sensor. 
     
     
         37 . The system of  claim 31 , wherein the evaluation metric associated with the machine learning model includes a frequency at which the machine learning model encounters at least one attribute associated with the one or more scenarios. 
     
     
         38 . The system of  claim 31 , wherein the evaluation metric associated with the machine learning model includes a difficulty level associated with the one or more scenarios, wherein the difficulty level is indicative of the amount of information or computations needed to complete the planned movement output by the machine learning model safely. 
     
     
         39 . The system of  claim 31 , wherein the evaluation metric associated with the machine learning model includes an error in an output of the machine learning model when planning the movement of the at least one vehicle while the at least one vehicle encounters the one or more scenarios, wherein the error indicates a difference in the planned movement and the actual movement of the vehicle, the error comprising an intensity and an error value. 
     
     
         40 . The system of  claim 31 , wherein the set of attributes include one or more vehicle level attributes comprising a body style, a means of powering, a wheelbase length, a track, a height, a speed, a distance to the object, a turning radius, a LiDAR sensor data, a radar sensor data, an audio data, inertial measurement units (IMUs), a global position system (GPS) data, real-time kinematics (RTK) data, Global Navigation Satellite System (GNSS) data, a latitude, a longitude, and/or a state in which the vehicle is licensed. 
     
     
         41 . The system of  claim 31 , wherein the set of attributes include one or more environment level attributes comprising weather condition, road condition, traffic condition, construction condition, intersection condition, a parked vehicle, an object in a roadway, a pedestrian, and/or an emergency siren. 
     
     
         42 . The system of  claim 31 , wherein the set of attributes include one or more object level attributes comprising a wheelbase length, a track, a height, a speed, a distance between the vehicle and the object, and a turning radius. 
     
     
         43 . The system of  claim 31 , wherein the set of attributes further include an indication that the vehicle is in cruise control mode, a duration of the vehicle being in cruise control mode, an indication of the vehicle engaging in a coasted stop, and/or a force of a break tap. 
     
     
         44 . The system of  claim 31 , wherein the machine learning model is trained to classify one or more objects present in the environment of the vehicle. 
     
     
         45 . The system of  claim 31 , wherein the machine learning model is trained to determine a trajectory for controlling the movement of the vehicle. 
     
     
         46 . The system of  claim 31 , wherein the operations further comprise:
 receiving, via a user interface, one or more user inputs selecting one or more attributes for inclusion in the set of attributes; and   generating, based at least on the one or more user inputs, the query to include a search string corresponding to the set of attributes.   
     
     
         47 . The system of  claim 46 , wherein the one or more user inputs further specifies a value or a range of values associated with at least one attribute in the set of attributes. 
     
     
         48 . The system of  claim 46 , wherein the one or more user inputs identifies the one or more attributes from across a selection of vehicle level attributes, environment level attributes, and object level attributes. 
     
     
         49 . A method, comprising:
 determining, using at least one processor, a set of attributes associated with at least one of a vehicle, an environment in which the vehicle is operating, and an object in the environment, wherein at least one attribute among the set of attributes comprises at least one of a feature, a condition, or a characteristic;   executing, using the at least one processor, a query at a scenario database to select one or more scenarios having the set of attributes, the one or more scenarios being selected based at least on an evaluation metric associated with a machine learning model trained to plan a movement of the at least one vehicle while the at least one vehicle encounters the one or more scenarios, wherein at least one scenario among the one or more scenarios comprises vehicle sensor data for a timeframe;   generating, using the at least one processor, training data, wherein the training data comprises the one or more scenarios selected by the executed query as having the determined set of attributes; and   updating, using the at least one processor and based at least on the training data, the machine learning model trained to plan the movement of the at least one vehicle.   
     
     
         50 . The method of  claim 49 , wherein each scenario included in the scenario database includes a combination of attributes associated with the at least one of the vehicle, the environment in which the vehicle is operating, and the object in the environment. 
     
     
         51 . The method of  claim 50 , further comprising:
 extracting, using the at least one processor and from data logged by one or more vehicles, the combination of attributes associated with each scenario included in the scenario database, the data logged by the one or more vehicles including sensor data from the one or more vehicles.   
     
     
         52 . The method of  claim 49 , wherein the evaluation metric associated with the machine learning model includes a frequency at which the machine learning model encounters at least one attribute associated with the one or more scenarios. 
     
     
         53 . The method of  claim 49 , wherein the evaluation metric associated with the machine learning model includes a difficulty level associated with the one or more scenarios, wherein the difficulty level is indicative of the amount of information or computations needed to complete the planned movement output by the machine learning model safely. 
     
     
         54 . The method of  claim 49 , wherein the evaluation metric associated with the machine learning model includes an error in an output of the machine learning model when planning the movement of the at least one vehicle while the at least one vehicle encounters the one or more scenarios, wherein the error indicates a difference in the planned movement and the actual movement of the vehicle, the error comprising an intensity and an error value. 
     
     
         55 . The method of  claim 49 , wherein the machine learning model is trained to classify one or more physical objects present in the environment of the vehicle. 
     
     
         56 . The method of  claim 49 , wherein the machine learning model is trained to determine a trajectory for controlling the movement of the vehicle, wherein a trajectory includes a sequence of actions for controlling the movement of the vehicle. 
     
     
         57 . The method of  claim 49 , further comprising:
 receiving, via a user interface, one or more user inputs selecting one or more attributes for inclusion in the set of attributes; and   generating, using the at least one processor and based at least on the one or more user inputs, the query to include a search string corresponding to the set of attributes.   
     
     
         58 . The method of  claim 57 , wherein the one or more user inputs further specifies a value or a range of values associated with at least one attribute in the set of attributes. 
     
     
         59 . The method of  claim 57 , wherein the one or more user inputs identifies the one or more attributes from across a selection of vehicle level attributes, environment level attributes, and object level attributes. 
     
     
         60 . A non-transitory computer readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:
 determining a set of attributes associated with at least one of a vehicle, an environment in which the vehicle is operating, and an object in the environment, wherein at least one attribute among the set of attributes comprises at least one of a feature, a condition, or a characteristic;   executing a query at a scenario database to select one or more scenarios having the set of attributes, the one or more scenarios being selected based at least on an evaluation metric associated with a machine learning model trained to plan a movement of the at least one vehicle while the at least one vehicle encounters the one or more scenarios, wherein at least one scenario among the one or more scenarios comprises vehicle sensor data for a timeframe;   generating training data, wherein the training data comprises including the one or more scenarios selected by the executed query as having the determined set of attributes; and   updating, based at least on the training data, the machine learning model trained to plan the movement of the at least one vehicle.