Automatic detection of data for annotation for autonomous vehicle perception

Abstract

Various technologies described herein pertain to detecting sensor data to be annotated for autonomous vehicle perception algorithm training. A label identifying a type of an object is assigned to an object at a particular location in an environment based on sensor data generated by a sensor system of an autonomous vehicle for a given time. The label is assigned based on a confidence score assigned to the type of the object by a computer-implemented perception algorithm. The computer-implemented perception algorithm assigns the confidence score to the type of the object based on the sensor data corresponding to the particular location in the environment for the given time. An output of a heuristic is generated based on the label and/or the confidence score, and the output of the heuristic is used to control whether to cause the sensor data for the given time to be annotated.

What is claimed is: 
     
         1 . An autonomous vehicle, comprising:
 a sensor system that generates sensor data;   a computing system that is in communication with the sensor system, wherein the computing system comprises:
 a processor; and 
 memory that stores computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising:
 assigning a first label to an object at a particular location in an environment based on the sensor data generated for a first time by the sensor system, wherein the first label identifies a first type of the object, wherein the first label is assigned based on a first confidence score assigned to the first type of the object from amongst a predefined set of types by a computer-implemented perception algorithm, and wherein the computer-implemented perception algorithm assigns the first confidence score to the first type of the object based on the sensor data corresponding to the particular location in the environment for the first time; 
 assigning a second label to the object at the particular location in the environment based on the sensor data generated for a second time by the sensor system, wherein the second label identifies a second type of the object, wherein the second label is assigned based on a second confidence score assigned to the second type of the object from amongst the predefined set of types by the computer-implemented perception algorithm, and wherein the computer-implemented perception algorithm assigns the second confidence score to the second type of the object based on the sensor data corresponding to the particular location in the environment for the second time; 
 generating an output of a heuristic based on at least one of:
 a comparison of the first label assigned based on the sensor data for the first time and the second label assigned based on the sensor data for the second time; or 
 a comparison of the first confidence score assigned based on the sensor data for the first time and the second confidence score assigned based on the sensor data for the second time; and 
 
 based on the output of the heuristic, one of:
 storing the sensor data for at least one of the first time or the second time for annotation, wherein the annotation of the sensor data for at least one of the first time or the second time is desirably used to retrain the computer-implemented perception algorithm; or 
 discarding the sensor data for the first time and the second time. 
 
 
   
     
     
         2 . The autonomous vehicle of  claim 1 , wherein the memory further stores computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising:
 selectively flagging the sensor data for at least one of the first time or the second time for annotation based on the output of the heuristic.   
     
     
         3 . The autonomous vehicle of  claim 1 , wherein the memory further stores computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising:
 selectively transmitting the sensor data for at least one of the first time or the second time to a server computing system based on the output of the heuristic.   
     
     
         4 . The autonomous vehicle of  claim 1 , wherein the first time and the second time are within a threshold time duration. 
     
     
         5 . The autonomous vehicle of  claim 1 , wherein the output of the heuristic is generated based on the comparison of the first label assigned based on the sensor data for the first time and the second label assigned based on the sensor data for the second time. 
     
     
         6 . The autonomous vehicle of  claim 1 , wherein the output of the heuristic is generated based on the comparison of the first confidence score assigned based on the sensor data for the first time and the second confidence score assigned based on the sensor data for the second time. 
     
     
         7 . The autonomous vehicle of  claim 6 , wherein the first type of the object and the second type of the object are the same. 
     
     
         8 . The autonomous vehicle of  claim 1 , wherein the sensor system a lidar sensor system. 
     
     
         9 . The autonomous vehicle of  claim 1 , wherein the sensor system is a camera sensor system. 
     
     
         10 . The autonomous vehicle of  claim 1 , wherein the sensor system is a radar sensor system. 
     
     
         11 . The autonomous vehicle of  claim 1 , further comprising:
 a mechanical system;   wherein the memory further stores computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising:
 controlling operation of the mechanical system of the autonomous vehicle to effectuate desired motion of the autonomous vehicle based on the object at the particular location in the environment detected using the computer-implemented perception algorithm. 
   
     
     
         12 . A method of detecting data for annotation for autonomous vehicle perception training, comprising
 assigning a first label to an object at a particular location in an environment based on sensor data generated for a first time by a sensor system of the autonomous vehicle, wherein the first label identifies a first type of the object, wherein the first label is assigned based on a first confidence score assigned to the first type of the object from amongst a predefined set of types by a computer-implemented perception algorithm, and wherein the computer-implemented perception algorithm assigns the first confidence score to the first type of the object based on the sensor data corresponding to the particular location in the environment for the first time;   assigning a second label to the object at the particular location in the environment based on the sensor data generated for a second time by the sensor system of the autonomous vehicle, wherein the second label identifies a second type of the object, wherein the second label is assigned based on a second confidence score assigned to the second type of the object from amongst the predefined set of types by the computer-implemented perception algorithm, and wherein the computer-implemented perception algorithm assigns the second confidence score to the second type of the object based on the sensor data corresponding to the particular location in the environment for the second time;   generating an output of a heuristic based on at least one of:
 a comparison of the first label assigned based on the sensor data for the first time and the second label assigned based on the sensor data for the second time; or 
 a comparison of the first confidence score assigned based on the sensor data for the first time and the second confidence score assigned based on the sensor data for the second time; and 
   based on the output of the heuristic, one of:
 storing the sensor data for at least one of the first time or the second time for annotation, wherein the annotation of the sensor data for at least one of the first time or the second time is desirably used to retrain the computer-implemented perception algorithm; or 
 discarding the sensor data for the first time and the second time. 
   
     
     
         13 . The method of  claim 12 , further comprising:
 selectively transmitting the sensor data for at least one of the first time or the second time to a server computing system based on the output of the heuristic.   
     
     
         14 . The method of  claim 12 , wherein the first time and the second time are within a threshold time duration. 
     
     
         15 . The method of  claim 12 , wherein the output of the heuristic is generated based on the comparison of the first label assigned based on the sensor data for the first time and the second label assigned based on the sensor data for the second time. 
     
     
         16 . The method of  claim 12 , wherein the output of the heuristic is generated based on the comparison of the first confidence score assigned based on the sensor data for the first time and the second confidence score assigned based on the sensor data for the second time. 
     
     
         17 . The method of  claim 16 , wherein the first type of the object and the second type of the object are the same. 
     
     
         18 . The method of  claim 12 , further comprising:
 ranking the sensor data for at least one of the first time or the second time stored for annotation relative to disparate sensor data stored for annotation when the sensor data is to be annotated.   
     
     
         19 . A computing system, comprising:
 a processor; and   memory that stores computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising:
 assigning a first label to an object at a particular location in an environment based on sensor data generated for a first time by a sensor system of an autonomous vehicle, wherein the first label identifies a first type of the object, wherein the first label is assigned based on a first confidence score assigned to the first type of the object from amongst a predefined set of types by a computer-implemented perception algorithm, and wherein the computer-implemented perception algorithm assigns the first confidence score to the first type of the object based on the sensor data corresponding to the particular location in the environment for the first time; 
 assigning a second label to the object at the particular location in the environment based on the sensor data generated for a second time by the sensor system of the autonomous vehicle, wherein the second label identifies a second type of the object, wherein the second label is assigned based on a second confidence score assigned to the second type of the object from amongst the predefined set of types by the computer-implemented perception algorithm, and wherein the computer-implemented perception algorithm assigns the second confidence score to the second type of the object based on the sensor data corresponding to the particular location in the environment for the second time; 
 generating an output of a heuristic based on at least one of:
 a comparison of the first label assigned based on the sensor data for the first time and the second label assigned based on the sensor data for the second time; or 
 a comparison of the first confidence score assigned based on the sensor data for the first time and the second confidence score assigned based on the sensor data for the second time; and 
 
 based on the output of the heuristic, one of:
 storing the sensor data for at least one of the first time or the second time for annotation, wherein the annotation of the sensor data for at least one of the first time or the second time is desirably used to retrain the computer-implemented perception algorithm; or 
 discarding the sensor data for the first time and the second time 
 
   
     
     
         20 . The computing system of  claim 19 , wherein the autonomous vehicle comprises the computing system.