Automotive sensor fusion of radar, lidar, camera systems with improved safety by use of machine learning

Abstract

The present invention provides a vehicle safety sensor system comprising a processor; a first radar sensor configuration to scan a first wide area; a first auxiliary sensor configuration to scan a second wide area; a second radar sensor configuration to scan a first narrow area, wherein the first narrow area is smaller than the first wide area; and a second auxiliary sensor configuration to scan a second narrow area, wherein the second narrow area is smaller than the second wide area. The system is configured to operate in a first mode wherein data from the first radar sensor configuration and data from the first auxiliary sensor configuration is fused together to provide a first set of fused data; and the processor is configured to detect objects from the first set of fused data. The system is configured to operate in a second mode when an object has been detected wherein data from the second radar sensor configuration and data from second auxiliary sensor configuration is fused together to provide a second set of fused data; and the processor is configured to monitor the detected objects based on the second set of fused data.

1 . A vehicle safety sensor system comprising:
 a processor;   a first radar sensor configuration to scan a first wide area;   a first auxiliary sensor configuration to scan a second wide area;   a second radar sensor configuration to scan a first narrow area, wherein the first narrow area is smaller than the first wide area; and   a second auxiliary sensor configuration to scan a second narrow area, wherein the second narrow area is smaller than the second wide area, wherein:   the system is configured to operate in a first mode wherein:
 data from the first radar sensor configuration and data from the first auxiliary sensor configuration is fused together to provide a first set of fused data; and 
 the processor is configured to detect objects from the first set of fused data; and 
   the system is configured to operate in a second mode when an object has been detected in the first mode wherein:
 data from the second radar sensor configuration and data from second auxiliary sensor configuration is fused together to provide a second set of fused data; and 
 the processor is configured to monitor the detected objects based on the second set of fused data; 
   wherein the processor is further configured to determine a risk of collision with the detected object based on the second set of fused data, wherein determining the risk of collision comprises selecting a classification for the object from a plurality of classifications.   
     
     
         2 . The system of  claim 1 , wherein:
 the first auxiliary sensor configuration comprises an array of a plurality of optical sensors and the optical sensors detect visible spectrum electromagnetic radiation; and   the second auxiliary sensor configuration comprises an array of a plurality of optical sensors and the optical sensors detect visible spectrum electromagnetic radiation.   
     
     
         3 . The system of  claim 1  comprising a LIDAR sensor. 
     
     
         4 . The system of  claim 1, 2, or 3  wherein the system is coupled to a global satellite navigation system antenna (GNSS). 
     
     
         5 . The system of  claim 4 , wherein at least one sensor is coupled to a GNSS antenna and data provided by the at least one sensor comprises a timestamp derived from a received GNSS signal. 
     
     
         6 . The system of  claim 4 , wherein the processor is coupled to a GNSS antenna and positional data in a GNSS signal received by the GNSS antenna is used to control at least one sensor. 
     
     
         7 . The system of  any preceding claim , wherein the processor is coupled to one or more environmental sensors, wherein environmental data provided by the one or more environmental sensors is used to determine a confidence weighting for at least one sensor, wherein the confidence weighting is indicative of the sensor's accuracy in a detected environmental condition. 
     
     
         8 . The system of  claim 7 , wherein the environmental sensor is one or more of:
 a light sensor or a precipitation sensor, and preferably wherein the processor is further coupled to one or more of: a compass, a wheel odometer, or a gyroscope.   
     
     
         9 . The system of  any preceding claim , wherein:
 the second auxiliary sensor configuration comprises at least one sensor operating at a higher resolution than the sensors of the first auxiliary sensor configuration; and   the second auxiliary sensor configuration is configured to process a narrow area containing the detected object, wherein the narrow area scanned by the second auxiliary sensor configuration is smaller than the area scanned by the first auxiliary sensor configuration.   
     
     
         10 . The system of  any preceding claim , wherein the first radar configuration is a radar operating in a first mode and the second radar configuration is the radar operating in a second mode. 
     
     
         11 . The system of  any preceding claim , wherein the first auxiliary sensor configuration is an array of optical sensors operating in a first mode and the second auxiliary sensor configuration is the array of optical sensors operating in a second mode. 
     
     
         12 . The system of  claim 1 , wherein selecting a classification comprises extracting the classification from a look up table using one or more of: a calculated size, shape, or colour of the object. 
     
     
         13 . The system of  claim 12 , wherein selecting a classification comprises using a neural network. 
     
     
         14 . The system of  claim 12 or 13 , wherein selecting a classification comprises using a random decision forest. 
     
     
         15 . A method for improving safety of a vehicle comprising:
 in a first mode:
 scanning a first wide area with a first radar sensor configuration; 
 scanning a second wide area with a first auxiliary sensor configuration; 
 fusing data from the first radar sensor configuration with the data from the first auxiliary sensor configuration to provide a first set of fused data; 
 determining if an object is present based on the first set of data; and 
 if it is determined than an object is present in the first mode, switching to a second mode; and 
   in the second mode:
 processing a first narrow area with a second radar sensor configuration, wherein the first narrow area is smaller than the first wide area; 
 processing a second narrow area with a second auxiliary sensor configuration, wherein the second narrow area is smaller than the second wide area; 
 fusing the data from the second radar sensor configuration and the data from second auxiliary sensor configuration to provide a second set of fused data; and 
 monitoring the detected object from the second set of fused data; wherein monitoring comprises: 
   determining a risk classification, where the risk classification is indicative of the risk of collision with the detected object.   
     
     
         16 . The method of  claim 15 , wherein monitoring further comprises:
 if the risk classification meets a predetermined criterion, switching to a third mode wherein a vehicle safety sensor system controls the vehicle to avoid the detected object.   
     
     
         17 . A computer readable storage medium comprising instructions, which when executed by a processor coupled to a radar and an auxiliary sensor configuration, causes the processor to perform a method according to  claim 15 or 16 .