Deep neural network processing for sensor blindness detection in autonomous machine applications

Abstract

In various examples, a deep neural network (DNN) is trained for sensor blindness detection using a region and context-based approach. Using sensor data, the DNN may compute locations of blindness or compromised visibility regions as well as associated blindness classifications and/or blindness attributes associated therewith. In addition, the DNN may predict a usability of each instance of the sensor data for performing one or more operations—such as operations associated with semi-autonomous or autonomous driving. The combination of the outputs of the DNN may be used to filter out instances of the sensor data—or to filter out portions of instances of the sensor data determined to be compromised—that may lead to inaccurate or ineffective results for the one or more operations of the system.

What is claimed is: 
     
         1 . A method comprising:
 determining, using one or more neural networks and based at least on first image data corresponding to an image, one or more pixels of the image associated with compromised visibility;   based at least on the one or more pixels being associated with the compromised visibility, filtering out at least one pixel of the one or more pixels to determine second image data corresponding to the image; and   performing, based at least on the second image data, one or more operations associated with a machine.   
     
     
         2 . The method of  claim 1 , further comprising:
 determining a value representative of a usability of the image to perform the one or more operations associated with the machine,   wherein the filtering out of the at last one pixel of the one or more pixels is further based at least on the value.   
     
     
         3 . The method of  claim 1 , further comprising:
 determining one or more classifications associated with the one or more pixels,   wherein the filtering out of the at last one pixel of the one or more pixels is further based at least on the one or more classifications.   
     
     
         4 . The method of  claim 1 , further comprising:
 determining one or more importance values associated with the one or more pixels,   wherein the filtering out of the at last one pixel of the one or more pixels is further based at least on the one or more importance values.   
     
     
         5 . The method of  claim 4 , further comprising:
 determining, using the one or more neural networks, a saliency map associated with the image, the saliency map indicating an importance of one or more regions of the image,   wherein the determining the one or more importance values associated with the one or more pixels comprises determining, based at least on the saliency map, that the one or more pixels are associated with a region of the one or more regions having an associated importance above a threshold.   
     
     
         6 . The method of  claim 1 , further comprising:
 determining, based at least on the one or more pixels, a percentage of pixels of the image that are associated with the compromised visibility; and   determining that the percentage of pixels is less than a threshold percentage,   wherein the filtering out of the at last one pixel of the one or more pixels is further based at least on the percentage of pixels being less than the threshold percentage.   
     
     
         7 . The method of  claim 1 , wherein the determining the one or more pixels of the image associated with the compromised visibility comprises:
 determining, using the one or more neural networks and based at least on the first image data corresponding to the image, one or more confidence values indicating whether the one or more pixels are associated with the compromised visibility; and   determining, based at least on the one or more confidence values, that the one or more pixels are associated with the compromised visibility.   
     
     
         8 . The method of  claim 1 , wherein the performing the one or more operations associated with the machine comprises:
 processing the second image data using one or more systems associated with the machine;   determining, using the one or more systems and based at least on the second image data, the one or more operations associated with the machine; and   performing the one or more operations associated with the machine.   
     
     
         9 . A method comprising:
 one or more processing units to:
 determine, using one or more neural networks and based at least on first image data corresponding to an image, one or more pixels of the image associated with compromised visibility; 
 based at least on the one or more pixels being associated with the compromised visibility, generate, by filtering out at least one pixel of the one or more pixels, second image data corresponding to the image; and 
 perform, based at least on the second image data, one or more operations associated with a machine. 
   
     
     
         10 . The system of  claim 9 , wherein the one or more processing units are further to:
 determine a value representative of a usability of the image to perform the one or more operations associated with the machine,   wherein the second image data is generated based at least on the value.   
     
     
         11 . The system of  claim 9 , wherein the one or more processing units are further to:
 determine one or more classifications associated with the one or more pixels,   wherein the generation of the second image data is further based at least on the one or more classifications.   
     
     
         12 . The system of  claim 9 , wherein the one or more processing units are further to:
 determine one or more importance values associated with the one or more pixels,   wherein the second image data is generated based at least on the one or more importance values.   
     
     
         13 . The system of  claim 12 , wherein the one or more processing units are further to:
 determine, using the one or more neural networks, a saliency map associated with the image, the saliency map indicating one or more first regions of the image as more important than one or more second regions of the image,   wherein the one or more importance values associated with the one or more pixels is determined based at least on determining, based at least on the saliency map, that the one or more pixels are associated with a second region of the one or more second regions.   
     
     
         14 . The system of  claim 9 , wherein the one or more processing units are further to:
 determine, based at least on the one or more pixels, a percentage of pixels of the image that are associated with the compromised visibility; and   determine that the percentage of pixels is less than a threshold percentage,   wherein the second image data is generated based at least on the percentage of pixels being less than the threshold percentage.   
     
     
         15 . The system of  claim 9 , wherein the one or more pixels of the image associated with the compromised visibility are determined, at least, by:
 determining, using the one or more neural networks and based at least on the first image data corresponding to the image, one or more confidence values indicating whether the one or more pixels are associated with the compromised visibility; and   determining, based at least on the one or more confidence values, that the one or more pixels are associated with the compromised visibility.   
     
     
         16 . The system of  claim 9 , wherein the one or more operations associated with the machine is performed, at least, by:
 processing the second image data using one or more systems associated with the machine;   determining, using the one or more systems and based at least on the second image data, the one or more operations associated with the machine; and   performing the one or more operations associated with the machine.   
     
     
         17 . The system of  claim 9 , wherein the system is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing real-time streaming;   a system for generating or presenting virtual reality (VR) content;   a system for generating or presenting augmented reality (AR) content;   a system for performing deep learning operations;   a system implemented using a robot;   a system for performing conversational AI operations;   a system for generating synthetic data;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         18 . A processor comprising:
 one or more processing units to perform one or more operations associated with a machine based at least on filtered image data corresponding to an image, the filtered image data being generated by filtering out one or more pixels of the image that are associated with compromised visibility.   
     
     
         19 . The processor of  claim 18 , wherein the one or more processing units are further to determine, using one or more neural networks and based at least on initial image data corresponding to the image, the one or more pixels that are associated with the compromised visibility, wherein the initial image data corresponds to the filtered image data prior to the filtering. 
     
     
         20 . The processor of  claim 18 , wherein the processor is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing real-time streaming;   a system for generating or presenting virtual reality (VR) content;   a system for generating or presenting augmented reality (AR) content;   a system for performing deep learning operations;   a system implemented using a robot;   a system for performing conversational AI operations;   a system for generating synthetic data;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.