Super resolution system trained based on obfuscated low-resolution data

Abstract

A super resolution system that increases a resolution of image data captured by one or more cameras include one or more controllers including one or more super resolution neural networks that include at least one of an obfuscated image data model and a focused loss model. The one or more super resolution neural networks receive paired training data during a training phase, where the paired training data is representative of the image data captured by the one or more cameras representing a surrounding environment and includes low-resolution image data and high-resolution image data. The obfuscated low-resolution image data and the high-resolution image data both represent identical images, and the obfuscated low-resolution image data includes an object of interest located in the surrounding environment that is obfuscated based on an obfuscation technique.

What is claimed is: 
     
         1 . A super resolution system that increases a resolution of image data captured by one or more cameras, the super resolution system comprising: 
 one or more controllers including one or more super resolution neural networks that include an obfuscated image data model, wherein the one or more controllers include one or more processors that execute instructions to: 
 receive, by the obfuscated image data model, paired training data during a training phase, wherein the paired training data is representative of the image data captured by the one or more cameras representing a surrounding environment and includes obfuscated low-resolution image data and high-resolution image data, and wherein the obfuscated low-resolution image data and the high-resolution image data both represent identical images, and the obfuscated low-resolution image data includes an object of interest located in the surrounding environment that is obfuscated based on an obfuscation technique; 
 increase, by the obfuscated image data model, a resolution of the obfuscated low-resolution image data to create a reconstructed high-resolution image; 
 calculate, by the obfuscated image data model, a total loss associated with the reconstructed high-resolution image, wherein the high-resolution image data of the paired training data acts as ground truth data and the obfuscated image data model is trained based on an iterative process to minimize the total loss;  
 receive, by the obfuscated image data model, real-life low-resolution image data a testing phase; and  
 increase, by the obfuscated image data model, the resolution of the real-life low-resolution image data to create real-life high-resolution image data. 
   
     
     
         2 . The super resolution system of  claim 1 , wherein the total loss associated with the reconstructed high-resolution image is a sum of a mean squared error loss, a perceptual loss, an adversarial loss, and a total variance loss. 
     
     
         3 . The super resolution system of  claim 1 , wherein the one or more super resolution neural networks includes a focused loss model. 
     
     
         4 . The super resolution system of  claim 3 , wherein the one or more controllers execute instructions to: 
 receive, by the focused loss model, the paired training data during the training phase;   increase, by the focused loss model, a resolution of the obfuscated low-resolution image data to create a focused reconstructed high-resolution image;    calculate, by the focused loss model, a focused loss associated with the focused reconstructed high-resolution image, wherein the high-resolution image data of the paired training data acts as ground truth data and the focused loss is a sum of a focused mean squared error loss, a focused perceptual loss, and a focused total variance loss;   receive, by the focused loss model, real-life low-resolution image data a testing phase; and    increase, by the focused loss model, the resolution of the real-life low-resolution image data to create real-life high-resolution image data.   
     
     
         5 . The super resolution system of  claim 4 , wherein the one or more controllers execute instructions to:  
       determine a bounding box defining a bounded area within an image frame of the obfuscated low-resolution image data, wherein the bounding box contains the object of interest. 
     
     
         6 . The super resolution system of  claim 5 , wherein the focused loss associated with the focused reconstructed high-resolution image assigns a higher value to a bounded weighting factor corresponding to the bounded area of the image frame when compared to a whole weighting factor corresponding to the entirety of the image frame. 
     
     
         7 . The super resolution system of  claim 6 , wherein the one or more controllers determine the focused mean squared error loss by: 
 determining a mean squared error loss associated with the bounded area of the image frame; and    determining a mean squared error loss associated with the entirety of the image frame, wherein the focused mean squared error loss is the sum of a weighted mean squared error loss associated with the bounded area within the image frame and a weighted mean squared error loss associated with the entirety of the image frame.   
     
     
         8 . The super resolution system of  claim 6 , wherein the one or more controllers determine the focused perceptual loss by: 
 determining a focused perceptual loss associated with the bounded area of the image frame; and    determining a focused perceptual loss associated with the entirety of the image frame, wherein the focused perceptual loss is the sum of a weighted perceptual loss associated with the bounded area within the image frame and a weighted perceptual loss associated with the entirety of the image frame.    
     
     
         9 . The super resolution system of  claim 6 , wherein the one or more controllers determine the focused total variance loss by: 
 determining a focused total variance loss associated with the bounded area of the image frame; and    determining a focused total variance loss associated with the entirety of the image frame, wherein the focused total variance loss is the sum of a weighted focused total variance loss associated with the bounded area within the image frame and a weighted focused total variance loss associated with the entirety of the image frame.   
     
     
         10 . The super resolution system of  claim 6 , wherein the focused mean squared error loss, the focused perceptual loss, and the focused total variance loss each include different values for the bounded weighting factor and whole weighting factor. 
     
     
         11 . The super resolution system of  claim 1 , wherein the object of interest is one of the following: a traffic sign, a pedestrian, a bicyclist, an animal, a street sign, a billboard, a commercial sign, a surrounding vehicle, and an infrastructure asset. 
     
     
         12 . The super resolution system of  claim 1 , wherein the obfuscated low-resolution image data includes a resolution that is less than or equal to 480 x 640 pixels, and the high-resolution image data includes a resolution that is greater than 480 x 640 pixels.  
     
     
         13 . The super resolution system of  claim 1 , wherein the obfuscation technique includes one of the following: deleting a portion of object of interest, randomly removing pixels that represent the object of interest, blurring the object of interest, and darkening image data associated with the object of interest.  
     
     
         14 . A super resolution system that increases a resolution of image data captured by one or more cameras, the super resolution system comprising: 
 one or more controllers including one or more super resolution neural networks that include a focused loss model, wherein the one or more controllers include one or more processors that execute instructions to: 
 receive, by the focused loss model, paired training data during a training phase, wherein the paired training data is representative of the image data captured by the one or more cameras representing a surrounding environment and includes obfuscated low-resolution image data and high-resolution image data, and wherein the obfuscated low-resolution image data and the high-resolution image data both represent identical images, and the obfuscated low-resolution image data includes an object of interest located in the surrounding environment that is obfuscated based on an obfuscation technique; 
 increase, by the focused loss model, a resolution of the obfuscated low-resolution image data to create a focused reconstructed high-resolution image; 
 calculate, by the focused loss model, a focused loss associated with the focused reconstructed high-resolution image, wherein the high-resolution image data of the paired training data acts as ground truth data and the focused loss is a sum of a focused mean squared error loss, a focused perceptual loss, and a focused total variance loss;  
 receive, by the obfuscated image data model, real-life low-resolution image data a testing phase; and  
 increase, by the obfuscated image data model, the resolution of the real-life low-resolution image data to create real-life high-resolution image data. 
   
     
     
         15 . The super resolution system of  claim 14 , wherein the one or more controllers execute instructions to: 
 determine a bounding box defining a bounded area within an image frame of the obfuscated low-resolution image data, wherein the bounding box contains the object of interest.   
     
     
         16 . The super resolution system of  claim 15 , wherein the focused loss associated with the focused reconstructed high-resolution image assigns a higher value to a bounded weighting factor corresponding to the bounded area of the image frame when compared to a whole weighting factor corresponding to the entirety of the image frame. 
     
     
         17 . The super resolution system of  claim 16 , wherein the one or more controllers determine the focused mean squared error loss by: 
 determining a mean squared error loss associated with the bounded area of the image frame; and    determining a mean squared error loss associated with the entirety of the image frame, wherein the focused mean squared error loss is the sum of a weighted mean squared error loss associated with the bounded area within the image frame and a weighted mean squared error loss associated with the entirety of the image frame.   
     
     
         18 . The super resolution system of  claim 16 , wherein the one or more controllers determine the focused perceptual loss by: 
 determining a focused perceptual loss associated with the bounded area of the image frame; and    determining a focused perceptual loss associated with the entirety of the image frame, wherein the focused perceptual loss is the sum of a weighted perceptual loss associated with the bounded area within the image frame and a weighted perceptual loss associated with the entirety of the image frame.   
     
     
         19 . The super resolution system of  claim 16 , wherein the one or more controllers determine the focused total variance loss by: 
 determining a focused total variance loss associated with the bounded area of the image frame; and    determining a focused total variance loss associated with the entirety of the image frame, wherein the focused total variance loss is the sum of a weighted focused total variance loss associated with the bounded area within the image frame and a weighted focused total variance loss associated with the entirety of the image frame.   
     
     
         20 . The super resolution system of  claim 16 , wherein the focused mean squared error loss, the focused perceptual loss, and the focused total variance loss each include different values for the bounded weighting factor and whole weighting factor.