Quantifying the Predictive Uncertainty of Neural Networks Via Residual Estimation With I/O Kernel

Abstract

A residual estimation with an I/O kernel (“RIO”) framework provides estimates of predictive uncertainty of neural networks, and reduces their point-prediction errors. The process captures neural network (“NN”) behavior by estimating their residuals with an I/O kernel using a modified Gaussian process (“GP”). RIO is applicable to real-world problems, and, by using a sparse GP approximation, scales well to large datasets. RIO can be applied directly to any pretrained NNs without modifications to model architecture or training pipeline.

1 . A process for correcting one or more predictions of a regression model comprising:
 training a regression model to make one or more predictions using an input training dataset (x i , y i );   storing output dataset (x i , ŷ i ) of the trained regression model;   training a Gaussian process using (x i , y i ) and (x 1 , ŷ i ) to estimate residuals of the trained regression model when applied to real-world input data x RW  to make one or more predictions y RW .   
     
     
         2 . The process according to  claim 1 , further comprising:
 applying the regression model to x RW  to generate one or more predicted outcomes (x RW ,y RW ); and   applying the trained GP to the regression model one or more predicted outcomes y RW  to generate calibrated predicted outcomes y RW ′.   
     
     
         3 . The process according to  claim 1 , wherein the training of the Gaussian process (GP) includes:
 calculating residuals 
         r   =             r   i     =     y   i     −       y   ^     i             i   =   1     n     ,         
 wherein r denotes the vector of all residuals and ŷ denotes the vector of all regression model predictions; 
   calculating an n × n covariance matrix at all pairs of training points based on a composite kernel K c ((X, ŷ), (X, ŷ)), where each entry is given by k c ((x i , ŷ i ), (x j , ŷ j )) = k in (x i , x j ) + k out (ŷ i ,ŷ j ), for i, j = 1,2, ..., n; and   optimizing GP hyperparameters 
           σ     in     2     ,     l     in       ,     σ     out     2     ,         l     out       ,       and         σ   n   2           
 by maximizing log marginal likelihood 
             log   p         r     X   ,     y   ^         =           −     1   2       r             K   c             X   ,     y   ^         ,       X   ,     y   ^             +     σ   n   2     I         −   1           r   −     1   2     log         K   c             X   ,     y   ^         ,       X   ,     y   ^             +             
 . 
   
     
     
         4 . The process according to  claim 2 , wherein applying the trained Gaussian process (GP) to predictions y RW  of the regression model includes:
 calculating residual mean; 
 calculating residual variance; and 
 returning a distribution of calibrated prediction y RW ′. 
 
     
     
         5 . The process according to  claim 4 , further comprises 
 calculating residual mean in accordance with
               r   ^     ¯     ∗     =     k   ∗               K   c             X   ,     y   ^         ,       X   ,     y   ^             +     σ   n   2     I           −   1       r   ;         
   calculating residual variance in accordance with var(r̂ * ) = k c ((x * , ŷ * ), (x * , ŷ * )) -
           k   ∗               K   c             X   ,     y   ^         ,       X   ,     y   ^             +     σ   n   2     I           −   1         k   ∗     ;         
 and 
   returning a distribution of calibrated prediction 
               y   ′     ^     ∗           
 in accordance with 
               y   ′     ^     ∗     ∼   N           y   ^     ∗     +             
 . 
   
     
     
         6 . The process according to  claim 1 , wherein the regression model is selected from the group consisting of a neural network (NN) and a random forest. 
     
     
         7 . A computer-readable medium storing instructions that, when executed by a computer, perform a process for correcting one or more predictions of a regression model comprising:
 training a regression model to make one or more predictions using an input training dataset (x i , y i );   storing output dataset (x i , ŷ i ) of the trained regression model;   training a Gaussian process using (x i , y i ) and (x i , ŷ i ) to estimate residuals of the trained regression model when applied to real-world input data x RW  to make one or more predictions y RW .   
     
     
         8 . The computer-readable medium storing instructions of  claim 7 , further comprising:
 applying the regression model to x RW  to generate one or more predicted outcomes (x RW ,y RW ); and   applying the trained GP to the regression model one or more predicted outcomes y RW  to generate calibrated predicted outcomes Y RW ′.   
     
     
         9 . The computer-readable medium storing instructions of  claim 7 , wherein the training of the Gaussian process (GP) includes:
 calculating residuals 
         r   =             r   i     =     y   i     −       y   ^     i             i   =   1     n     ,         
 wherein r denotes the vector of all residuals and ŷ denotes the vector of all regression model predictions; 
   calculating an n × n covariance matrix at all pairs of training points based on a composite kernel K c ((X, ŷ), (X, ŷ)), where each entry is given by k c ((x i , ŷ i ), (x j , ŷ j )) = k in (x i , x j ) + k out (ŷ i , ŷ j ), for i, j = 1,2, ..., n; and   optimizing GP hyperparameters 
           σ     in     2     ,     l     in       ,     σ     out     2     ,         l     out       ,       and         σ   n   2           
 by maximizing log marginal likelihood 
             log   p         r     X   ,     y   ^         =           −     1   2       r   ⊤             K   c             X   ,     y   ^         ,       X   ,     y   ^             +     σ   n   2     I         −   1           r   −     1   2     log         K   c             X   ,     y   ^         ,       X   ,     y   ^             +             
 . 
   
     
     
         10 . The computer-readable medium storing instructions of  claim 8 , wherein applying the trained Gaussian process (GP) to predictions y RW  of the regression model includes:
 calculating residual mean; 
 calculating residual variance; and 
 returning a distribution of calibrated prediction y RW ′. 
 
     
     
         11 . The computer-readable medium storing instructions of  claim 10 , further comprises 
 calculating residual mean in accordance with
               r   ^     ¯     ∗     =     k   ∗   ⊤               K   c             X   ,     y   ^         ,       X   ,     y   ^             +     σ   n   2     I           −   1       r   ;         
   calculating residual variance in accordance with var(r̂ * ) = k c ((x * , ŷ * ), (x * , ŷ * )) -
           k   ∗               K   c             X   ,     y   ^         ,       X   ,     y   ^             +     σ   n   2     I           −   1         k   ∗     ;         
 and 
   returning a distribution of calibrated prediction 
               y   ′     ^     ∗           
 in accordance with 
               y   ′     ^     ∗     ∼   N           y   ^     ∗     +             
 .