Confidence calibration for systems with cascaded predictive models

Abstract

In general, techniques are described that address the limitations of existing conformal prediction methods for cascaded models. In an example, a method includes receiving a first validation data set for validating performance of an upstream model of the two or more cascaded models and receiving a second validation data set for validating performance of a downstream model of the two or more cascaded models wherein the second validation data set is different than the first validation set; estimating system-level errors caused by predictions of the upstream model based on the first validation data set; estimating system-level errors caused by predictions of the downstream model based on the second validation data set; and generating a prediction confidence interval that indicates a confidence for the system based on the system-level errors caused by predictions of the upstream model and based on the system-level errors caused by predictions of the downstream model.

What is claimed is: 
     
         1 . A method for determining confidence for a system having two or more cascaded models, the method comprising:
 receiving a first validation data set for validating performance of an upstream model of the two or more cascaded models and receiving a second validation data set for validating performance of a downstream model of the two or more cascaded models wherein the second validation data set is different than the first validation set;   estimating one or more system-level errors caused by predictions of the upstream model based on the first validation data set;   estimating one or more system-level errors caused by predictions of the downstream model based on the second validation data set; and   generating a confidence interval that indicates a confidence for the system based on the one or more system-level errors caused by predictions of the upstream model and based on the one or more system-level errors caused by predictions of the downstream model.   
     
     
         2 . The method of  claim 1 , further comprising:
 evaluating confidence of the system based on the generated confidence interval; and   adjusting the system to enhance precision of the system.   
     
     
         3 . The method of  claim 1 , wherein the upstream model comprises an object detection model and wherein the downstream model comprises a classification model. 
     
     
         4 . The method of  claim 1 , wherein generating the confidence interval further comprises:
 estimating one or more empirical quantiles at a predefined probability level.   
     
     
         5 . The method of  claim 4 , wherein generating the confidence interval further comprises:
 determining empirical error distribution for the first validation data set and the second validation data set.   
     
     
         6 . The method of  claim 5 , further comprising:
 grouping a plurality of data points in the first validation data set and the second validation data set into two or more clusters based on similarity of intermediate features, wherein the intermediate features comprise intermediate output from the upstream model and input to the downstream model.   
     
     
         7 . The method of  claim 5 , wherein determining empirical error distribution further comprises:
 determining cluster-level error distribution.   
     
     
         8 . The method of  claim 1 , wherein generating the confidence interval further comprises generating the confidence interval using a split conformal prediction technique. 
     
     
         9 . A computing system for determining confidence for a system having two or more cascaded models, the computing system comprising:
 processing circuitry in communication with storage media, the processing circuitry configured to execute a machine learning system configured to:   receive a first validation data set for validating performance of an upstream model of the two or more cascaded models and receive a second validation data set for validating performance of a downstream model of the two or more cascaded models wherein the second validation data set is different than the first validation set;   estimate one or more system-level errors caused by predictions of the upstream model based on the first validation data set;   estimate one or more system-level errors caused by predictions of the downstream model based on the second validation data set; and   generate a confidence interval that indicates a confidence for the system based on the one or more system-level errors caused by predictions of the upstream model and based on the one or more system-level errors caused by predictions of the downstream model.   
     
     
         10 . The system of  claim 9 , wherein the machine learning system is further configured to:
 evaluate confidence of the system based on the generated confidence interval; and   adjust the system to enhance precision of the system.   
     
     
         11 . The system of  claim 9 , wherein the upstream model comprises an object detection model and wherein the downstream model comprises a classification model. 
     
     
         12 . The system of  claim 9 , wherein the machine learning system configured to generate the confidence interval is further configured to:
 estimate one or more empirical quantiles at a predefined probability level.   
     
     
         13 . The system of  claim 12 , wherein the machine learning system configured to generate the confidence interval is further configured to:
 determine empirical error distribution for the first validation data set and the second validation data set.   
     
     
         14 . The system of  claim 13 , wherein the machine learning system is further configured to:
 group a plurality of data points in the first validation data set and the second validation data set into two or more clusters based on similarity of intermediate features, wherein the intermediate features comprise intermediate output from the upstream model and input to the downstream model.   
     
     
         15 . The system of  claim 13 , wherein the machine learning system configured to determine empirical error distribution is further configured to:
 determine cluster-level error distribution.   
     
     
         16 . The system of  claim 9 , wherein the machine learning system configured to generate the confidence interval is further configured to:
 generate the confidence interval using a split conformal prediction technique.   
     
     
         17 . A method for determining confidence for a system having two or more cascaded models, the method comprising:
 generating a confidence interval that indicates a confidence for the system based on one or more system-level errors caused by predictions of an upstream model of the two or more cascaded models and based on the one or more system-level errors caused by predictions of a downstream model of the two or more cascaded models without using end to end system level data;   evaluating and adjusting the system to enhance precision of the system.   
     
     
         18 . The method of  claim 17 , wherein the upstream model comprises an object detection model and wherein the downstream model comprises a classification model. 
     
     
         19 . The method of  claim 17 , wherein generating the confidence interval further comprises:
 estimating one or more empirical quantiles at a predefined probability level.   
     
     
         20 . The method of  claim 19 , wherein generating the confidence interval further comprises:
 determining empirical error distribution for the first validation data set and the second validation data set.   
     
     
         21 . Non-transitory computer-readable storage media having instructions for determining confidence for a system having two or more cascaded models, the instructions configured to cause processing circuitry to:
 receive a first validation data set for validating performance of an upstream model of the two or more cascaded models and receive a second validation data set for validating performance of a downstream model of the two or more cascaded models wherein the second validation data set is different than the first validation set;   estimate one or more system-level errors caused by predictions of the upstream model based on the first validation data set;   estimate one or more system-level errors caused by predictions of the downstream model based on the second validation data set; and   generate a confidence interval that indicates a confidence for the system based on the one or more system-level errors caused by predictions of the upstream model and based on the one or more system-level errors caused by predictions of the downstream model.