Training ensemble models to improve performance in the presence of unreliable base classifiers

Abstract

A system and method for training base classifiers in a boosting algorithm includes optimally training base classifiers considering an unreliability model, and then using a scheme with an aggregator decoder that reverse-flips inputs using inter-classifier redundancy introduced in training.

What is claimed is: 
     
         1 . A method of training a machine learning model, comprising:
 training a plurality of base classifiers of an ensemble model in a sequence of training iterations to minimize an expected loss function, wherein each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers;   determining a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers;   training a plurality of redundant base classifiers of the ensemble model based on the plurality of trained base classifiers;   determining a plurality of redundant base classifier weights for the plurality of redundant base classifiers, wherein each redundant base classifier weight of the plurality of redundant base classifier weights is associated with one redundant base classifier of the plurality of redundant base classifiers; and   updating the ensemble model with the plurality of base classifier weights and the plurality of redundant base classifier weights.   
     
     
         2 . The method of  claim 1 , wherein each redundant base classifier of the plurality of redundant base classifiers is trained based on a unique set of two base classifiers of the plurality of base classifiers to form a triple of classifiers. 
     
     
         3 . The method of  claim 2 , wherein training each respective redundant base classifier of the plurality of redundant base classifiers comprises minimizing 
       
         
           
             
               
                 
                   [ 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       m 
                     
                     
                       
                         v 
                         i 
                         
                           
                             t 
                             + 
                             1 
                           
                           2 
                         
                       
                       ⁢ 
                       
                         
                           J 
                           
                             ( 
                             1 
                             ) 
                           
                         
                         ( 
                         
                           
                             
                               
                                 x 
                                 i 
                               
                               ⁢ 
                               
                                 y 
                                 i 
                               
                             
                             ; 
                             
                               h 
                               t 
                             
                           
                           , 
                           
                             h 
                             
                               t 
                               + 
                               1 
                             
                           
                           , 
                           
                             f 
                             
                               
                                 t 
                                 + 
                                 1 
                               
                               2 
                             
                           
                         
                         ) 
                       
                     
                   
                   ] 
                 
                 s 
               
               + 
               
                 
                   [ 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       m 
                     
                     
                       
                         v 
                         i 
                         
                           
                             t 
                             + 
                             1 
                           
                           2 
                         
                       
                       ⁢ 
                       
                         
                           J 
                           
                             ( 
                             2 
                             ) 
                           
                         
                         ( 
                         
                           
                             
                               
                                 x 
                                 i 
                               
                               ⁢ 
                               
                                 y 
                                 i 
                               
                             
                             ; 
                             
                               h 
                               t 
                             
                           
                           , 
                           
                             h 
                             
                               t 
                               + 
                               1 
                             
                           
                           , 
                           
                             f 
                             
                               
                                 t 
                                 + 
                                 1 
                               
                               2 
                             
                           
                         
                         ) 
                       
                     
                   
                   ] 
                 
                 s 
               
             
           
         
       
       for the respective redundant base classifier, wherein: 
       
         
           
             
               v 
               i 
               
                 
                   t 
                   + 
                   1 
                 
                 2 
               
             
           
         
       
       comprises a risk averaging weight,
 J (1)  comprises a second risk function, 
 J (2)  comprises a second risk function, 
 h t  is a first base classifier, 
 h t+1  is a second base classifier, 
 
       
         
           
             
               f 
               
                 
                   t 
                   + 
                   1 
                 
                 2 
               
             
           
         
       
       is a redundant base classifier trained based on h t  and h t+1 ,
 s is hyperparameter, and 
 {x i , y i } is a data point. 
 
     
     
         4 . The method of  claim 1 , further comprising:
 performing a classification with an aggregator of the ensemble model,   wherein the aggregator is configured to decode input data based on the plurality of redundant base classifiers.   
     
     
         5 . The method of  claim 1 , further comprising receiving data at the trained base classifiers. 
     
     
         6 . The method of  claim 5 , further comprising sending a result of each base classifier to an aggregation node. 
     
     
         7 . The method of  claim 6 , further comprising combining results from each base classifier to obtain a final result from the aggregation node. 
     
     
         8 . A processing system configured to train a machine learning model, comprising:
 a memory comprising computer-executable instructions;   one or more processors configured to execute the computer-executable instructions and cause the processing system to:   train a plurality of base classifiers of an ensemble model in a sequence of training iterations to minimize an expected loss function, wherein each training iteration of the sequence of training iterations is associated with one base classifier of the plurality of base classifiers;   determine a plurality of base classifier weights for the plurality of base classifiers, wherein each base classifier weight of the plurality of base classifier weights is associated with one base classifier of the plurality of base classifiers;   train a plurality of redundant base classifiers of the ensemble model based on the plurality of trained base classifiers;   determine a plurality of redundant base classifier weights for the plurality of redundant base classifiers, wherein each redundant base classifier weight of the plurality of redundant base classifier weights is associated with one redundant base classifier of the plurality of redundant base classifiers; and   update the ensemble model with the plurality of base classifier weights and the plurality of redundant base classifier weights.   
     
     
         9 . The processing system of  claim 8 , wherein each redundant base classifier of the plurality of redundant base classifiers is trained based on a unique set of two base classifiers of the plurality of base classifiers to form a triple of classifiers. 
     
     
         10 . The processing system of  claim 8 , wherein the one or more processors are further configured to cause the processing system to:
 perform a classification with an aggregator of the ensemble model,   wherein the aggregator is configured to decode input data based on the plurality of redundant base classifiers.   
     
     
         11 . The processing system of  claim 8 , wherein the one or more processors are further configured to receive individual results from one or more base classifiers at an aggregator. 
     
     
         12 . The processing system of  claim 11 , wherein the one or more processors are further configured to use a decoder to correct one or more flipped results from base classifiers. 
     
     
         13 . The processing system of  claim 12 , wherein the one or more processors is further configured to combine individual results to obtain a classification. 
     
     
         14 . A method for classifying with a machine learning model, comprising:
 receiving at a decoder of an ensemble model an input vector comprising output from a plurality of base classifiers and a redundant base classifier trained based on the plurality of base classifiers;   decoding the input vector with the decoder to generate input data for an aggregator of the ensemble model;   providing the input data to the aggregator; and   generating, by the aggregator, a classification for the input vector based on the input data.   
     
     
         15 . The method of  claim 14 , further comprising:
 generating a syndrome vector based on the input vector,   wherein the syndrome vector is based on a concurrence of outputs from the plurality of base classifiers and a redundant base classifier.   
     
     
         16 . The method of  claim 14 , further comprising determining a probability of an unreliable base classifier output for a plurality of base classifiers of an ensemble model. 
     
     
         17 . The method of  claim 16 , further comprising training the plurality of base classifiers in a sequence of training iterations. 
     
     
         18 . A processing system configured to classify with a machine learning model, comprising:
 a memory comprising computer-executable instructions;   one or more processors configured to execute the computer-executable instructions and cause the processing system to:
 receive at a decoder of an ensemble model an input vector comprising output from a plurality of base classifiers and a redundant base classifier trained based on the plurality of base classifiers; 
 decode the input vector with the decoder to generate input data for an aggregator of the ensemble model; 
 provide the input data to the aggregator; and 
 generate, by the aggregator, a classification for the input vector based on the input data. 
   
     
     
         19 . The processing system of  claim 18 , wherein the one or more processors are further configured to cause the processing system to:
 generate a syndrome vector based on the input vector,   wherein the syndrome vector is based on a concurrence of outputs from the plurality of base classifiers and a redundant base classifier.   
     
     
         20 . The processing system of  claim 18 , wherein the one or more processors are further configured to update an ensemble model with a plurality of base classifier weights.