Iteratively improving an advertisement response model

Abstract

There are provided systems and methods for iteratively improving an advertisement response model. A payment provider may perform operations that include training an advertisement response model using a training data set. The operations include determining that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold. The operations further include identifying, based on executing the advertisement response model using a target data set that is different from the training data set, one or more units from the target data set for which to run the advertising campaign. The operations also include receiving one or more responses corresponding to a run of the advertising campaign with respect to the identified one or more units from the target data set and updating the training data set based on the one or more responses. The operations further include training an advertisement response model using resulting training data and repeating the operations as long as the accuracy value of the resulting model stays below the threshold or until the increase in the accuracy value with each iteration becomes unprofitable with respect to the costs of acquiring responses from further units from the target dataset.

1 . (canceled) 
     
     
         2 . A system, comprising:
 at least one processor; and   at least one memory storing computer-executable instructions, that in response to execution by the at least one processor, causes the system to perform operations comprising
 executing an advertisement response model using a first training data set, wherein the first training data set indicates, for each unit included in the training data set, a respective response to an advertising campaign; 
 determining, based on the executing the advertisement response model using the first training data set, that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold; 
 calculating, based on executing the advertisement response model using a target data set that is different from the first training data set, a respective certainty score corresponding to each unit of the target data set; 
 determining, based on the respective certainty score corresponding to each unit of the target data set, a subset of the target data set for which to run the advertising campaign; 
 receiving one or more responses corresponding to a run of the advertising campaign with respect to the subset of the target data set, and generating a second training data set based on the one or more responses; and 
 determining, based on executing the advertisement response model using the second training data set, whether a second accuracy value corresponding to the advertisement response model is greater than the accuracy value threshold. 
   
     
     
         3 . The system of  claim 2 , wherein the operations further comprise:
 determining that the second accuracy value is greater than or equal to the accuracy value threshold;   adjusting the target data set by removing the units included in the subset from the target data set; and   based on executing the advertisement response model with respect to the adjusted target data set, determining one or more units of the adjusted target data set that have a positive predicted response to the advertising campaign.   
     
     
         4 . The system of  claim 2 , wherein the operations further comprise removing the subset of the target data set from the target data set. 
     
     
         5 . The system of  claim 2 , wherein determining the subset of the target data set comprises:
 ranking each unit of the target data set according to the respective certainty score for each unit of the target data set, wherein the subset includes a predetermined number of units from the target data set that are each associated with a lower respective certainty score than respective certainty scores of remaining units in the target data set that are not included in the sub set.   
     
     
         6 . The system of  claim 2 , wherein each unit of the subset of the target data set is associated with a certainty score that is less than a certainty score threshold. 
     
     
         7 . The system of  claim 2 , wherein the operations further comprise generating the first training data set by transmitting advertisements corresponding to the advertising campaign to respective devices of a random sample of a particular population. 
     
     
         8 . The system of  claim 2 , wherein the subset of the target data corresponds to one or more characteristics selected from a group comprising website characteristics, geographic location characteristics, and demographic characteristics. 
     
     
         9 . The system of  claim 2 , wherein the second accuracy value is greater than the first accuracy value. 
     
     
         10 . A method comprising:
 executing an advertisement response model using a first training data set, wherein the first training data set indicates, for each unit included in the training data set, a respective response to an advertising campaign;   determining, based on the executing the advertisement response model using the first training data set, that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold;   calculating, based on executing the advertisement response model using a target data set that is different from the first training data set, a respective certainty score corresponding to each unit of the target data set;   determining, based on the respective certainty score corresponding to each unit of the target data set, a subset of the target data set for which to run the advertising campaign;   receiving one or more responses corresponding to a run of the advertising campaign with respect to the subset of the target data set, and generating a second training data set based on the one or more responses; and   determining, based on executing the advertisement response model using the second training data set, whether a second accuracy value corresponding to the advertisement response model is greater than the accuracy value threshold.   
     
     
         11 . The method of  claim 10 , further comprising:
 determining that the second accuracy value is greater than or equal to the accuracy value threshold;   adjusting the target data set by removing the units included in the subset from the target data set; and   based on executing the advertisement response model with respect to the adjusted target data set, determining one or more units of the adjusted target data set that have a positive predicted response to the advertising campaign.   
     
     
         12 . The method of  claim 10 , further comprising:
 removing the subset of the target data set from the target data set.   
     
     
         13 . The method of  claim 10 , wherein determining the subset of the target data set comprises:
 ranking each unit of the target data set according to the respective certainty score for each unit of the target data set, wherein the subset includes a predetermined number of units from the target data set that are each associated with a lower respective certainty score than respective certainty scores of remaining units in the target data set that are not included in the subset.   
     
     
         14 . The method of  claim 10 , wherein each unit of the subset of the target data set is associated with a certainty score that is less than a certainty score threshold. 
     
     
         15 . The method of  claim 10 , further comprising:
 generating the first training data set by transmitting advertisements corresponding to the advertising campaign to respective devices of a random sample of a particular population.   
     
     
         16 . The method of  claim 10 , wherein the subset of the target data corresponds to one or more characteristics selected from a group comprising website characteristics, geographic location characteristics, and demographic characteristics. 
     
     
         17 . The method of  claim 10 , wherein the second accuracy value is greater than the first accuracy value. 
     
     
         18 . A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
 executing an advertisement response model using a first training data set, wherein the first training data set indicates, for each unit included in the training data set, a respective response to an advertising campaign;   determining, based on the executing the advertisement response model using the first training data set, that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold;   calculating, based on executing the advertisement response model using a target data set that is different from the first training data set, a respective certainty score corresponding to each unit of the target data set;   determining, based on the respective certainty score corresponding to each unit of the target data set, a subset of the target data set for which to run the advertising campaign;   receiving one or more responses corresponding to a run of the advertising campaign with respect to the subset of the target data set, and generating a second training data set based on the one or more responses; and   determining, based on executing the advertisement response model using the second training data set, whether a second accuracy value corresponding to the advertisement response model is greater than the accuracy value threshold.   
     
     
         19 . The non-transitory machine-readable medium of  claim 18 , further comprising:
 determining that the second accuracy value is greater than or equal to the accuracy value threshold;   adjusting the target data set by removing the units included in the subset from the target data set; and   based on executing the advertisement response model with respect to the adjusted target data set, determining one or more units of the adjusted target data set that have a positive predicted response to the advertising campaign.   
     
     
         20 . The non-transitory machine-readable medium of  claim 18 , further comprising:
 removing the subset of the target data set from the target data set.   
     
     
         21 . The non-transitory machine-readable medium of  claim 18 , wherein the subset of the target data corresponds to one or more characteristics selected from a group comprising website characteristics, geographic location characteristics, and demographic characteristics.