Methods and systems for programmatic control of transmitted electronic content

Abstract

Aspects of the invention provide methods, systems, apparatuses and computer program products for implementing a dynamic experimentation service. An example of a method for implementing a dynamic experimentation service includes receiving a set of experiment configuration parameters, determining, by a processor and based on the experiment configuration parameters, an experiment model, the experiment model comprising electronic data indicating an audience selected from a plurality of consumers and a plurality of levels for controlling a communication service in operation to generate an electronic marketing communication for transmission to at least one of the plurality of consumers, receiving a request, from the communication service, the request comprising a communication identifier, determining, based on the communication identifier, at least one of the plurality of levels, and causing the electronic marketing communication to be generated as a treatment comprising the at least one of the plurality of levels.

1 .- 24 . (canceled) 
     
     
         25 . An apparatus for performing a dynamic experimentation service, the apparatus comprising:
 experiment management circuitry configured to:
 initialize one or more experiments comprising one or more treatments including factors and levels, the factors and levels used to generate one or more electronic communications; 
 transmit one or more electronic communications using the one or more treatments; 
 receive one or more preliminary results associated with the one or more electronic communications; 
 map the one or more preliminary results to the one or more treatments; and 
 alter the one or more experiments while the one or more experiments are running based on the one or more preliminary results in advance of further transmission of the one or more electronic communications, wherein the experiment management circuitry comprises a script to programmatically alter running experiments or define new experiments based on at least one of the one or more treatments associated with a selected one of the one or more preliminary results performing at least a threshold amount worse than one or more other treatments and remove the at least one of the one or more treatments from being used to generate the one or more electronic communications while the one or more experiments are still ongoing. 
   
     
     
         26 . The apparatus of  claim 25 , wherein the experiment management circuitry is further configured to:
 assess impact of the one or more treatments by attributing one or more clickstream events to the one or more treatments.   
     
     
         27 . The apparatus of  claim 26 , wherein the one or more clickstream events comprise one or more identifiers associated with the one or more electronic communications that caused the clickstream event. 
     
     
         28 . The apparatus of  claim 25 , wherein the script programmatically examines the levels of the one or more treatments and performs further experimentation of one or more performance characteristics associated with top performing ones of the levels. 
     
     
         29 . The apparatus of  claim 25 , wherein the experiment management circuitry is further configured to:
 determine a first audience mapping, the first audience mapping defining a first relationship between each of a plurality of consumers and the one or more experiments to which each is apportioned.   
     
     
         30 . The apparatus of  claim 29 , wherein the experiment management circuitry is further configured to:
 reapportion the plurality of consumers according to a second relationship between each of the plurality of consumers and the one or more experiments, wherein the second relationship differs from the first relationship.   
     
     
         31 . The apparatus of  claim 30 , wherein the experiment management circuitry is further configured to:
 store log data indicative of a mapping between the first relationship and the second relationship, wherein the mapping identifies movement of the plurality of consumers across a plurality of audience groups;   provide access to the one or more experiments, allowing utilization of the log data to map each of the plurality of consumers from one particular associated audience group to each of the one or more treatments; and   perform one or more additional experiments utilizing the second audience mapping while the one or more experiments are being performed and the results are still being received.   
     
     
         32 . A method for performing a dynamic experimentation service, the method comprising:
 initializing, using a processor, one or more experiments comprising one or more treatments including factors and levels, the factors and levels used to generate one or more electronic communications;   transmitting, using the processor, one or more electronic communications using the one or more treatments;   receiving, using the processor, one or more preliminary results associated with the one or more electronic communications;   mapping, using the processor, the one or more preliminary results to the one or more treatments; and   altering, using the processor, the one or more experiments while the one or more experiments are running based on the one or more preliminary results in advance of further transmission of the one or more electronic communications, wherein the experiment management circuitry comprises a script to programmatically alter running experiments or define new experiments based on at least one of the one or more treatments associated with a selected one of the one or more preliminary results performing at least a threshold amount worse than one or more other treatments and remove the at least one of the one or more treatments from being used to generate the one or more electronic communications while the one or more experiments are still ongoing.   
     
     
         33 . The method of  claim 32 , further comprising:
 assessing impact of the one or more treatments by attributing one or more clickstream events to the one or more treatments.   
     
     
         34 . The method of  claim 33 , wherein the one or more clickstream events comprise one or more identifiers associated with the one or more electronic communications that caused the clickstream event. 
     
     
         35 . The method of  claim 32 , wherein the script programmatically examines the levels of the one or more treatments and performs further experimentation of one or more performance characteristics associated with top performing ones of the levels. 
     
     
         36 . The method of  claim 32 , further comprising:
 determining a first audience mapping, the first audience mapping defining a first relationship between each of a plurality of consumers and the one or more experiments to which each is apportioned.   
     
     
         37 . The method of  claim 36 , further comprising:
 reapportion the plurality of consumers according to a second relationship between each of the plurality of consumers and the one or more experiments, wherein the second relationship differs from the first relationship.   
     
     
         38 . The method of  claim 37 , further comprising:
 storing log data indicative of a mapping between the first relationship and the second relationship, wherein the mapping identifies movement of the plurality of consumers across a plurality of audience groups;   providing access to the one or more experiments, allowing utilization of the log data to map each of the plurality of consumers from one particular associated audience group to each of the one or more treatments; and   performing one or more additional experiments utilizing the second audience mapping while the one or more experiments are being performed and the results are still being received.   
     
     
         39 . A non-transitory computer readable storage medium comprising instructions that, when executed by a processor, cause the processor to:
 initialize one or more experiments comprising one or more treatments including factors and levels, the factors and levels used to generate one or more electronic communications;   transmit one or more electronic communications using the one or more treatments;   receive one or more preliminary results associated with the one or more electronic communications;   map the one or more preliminary results to the one or more treatments; and   alter the one or more experiments while the one or more experiments are running based on the one or more preliminary results in advance of further transmission of the one or more electronic communications, wherein the experiment management circuitry comprises a script to programmatically alter running experiments or define new experiments based on at least one of the one or more treatments associated with a selected one of the one or more preliminary results performing at least a threshold amount worse than one or more other treatments and remove the at least one of the one or more treatments from being used to generate the one or more electronic communications while the one or more experiments are still ongoing.   
     
     
         40 . The non-transitory computer readable storage medium of  claim 39 , wherein the instructions further cause the processor to:
 assess impact of the one or more treatments by attributing one or more clickstream events to the one or more treatments.   
     
     
         41 . The non-transitory computer readable storage medium of  claim 40 , wherein the one or more clickstream events comprise one or more identifiers associated with the one or more electronic communications that caused the clickstream event. 
     
     
         42 . The non-transitory computer readable storage medium of  claim 39 , wherein the script programmatically examines the levels of the one or more treatments and performs further experimentation of one or more performance characteristics associated with top performing ones of the levels. 
     
     
         43 . The non-transitory computer readable storage medium of  claim 39 , wherein the instructions further cause the processor to:
 determine a first audience mapping, the first audience mapping defining a first relationship between each of a plurality of consumers and the one or more experiments to which each is apportioned.   
     
     
         44 . The non-transitory computer readable storage medium of  claim 43 , wherein the instructions further cause the processor to:
 reapportion the plurality of consumers according to a second relationship between each of the plurality of consumers and the one or more experiments, wherein the second relationship differs from the first relationship.