Methods and systems for programmatic control of transmitted electronic content

Abstract

Aspects of the invention provide methods, systems, apparatuses and computer program products for implementing a dynamic experimentation service. An example of a method for implementing a dynamic experimentation service includes receiving a set of experiment configuration parameters, determining, by a processor and based on the experiment configuration parameters, an experiment model, the experiment model comprising electronic data indicating an audience selected from a plurality of consumers and a plurality of levels for controlling a communication service in operation to generate an electronic marketing communication for transmission to at least one of the plurality of consumers, receiving a request, from the communication service, the request comprising a communication identifier, determining, based on the communication identifier, at least one of the plurality of levels, and causing the electronic marketing communication to be generated as a treatment comprising the at least one of the plurality of levels.

1 .- 51 . (canceled) 
     
     
         52 . A method for implementing a dynamic experimentation service, the method comprising:
 receiving, via an experiment configuration interface, programmatic definition of a set of experiment configuration parameters;   implementing, by a communication service interface component, an application programming interface, for providing a mechanism by which an experiment management component is provided the set of experiment configuration parameters;   determining, by the experiment management component via a processor and based on the experiment configuration parameters, an experiment model,   wherein the experiment model comprises electronic data indicating a type of experiment, including one of an A/A test, an A/B test, or a factorial test, an audience selected from a plurality of consumers, and a set of email cadence factors,   wherein each set of email cadence factors comprises a plurality of levels, each level indicative of a setting for controlling a communication service in operation to generate an electronic marketing communication for transmission to at least one of the plurality of consumers, wherein each of the plurality of levels is indicative of a time period between instances of electronic marketing communication;   determining the communication services from a plurality of communication services that support the factors and levels defined in the experiment configuration parameters;   receiving a request, from the communication service, the request comprising a communication identifier, the communication identifier being a unique identifier indicative of the electronic marketing communication under construction;   determining, by applying the communication identifier to the experimental model, at least one of the plurality of levels; and   causing the electronic marketing communication to be generated as a treatment in accordance with the at least one of the plurality of levels by transmitting the at least one of the plurality of levels to the communication service in response to receiving the request.   
     
     
         53 . The method of  claim 52 , wherein the communication identifier identifies a particular consumer recipient of the electronic marketing communication. 
     
     
         54 . The method of  claim 52 , further comprising:
 receiving clickstream data comprising a plurality of clickstream events;   determining that at least one of the clickstream events is associated with the electronic marketing communication; and   determining a performance characteristic of the at least one level used to generate the electronic marketing communication based at least in part on the at least one clickstream event.   
     
     
         55 . The method of  claim 54 , further comprising:
 determining that a second of the clickstream events is associated with a different electronic marketing communication; and   determining a second performance characteristic of a different level associated with the different electronic marketing communication.   
     
     
         56 . The method of  claim 54 , wherein the at least one of the clickstream events is determined to be associated with the electronic marketing communication based at least in part on the at least one clickstream event being associated with a consumer to which the electronic marketing communication was transmitted. 
     
     
         57 . The method of  claim 54 , wherein the at least one of the clickstream events is determined to be associated with the electronic marketing communication based at least in part on the at least one of the clickstream events occurring within a threshold time period of when the electronic marketing communication was transmitted. 
     
     
         58 . The method of  claim 52 , further comprising providing an application programming interface by which the request is received from the communication service. 
     
     
         59 . A non-transitory computer readable storage medium comprising instructions that, when executed by a processor, cause the processor to:
 receive, via an experiment configuration interface, programmatic definition of a set of experiment configuration parameters;   implement, by a communication service interface component, an application programming interface, for providing a mechanism by which an experiment management component is provided the set of experiment configuration parameters;   determine, by the experiment management component via the processor, based on the experiment configuration parameters, an experiment model,   wherein the experiment model comprise electronic data indicating a type of experiment, including one of an A/A test, an A/B test, or a factorial test, an audience selected from a plurality of consumers, and a set of email cadence factors,   wherein each set of email cadence factors comprises a plurality of levels, each level indicative of a setting for controlling a communication service in operation to generate an electronic marketing communication for transmission to at least one of the plurality of consumers, wherein each of the plurality of levels is indicative of a time period between instances of electronic marketing communication;   determine the communication services from a plurality of communication services that support the factors and levels defined in the experiment configuration parameters;   receive a request, from the communication service, the request comprising a communication identifier, the communication identifier being a unique identifier indicative of the electronic marketing communication under construction;   determine, by applying the communication identifier to the experimental model, at least one of the plurality of levels; and   cause the electronic marketing communication to be generated as a treatment in accordance with the at least one of the plurality of levels by transmitting the at least one of the plurality of levels to the communication service in response to receiving the request.   
     
     
         60 . The computer readable storage medium of  claim 59 , wherein the communication identifier identifies a particular consumer recipient of the electronic marketing communication. 
     
     
         61 . The computer readable storage medium of  claim 59 , wherein the instructions further cause the processor to:
 receive clickstream data comprising a plurality of clickstream events;   determine that at least one of the clickstream events is associated with the electronic marketing communication; and   determine a performance characteristic of the at least one level used to generate the electronic marketing communication based at least in part on the at least one clickstream event.   
     
     
         62 . The computer readable storage medium of  claim 61 , wherein the instructions further cause the processor to:
 determine that a second of the clickstream events is associated with a different electronic marketing communication; and   determine a second performance characteristic of a different level associated with the different electronic marketing communication.   
     
     
         63 . The computer readable storage medium of  claim 61 , wherein the at least one of the clickstream events is determined to be associated with the electronic marketing communication based at least in part on the at least one clickstream event being associated with a consumer to which the electronic marketing communication was transmitted. 
     
     
         64 . The computer readable storage medium of  claim 61 , wherein the at least one of the clickstream events is determined to be associated with the electronic marketing communication based at least in part on the at least one of the clickstream events occurring within a threshold time period of when the electronic marketing communication was transmitted. 
     
     
         65 . The computer readable storage medium of  claim 59 , wherein the instructions further cause the processor to provide an application programming interface by which the request is received from the communication service. 
     
     
         66 . An apparatus for implementing a dynamic experimentation service, the apparatus comprising:
 experiment configuration circuitry configured to:
 receive, via an experiment configuration interface, programmatic definition of a set of experiment configuration parameters; and 
 implement, by a communication service interface component, an application programming interface, for providing a mechanism by which an experiment management component is provided the set of experiment configuration parameters; and 
 the experiment management circuitry configured to: 
 determine, by the experiment management component via a processor, based on the experiment configuration parameters, 
 wherein an experiment model comprises electronic data indicating a type of experiment, including one of an A/A test, an AB test, or a factorial test, an audience selected from a plurality of consumers and a set of email cadence factors, 
 wherein each set of email cadence factors comprises a plurality of levels, each level indicative of a setting for controlling a communication service in operation to generate an electronic marketing communication for transmission to at least one of the plurality of consumers, wherein each of the plurality of levels is indicative of a time period between instances of electronic marketing communication; 
 determine the communication services from a plurality of communication services that support the factors and levels defined in the experiment configuration parameters; 
 receive a request from the communication service, the request comprising a communication identifier, the communication identifier being a unique identifier indicative of the electronic marketing communication under construction; 
 determine, by applying the communication identifier to the experimental model, at least one of the plurality of levels; and 
 cause the electronic marketing communication to be generated as a treatment in accordance with the at least one of the plurality of levels by transmitting the at least one of the plurality of levels to the communication service in response to receiving the request. 
   
     
     
         67 . The apparatus of  claim 66 , wherein the communication identifier identifies a particular consumer recipient of the electronic marketing communication. 
     
     
         68 . The apparatus of  claim 66 , further comprising:
 result analysis circuitry configured to:
 receive clickstream data comprising a plurality of clickstream events; 
 determine that at least one of the clickstream events is associated with the electronic marketing communication; and 
 determine a performance characteristic of the at least one level used to generate the electronic marketing communication based at least in part on the at least one clickstream event. 
   
     
     
         69 . The apparatus of  claim 68 , wherein the result analysis circuitry is further configured to:
 determine that a second of the clickstream events is associated with a different electronic marketing communication; and   determine a second performance characteristic of a different level associated with the different electronic marketing communication.   
     
     
         70 . The apparatus of  claim 68 , wherein the at least one of the clickstream events is determined to be associated with the electronic marketing communication based at least in part on the at least one clickstream event being associated with a consumer to which the electronic marketing communication was transmitted. 
     
     
         71 . The apparatus of  claim 68 , wherein the at least one of the clickstream events is determined to be associated with the electronic marketing communication based at least in part on the at least one of the clickstream events occurring within a threshold time period of when the electronic marketing communication was transmitted. 
     
     
         72 . The apparatus of  claim 66 , wherein the experiment management circuitry is further configured to provide an application programming interface by which the request is received from the communication service.