Adaptive Online Experimentation

Abstract

An adaptive online experimentation method and system is disclosed for more effectively and efficiently determining and conducting information gathering and evaluations associated with computer-based applications. The adaptive online experimentation process integrates decision analysis, value of information analysis, design of experiment models, and the inferencing of gathered information, including experimental results. The experimental results may include behaviors of users of a computer-based system. The process enables an automatic, adaptive, closed-loop process for attaining additional information and assimilating the attained information into the decision model.

1 . A computer-based experimentation method comprising: 
 selecting a first online experiment, wherein the selection of the first online experiment is based, at least in part, on the expected net value of information of the experiment;    performing automatically the first online experiment;    analyzing automatically user behaviors resulting from the first online experiment; and    selecting automatically a second online experiment based, at least in part, on the user behaviors resulting from the first online experiment.    
   
   
       2 . The method of  claim 1 , wherein performing automatically the first online experiment comprises: 
 generating a recommendation.    
   
   
       3 . The method of  claim 1 , wherein performing automatically the first online experiment comprises: 
 performing a computer-based search.    
   
   
       4 . The method of  claim 1 , wherein performing automatically the first online experiment comprises: 
 performing a computer-based information retrieval.    
   
   
       5 . The method of  claim 1 , wherein performing automatically the first online experiment comprises: 
 modifying a computer-based structural element.    
   
   
       6 . The method of  claim 1 , wherein selecting automatically a second online experiment based, at least in part, on the user behaviors resulting from the first online experiment comprises: 
 applying information about the intrinsic characteristics of items of content.    
   
   
       7 . The method of  claim 1 , wherein selecting automatically a second online experiment based, at least in part, on the user behaviors resulting from the first online experiment comprises: 
 applying a statistical learning algorithm.    
   
   
       8 . The method of  claim 1 , wherein selecting automatically a second online experiment based, at least in part, on the user behaviors resulting from the first online experiment comprises: 
 applying an experimental design algorithm.    
   
   
       9 . The method of  claim 1 , wherein selecting automatically a second online experiment based, at least in part, on the user behaviors resulting from the first online experiment comprises: 
 generating automatically an expected net value of information for the second online experiment.    
   
   
       10 . An adaptive experimentation system comprising: 
 a value of information function comprising a means to determine an expected value of information associated with a potential online experiment;    an information gathering function comprising a means to generate a first online experiment;    a function to analyze user behaviors resulting from the first online experiment; and    an experimental design and inferencing function, wherein the experimental design and inferencing function automatically selects a second online experiment to perform based, at least in part, on user behaviors resulting from the first online experiment and the expected value of information of the second online experiment.    
   
   
       11 . The system of  claim 10 , wherein an information gathering function comprising a means to generate a first online experiment comprises: 
 a recommendation generating function.    
   
   
       12 . The system of  claim 10 , wherein an information gathering function comprising a means to generate a first online experiment comprises: 
 a function to modify a computer-based structural element.    
   
   
       13 . The system of  claim 10 , wherein a function to analyze user behaviors resulting from the first online experiment comprises: 
 a statistical learning function.    
   
   
       14 . The system of  claim 10 , wherein an experimental design and inferencing function, wherein the experimental design and inferencing function automatically selects a second online experiment to perform based, at least in part, on user behaviors resulting from the first online experiment and the expected value of information of the second online experiment comprises: 
 a statistical learning function.    
   
   
       15 . The system of  claim 10 , wherein an experimental design and inferencing function, wherein the experimental design and inferencing function automatically selects a second online experiment to perform based, at least in part, on user behaviors resulting from the first online experiment and the expected value of information of the second online experiment comprises: 
 an experimental design function.    
   
   
       16 . The system of  claim 10 , wherein an experimental design and inferencing function, wherein the experimental design and inferencing function automatically selects a second online experiment to perform based, at least in part, on user behaviors resulting from the first online experiment and the expected value of information of the second online experiment comprises: 
 a function that applies an intrinsic characteristic of an item of content.    
   
   
       17 . An adaptive experimental infrastructure decision system comprising: 
 a function to automatically simulate an experimental infrastructure;    a function to evaluate the expected value of information generated by an implementation of the said experimental infrastructure, based, at least in part on a simulation of the experimental infrastructure; and    a function to generate an expected value of the said experimental infrastructure based on the said expected value of information and the expected cost of the said experimental infrastructure.    
   
   
       18 . The system of  claim 17 , wherein a function to automatically simulate an experimental infrastructure comprises: 
 an experimental design function.