Automated optimization of an advertisement placement engine

Abstract

A system and method for predicting a performance of a target advertisement placement via a simulation for optimum tuning is disclosed. A simulator receives a set of queries from a production engine, selects a subset of simulation queries from the queries by filtering, modifies query parameters as needed, transmits the simulation queries to the target engine for simulation, collects search results from the target engine, and computes a summary metric, which includes data fields retrieved from the search results and quantities calculated by using a prediction model derived from a production data warehouse. The simulator may further produce a simulation report, an indicator of performance prediction for the target engine, which comprises multiple metrics for the target engine, calculated with varying engine parameters.

What is claimed is: 
     
         1 . A method comprising:
 determining a plurality of queries from a production engine running in a production environment, wherein the production engine generates production data based at least on the plurality of queries;   determining one or more queries from the plurality of queries, wherein the one or more queries correspond to a particular product;   running a simulator engine in a simulated production environment with the one or more queries, wherein the simulated production engine generates a list of simulated results;   determining target metrics for the simulated production engine based at least on the production data and the list of simulated results, wherein the target metrics comprise a target relevance metric and a target value metric, wherein the target relevance metric indicates a degree of accuracy of the list of simulated results corresponding to the particular product, and wherein the target value metric indicates a value associated with the list of simulated results; and   tuning control parameters of the simulator engine based on the degree of accuracy of the list of simulated results corresponding to the particular product and the value associated with each result from the list of simulated results.   
     
     
         2 . The method of  claim 1 , further comprising generating a simulation report comprising the target metrics determined for the simulator engine, wherein the simulation report is displayed on a user interface coupled with the target engine. 
     
     
         3 . The method of  claim 1 , wherein the simulator engine caused a dynamic modification to the one or more queries to further correspond to the particular product, wherein the list of simulated results is further based on the dynamic modification. 
     
     
         4 . The method of  claim 1 , further comprising determining a list of production results from the production engine corresponding to the one or more queries, and wherein determining the target relevance metric is further based on respective positions of results from the list of production results compared with respective positions of results from the list of simulated results. 
     
     
         5 . The method of  claim 4 , wherein the target relevance metric further indicates respective target positions of results from the list of simulated results based on the compared positions of results from the list of production results and the list of simulated results, wherein tuning the control parameters of the simulator engine is further based on the respective target positions for results from the list of simulated results. 
     
     
         6 . The method of  claim 1 , further comprising determining a list of production results from the production engine corresponding to the one or more queries, wherein determining the target value metric is further based on respective values of results from the list of production results compared with respective values of results from the list of simulated results. 
     
     
         7 . The method of  claim 6 , wherein the target value metric further indicates respective target values of results from the list of simulated results based on the compared values of results from the list of production results and the list of simulated results, wherein tuning the control parameters of the simulator engine is further based on the respective target values of results from the list of simulated results. 
     
     
         8 . A system, comprising:
 a non-transitory memory storing instructions; and   one or more processors couples to the non-transitory memory and configured to read the instructions from the non-transitory memory to cause the system to perform operations comprising:
 determining a plurality of queries from a production engine running in a production environment, wherein the production engine generates production data based at least on the plurality of queries; 
 determining one or more queries from the plurality of queries, wherein the one or more queries correspond to a particular product; 
 running a simulator engine in a simulated production environment with the one or more queries, wherein the simulated production engine generates a list of simulated results; 
 determining target metrics for the simulated production engine based at least on the production data and the list of simulated results, wherein the target metrics comprise a target relevance metric and a target value metric, wherein the target relevance metric indicates a degree of accuracy of the list of simulated results corresponding to the particular product, and wherein the target value metric indicates a value associated with the list of simulated results; and 
 tuning control parameters of the simulator engine based on the degree of accuracy of the list of simulated results corresponding to the particular product and the value associated with each result from the list of simulated results. 
   
     
     
         9 . The system of  claim 8 , wherein the operations further comprise generating a simulation report comprising the target metrics determined for the simulator engine, wherein the simulation report is displayed on a user interface coupled with the target engine. 
     
     
         10 . The system of  claim 8 , wherein the simulator engine causes a dynamic modification to the one or more queries to further correspond to the particular product, and wherein the list of simulated results is further based on the dynamic modification. 
     
     
         11 . The system of  claim 8 , wherein the operations further comprise determining a list of production results from the production engine corresponding to the one or more queries, and wherein determining the target relevance metric is further based on respective positions of results from the list of production results compared with respective positions of results from the list of simulated results. 
     
     
         12 . The system of  claim 8 , wherein the target relevance metric further indicates respective target positions of results from the list of simulated results based on the compared positions of results from the list of production results and the list of simulated results, wherein tuning the control parameters of the simulator engine is further based on the respective target positions for results from the list of simulated results. 
     
     
         13 . The system of  claim 8 , wherein the operations further comprise determining a list of production results from the production engine corresponding to the one or more queries, wherein determining the target value metric is further based on respective values of results from the list of production results compared with respective values of results from the list of simulated results. 
     
     
         14 . The system of  claim 13 , wherein the target value metric further indicates respective target values of results from the list of simulated results based on the compared values of results from the list of production results and the list of simulated results, wherein tuning the control parameters of the simulator engine is further based on the respective target values of results from the list of simulated results. 
     
     
         15 . A non-transitory machine readable medium having stored thereon machine readable instructions executable to cause a machine to perform operations comprising:
 determining a plurality of queries from a production engine running in a production environment, wherein the production engine generates production data based at least on the plurality of queries;   determining one or more queries from the plurality of queries, wherein the one or more queries correspond to a particular product;   running a simulator engine in a simulated production environment with the one or more queries, wherein the simulated production engine generates a list of simulated results;   determining target metrics for the simulated production engine based at least on the production data and the list of simulated results, wherein the target metrics comprise a target relevance metric and a target value metric, wherein the target relevance metric indicates a degree of accuracy of the list of simulated results corresponding to the particular product, and wherein the target value metric indicates a value associated with the list of simulated results; and   tuning control parameters of the simulator engine based on the degree of accuracy of the list of simulated results corresponding to the particular product and the value associated with each result from the list of simulated results.   
     
     
         16 . The non-transitory machine readable medium of  claim 15 , wherein the operations further comprise generating a simulation report comprising the target metrics determined for the simulator engine, wherein the simulation report is displayed on a user interface coupled with the target engine. 
     
     
         17 . The non-transitory machine readable medium of  claim 15 , wherein the simulator engine causes a dynamic modification to the one or more queries to further correspond to the particular product, and wherein the list of simulated results is further based on the dynamic modification. 
     
     
         18 . The non-transitory machine readable medium of  claim 15 , wherein the operations further comprise determining a list of production results from the production engine corresponding to the one or more queries, and wherein determining the target relevance metric is further based on respective positions of results from the list of production results compared with respective positions of results from the list of simulated results. 
     
     
         19 . The non-transitory machine readable medium of  claim 15 , wherein the target relevance metric further indicates respective target positions of results from the list of simulated results based on the compared positions of results from the list of production results and the list of simulated results, wherein tuning the control parameters of the simulator engine is further based on the respective target positions for results from the list of simulated results. 
     
     
         20 . The non-transitory machine readable medium of  claim 15 , wherein the operations further comprise determining a list of production results from the production engine corresponding to the one or more queries, wherein determining the target value metric is further based on respective values of results from the list of production results compared with respective values of results from the list of simulated results.