Online Advertisement Effectiveness Measurements

Abstract

Methods, systems, and apparatus, including computer program products, in which evaluation versions of advertisements are generated and served in different contexts. Based on received evaluation results, engagement rates, brand linkage and advertisement effectiveness measures are generated for each of the different contexts and for an overall context.

1 . A computer-implemented method for evaluating advertisements, comprising:
 generating an evaluation version of an advertisement, the evaluation version of the advertisement including an evaluation tool for collecting evaluation results;   serving the evaluation version of the advertisement for presentation in each of a plurality of contexts;   for each serving of the evaluation version of the advertisement:
 determining the context in which the evaluation version of the advertisement is presented; 
 receiving advertisement evaluation results for an evaluation conducted using the evaluation tool; and 
 associating the advertisement evaluation results with the context in which the evaluation version of the advertisement is presented; and 
   generating engagement rates for the evaluation version of the advertisement and the advertisement, each engagement rate being associated with at least one corresponding context of the plurality of contexts, each engagement rate being a measure of a rate of participation in evaluating the advertisement for the corresponding context using the evaluation tool, and each engagement rate being determined independent of a performance rate of the advertisement.   
     
     
         2 . The method of  claim 1 , wherein:
 the advertisement is brand obfuscated to facilitate measuring advertisement recognition and brand linkage, and the evaluation tool collects evaluation results from which an advertisement recognition value and brand linkage value for the advertisement in each context can be determined; and   the method further comprises generating the advertisement recognition values and the brand linkage values for the advertisement from the evaluation results and associating the advertisement recognition values and the brand linkage values with the contexts.   
     
     
         3 . The method of  claim 2 , further comprising determining an advertisement effectiveness measure for each of the contexts, each advertisement effectiveness measure based on the advertisement recognition value, the brand linkage value, and the engagement rate associated with the context. 
     
     
         4 . The method of  claim 3 , wherein each advertisement effectiveness measure is a product of the advertisement recognition value, the brand linkage value, and the engagement rate associated with the context. 
     
     
         5 . The method of  claim 3 , wherein:
 the evaluation tool further collects evaluation results from which an advertisement likeability value for the advertisement in each context can be determined;   the method further comprises generating the advertisement likeability values for the advertisement from the evaluation results and associating the advertisement likeability values with the contexts; and   each advertisement effectiveness measure for a context is a product of the advertisement recognition value, the brand linkage value, the advertisement likeability value and the engagement rate associated with the context.   
     
     
         6 . The method of  claim 1 , wherein the plurality of contexts are defined by corresponding publisher content of publishers in which the evaluation version of the advertisement is presented. 
     
     
         7 . The method of  claim 1 , wherein the plurality of contexts are defined by a corresponding plurality of advertisement content types, each advertisement content type determinative of one or more presentation features of the advertisement. 
     
     
         8 . The method of  claim 7 , wherein the one or more presentation features include a visual presentation feature or an audio presentation feature. 
     
     
         9 . The method of  claim 1 , wherein:
 the evaluation tool comprises hierarchal evaluation questions that are presented in response to receiving a response to a first hierarchal evaluation question or a previous hierarchal evaluation question, and wherein the evaluation results include data indicating a quantity of hierarchal evaluation questions presented; and   generating engagement rates for the evaluation version of the advertisement comprises weighing each engagement rate proportional to the quantity of hierarchal evaluation questions presented by the evaluation tool in each of the contexts.   
     
     
         10 . The method of  claim 1 , wherein:
 the advertisement is an advertisement in an advertisement campaign that defines advertising budgets, each advertising budget specifying a budget for a corresponding context; and   the method further comprises adjusting the advertising budgets for each corresponding context based on the engagement rate associated with the corresponding context.   
     
     
         11 . The method of  claim 1 , wherein the plurality of contexts comprise:
 a search context in which the evaluation version of the advertisement is presented in response to identification of a search query for which the evaluation version of the advertisement is determined to be relevant; and   a content context in which the evaluation version of the advertisement is presented in response to identification of publisher content for which the evaluation version of the advertisement is determined to be relevant.   
     
     
         12 . The method of  claim 1 , wherein the evaluation tool is an in situ evaluation. 
     
     
         13 . A computer-implemented method for measuring advertising effectiveness, comprising:
 generating evaluation versions of advertisements in an advertising campaign, each evaluation version including an evaluation tool for collecting evaluation results;   serving the evaluation versions of the advertisements for presentation in each of a plurality of contexts;   for each serving of an evaluation version of an advertisement, determining the context in which the evaluation version of the advertisement is presented;   receiving advertisement evaluation results for an evaluation conducted using the evaluation tool; and   generating engagement rates, each engagement rate being associated with an advertisement in the advertisement campaign and the corresponding evaluation version of the advertisement, and further associated with a corresponding context of the plurality of contexts, each engagement rate being a measure of a rate of participation in evaluating advertisement in the corresponding context, and each engagement rate being determined independently of performance rates of the advertisements.   
     
     
         14 . The computer-implemented method of  claim 13 , wherein the evaluation versions of the advertisements comprise brand obfuscated evaluation versions of the advertisements and the evaluation tool collects evaluation results from which an advertisement recognition value and brand linkage value for the advertisement in each context can be determined; and
 the method further comprises generating the advertisement recognition values and the brand linkage values for the advertisement from the evaluation results and associating the advertisement recognition values and the brand linkage values with the contexts.   
     
     
         15 . The method of  claim 14 , further comprising determining an advertisement effectiveness measure for each of the advertisements in each of the plurality of contexts, each advertisement effectiveness measure being based on the advertisement recognition value, the brand linkage value, and the corresponding engagement rate. 
     
     
         16 . The method of  claim 13 , wherein:
 the evaluation tool comprises hierarchal evaluation questions that are presented in response to receiving a response to a first hierarchal evaluation question or a previous hierarchal evaluation question, and wherein the evaluation results include data indicating a quantity of hierarchal evaluation questions presented; and   generating engagement rates for the evaluation version of the advertisement comprises weighing each engagement rate proportional to the quantity of hierarchal evaluation questions presented by the evaluation tool in each of the contexts.   
     
     
         17 . The method of  claim 13 , wherein:
 the advertisement campaign defines advertising budgets and advertisement placement requirements for the advertisements; and   the method further comprises adjusting the advertising budgets and the advertisement placement requirements of the advertisements based on the engagement rate associated with the advertisements.   
     
     
         18 . The method of  claim 13 , wherein the evaluation tool is an in situ evaluation. 
     
     
         19 . Computer-readable media that stores computer instructions executable by one or more computers, and that upon such execution cause the one or more computers to perform the operations of:
 identifying advertisements in an advertisement campaign;   generating evaluation versions of the advertisements;   serving the evaluation versions of the advertisements for presentation in a plurality of contexts;   for each serving of an evaluation version of an advertisement:
 determining the context in which the evaluation version of the advertisement is presented; and 
 determining whether a user selects the evaluation version of the advertisement in the context; 
   receiving evaluation results for evaluation conducted from the evaluation version the advertisements; and   generating an evaluation value for the advertisement for each of the plurality of contexts, the evaluation values including one or more of a brand linkage value, and advertisement likeability value, and an advertisement recognition value.   
     
     
         20 . The method of  claim 19 , further comprising:
 generating engagement rates, each engagement rate associated with an advertisement in the advertisement campaign and the corresponding evaluation version of the advertisement, and further associated with a corresponding context of the plurality of contexts, and each engagement rate a measure of a rate of participation in evaluating the advertisement for the corresponding context.   
     
     
         21 . The method of  claim 20 , further comprising:
 generating an effectiveness measure for each of the plurality of contexts for each advertisement, each effectiveness measure for each context being based on the engagement rate associated with the context and evaluation value associated with the context.   
     
     
         22 . A system, comprising:
 a client device that issues content requests to request content from publishers;   an advertisement manager that receives advertisement requests from one or more of the client and publishers in response to the content requests from the client device and selects and serves advertisements for presentation on the client device in response to the advertisement requests, the advertisement manager including a contextual evaluation subsystem that for a subset of the advertisement requests:
 generates evaluation versions of advertisements in an advertising campaign, each evaluation version including an evaluation tool for collecting evaluation results; 
 serves the evaluation versions of the advertisements for presentation in each of a plurality of contexts; 
 for each serving of an evaluation version of an advertisement, determines the context in which the evaluation version of the advertisement is presented, receives advertisement evaluation results from the evaluation conducted, and associates the evaluation results with the context; 
   wherein the contextual evaluation subsystem further generates engagement rates, each engagement rate being associated with an advertisement in the advertisement campaign and the corresponding evaluation version of the advertisement, and further being associated with a corresponding context of the plurality of contexts, each engagement rate being a measure of a participation rate of audience participation in evaluating the advertisement for a corresponding context, and each engagement rate being determined independently of performance rates of the advertisements;   wherein the client device receives the evaluation version of the advertisement and, in response to one or more user selections, generates evaluation results data and provides the evaluation results data to the advertisement manager.