Determining Advertising Effectiveness Based on Observed Actions in a Social Networking System

Abstract

Actions and/or behaviors of social networking system users are observed and used for measuring advertising effectiveness. More specifically, advertisements from an advertising campaign are selectively targeted and presented to specific subsets of social network users and withheld from other subsets of social network users. After the advertisements are presented, actions performed by users in the different subsets are be identified and analyzed to determine metrics describing the effectiveness of the particular advertising campaign.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 selecting a holdout subset of users from a plurality of users of a social networking system, the holdout subset being associated with one or more advertisements;   selecting an advertisement from the one or more advertisements to serve to a user of the social networking system;   determining whether the user is in the holdout subset;   responsive to a determination that the user is in the holdout subset, preventing the user from being presented with the selected advertisement;   responsive to a determination that the user is not in the holdout subset, presenting the selected advertisement to the user;   storing observed actions performed by the plurality of users of the social networking system on objects of the social networking system;   determining, based on the stored actions, one or more actions performed by users of the social networking system in the holdout subset;   determining, based on the stored actions, one or more actions performed by users of the social networking system not in the holdout subset; and   calculating a measure of effectiveness for the one or more advertisements based at least in part on the determined actions performed by the one or more users of the social networking system not in the holdout subset and the determined actions performed by the users of the social networking system in the holdout subset.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the stored actions performed by the plurality of users of the social networking system comprise actions associated with the one or more advertisements. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein the determined one or more actions performed by the users of the social networking system not in the holdout subset include at least one action selected from a group consisting of: a search, a like, a share, a comment, and a join action. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein calculating the measure of effectiveness comprises:
 determining a total number for actions performed by the users not in the holdout subset that are associated with the one or more advertisements;   determining a total number for the actions performed by the users of the holdout subset that are associated with the one or more advertisements; and   computing a lift metric based on a comparison of the total number for the actions performed by the users not in the holdout subset and the total number for the actions performed by the users of the holdout subset.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein a determined action performed by a particular user not in the holdout subset is indicative of an awareness of a subject of an advertisement from the one or more advertisements. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein a determined action performed by a particular user not in the holdout subset is indicative of a positive perception of a subject of an advertisement from the one or more advertisements. 
     
     
         7 . The computer-implemented method of  claim 6 , wherein the action indicative of a positive perception is at least one of a like, a share, or a join action. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein calculating the measure of effectiveness for the one or more advertisements is further based in part on polling data from the plurality of users of the social networking system. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein calculating the measure of effectiveness for the one or more advertisements is further based in part on purchase transaction data for the plurality of users of the social networking system. 
     
     
         10 . The computer-implemented method of  claim 9 , wherein the purchase transaction data comprises information for a set of purchases made by users of the social networking system with respect to products promoted by the one or more advertisements. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein selecting the holdout subset comprises:
 generating a hash value associated with a particular user of the social networking system by applying a hash function to a user identifier of the particular user of the social networking system; and   responsive to the hash value being within a specified range, including the particular user in the holdout subset.   
     
     
         12 . The method of  claim 1 , wherein a determined action performed by a particular user of the social networking system not in the holdout subset comprises initiating the transmission of a story describing content of an advertisement from the one or more advertisements to another user of the social networking system, wherein the another user has a social network connection with the particular user of the social networking system not in the holdout subset. 
     
     
         13 . A computer-implemented method comprising:
 selecting an advertisement from an advertising campaign to serve to a user of a social networking system;   determining if the user is in a control group associated with the advertising campaign;   if the user is in the control group, preventing the user from being served with the selected advertisement;   if the user is in not in the control group, serving the selected advertisement to the user;   storing actions performed by users of the social networking system over the social networking system;   determining actions performed by users in the control group based on the stored actions; and   calculating a metric describing an effectiveness of the advertising campaign based in part on the determined actions performed by the users in the control group.   
     
     
         14 . The computer-implemented method of  claim 13 , wherein an action performed by a particular user in the control group comprises transmitting a story describing content associated with an advertisement in the advertising campaign to another user connected to the particular user. 
     
     
         15 . The computer-implemented method of  claim 14 , wherein the content associated with the advertisement is included in at least one of: a page maintained by the social networking system, an event maintained by the social networking system, or a group maintained by the social networking system. 
     
     
         16 . The computer-implemented method of  claim 13 , wherein the determined actions performed by the users in the control group include a search performed by a particular user in the control group for a product, service or brand associated with an advertisement from the advertising campaign. 
     
     
         17 . The computer-implemented method of  claim 13 , wherein the metric describing the effectiveness of the advertising campaign represents a difference in awareness of a brand associated with the advertising campaign between users of the social networking system not in the control group and the users of the social networking system in the control group. 
     
     
         18 . A computer-implemented method comprising:
 preventing presentation of an advertisement to users of a social networking system in a holdout subset associated with the advertisement;   presenting the advertisement to users of the social networking system not in the holdout subset;   determining actions performed in the social networking system by users in the holdout subset;   determining actions performed in the social networking system by users not in the holdout subset; and   calculating a metric describing an effectiveness of the advertisement based on the actions performed in the social networking system by the users in the holdout subset and actions performed in the social networking system by the users not in the holdout subset.   
     
     
         19 . The computer-implemented method of  claim 18 , wherein calculation of the metric is based in part on a comparison between (1) an awareness metric for a product, service or brand related to the advertisement with respect to users not in the holdout subset and (2) an awareness metric for the product, service or brand related to the advertisement with respect to the holdout subset of users. 
     
     
         20 . The computer-implemented method of  claim 18 , wherein calculation of the metric is based in part on a comparison between (1) a perception metric for a product, service or brand related to the advertisement with respect to the users not in the holdout subset and (2) a perception metric for the product, service or brand related to the advertisement with respect to the holdout subset of users.