Mixing items, such as ad targeting keyword suggestions, from heterogeneous sources

Abstract

The way in which targeting keywords are suggested to advertisers, particularly in instances where different suggestion tools or techniques provide a number of different sets of targeting keyword suggestions, may be improved by (a) accepting at least two heterogeneous sets of suggestions (where each of the sets of suggestions includes suggestions that are ranked and/or scored), (b) determining, for each of the sets of suggestions, a new score for each of the suggestions of the set, where the new score combines an cardinal aspect of the suggestion and an ordinal aspect of the suggestion, and (c) combining at least some of the suggestions from each of the sets using the new scores to generate a final set of ordered and/or scored suggestions. Each of the suggestions may include information for targeting the serving of an online advertisement. For example, each of the suggestions may include a keyword for targeting the serving of an online advertisement. Each of the suggestions may include a keyword type.

1 - 25 . (canceled) 
     
     
         26 . A method performed by a machine, the method comprising:
 receiving, from a first source, a first set of suggested targeting criteria, the first set of targeting criteria including one or more targeting criteria each of which has a first score and a first rank in the first set, the first rank being based on the first score;   receiving, from a second source, a second set of suggested targeting criteria, the second set of targeting criteria including one or more targeting criteria each of which has a second score and a second rank in the second set, the second rank being based on the second score;   for each suggested targeting criterion from the first set of suggested targeting criteria,
 computing a normalized first score using the first score and the first rank for the suggested targeting criterion, and 
 computing a final score using the normalized first score and a trust factor for the first source, the trust factor for the first source representing a measure of performance for the suggested targeting criterion from the first source; 
   for each suggested targeting criterion from the second set of suggested targeting criteria,
 computing a normalized second score using the second score and the second rank for the suggested targeting criterion, and 
 computing a final score using the normalized second score and a trust factor for the second source, the trust factor for the second source representing a measure of performance for the suggested targeting criterion from the second source; and 
   generating a third set of suggested targeting criteria that includes at least one suggested targeting criterion from the first set and at least one suggested targeting criterion from the second set, each suggested targeting criterion in the set having a final score that exceeds a specified threshold.   
     
     
         27 . The method of  claim 26 , further comprising:
 receiving trust factors for the first source and the second source;   receiving performance feedback for the first set of suggested targeting criteria and the second set of suggested targeting criteria; and   adjusting the trust factors based on the received performance measures.   
     
     
         28 . The method of  claim 27 , wherein receiving performance feedback comprises receiving data specifying a portion of the first set of suggested targeting criteria and a portion of the second set of suggested targeting criteria that have been used by users. 
     
     
         29 . The method of  claim 28 , wherein adjusting the trust factors comprises:
 increasing the trust factor for a source when at least a threshold portion of the suggested targeting criteria from the source have been used by users; and   decreasing the trust factor for a source when less than the threshold portion of the suggested targeting criteria from the source have been used by users.   
     
     
         30 . The method of  claim 27 , wherein receiving performance feedback comprises:
 receiving performance feedback representing performance measures for content that was presented using the first suggested targeting criteria; and   receiving performance feedback representing performance measures for content that was presented using the second suggested targeting criteria.   
     
     
         31 . The method of  claim 30 , wherein adjusting the trust factors comprises:
 increasing the trust factor for a source when performance feedback for the content presented using the suggested targeting criteria from the source exceeds a reference performance; and   decreasing the trust factor for a source when performance feedback for the content presented using the suggested targeting criteria from the source is lower than a reference performance.   
     
     
         32 . The method of  claim 26 , further comprising receiving a seed keyword with which targeting keyword suggestions are generated, wherein:
 the first source is a first keyword suggestion tool and the first set of suggested targeting criteria is a first set of suggested targeting keywords that control presentation of advertisements, the first set of suggested targeting keywords being generated by the first keyword suggestion tool using the seed keyword, and   the second source is a second keyword suggestion tool and the second set of suggested targeting criteria is a second set of suggested targeting keywords that control presentation of advertisements, the second set of suggested targeting keywords being generated by the second keyword suggestion tool using the seed keyword.   
     
     
         33 . The method of  claim 32 , wherein the first set of targeting keywords and the second set of targeting keywords each include at least one keyword having a keyword type selected from the set consisting of broadening keyword, expansion keyword, and negative keyword. 
     
     
         34 . The method of  claim 32 , further comprising providing the seed keyword to the first source and the second source. 
     
     
         35 . The method of  claim 26 , wherein generating a third set of suggested targeting criteria comprises generating a third set of suggested targeting criteria in which each suggested targeting criterion in the third set has a final score that exceeds final scores of targeting criteria from the first set and targeting criteria from the second set. 
     
     
         36 . A system comprising:
 two or more sources of suggested targeting criteria; and   one or more computers coupled to the two or more sources, the one or more computers being configured to interact with the two or more sources and perform operations including:
 receiving, from a first source, a first set of suggested targeting criteria, the first set of targeting criteria including one or more targeting criteria each of which has a first score and a first rank in the first set, the first rank being based on the first score; 
   receiving, from a second source, a second set of suggested targeting criteria, the second set of targeting criteria including one or more targeting criteria each of which has a second score and a second rank in the second set, the second rank being based on the second score;   for each suggested targeting criterion from the first set of suggested targeting criteria,
 computing a normalized first score using the first score and the first rank for the suggested targeting criterion, and 
 computing a final score using the normalized first score and a trust factor for the first source, the trust factor for the first source representing a measure of performance for the suggested targeting criterion from the first source; 
   for each suggested targeting criterion from the second set of suggested targeting criteria,
 computing a normalized second score using the second score and the second rank for the suggested targeting criterion, and 
 computing a final score using the normalized second score and a trust factor for the second source, the trust factor for the second source representing a measure of performance for the suggested targeting criterion from the second source; and 
   generating a third set of suggested targeting criteria that includes at least one suggested targeting criterion from the first set and at least one suggested targeting criterion from the second set, each suggested targeting criterion in the set having a final score that exceeds a specified threshold.   
     
     
         37 . The system of  claim 36 , wherein the one or more computers are further configured to perform operations including:
 receiving trust factors for the first source and the second source;   receiving performance feedback for the first set of suggested targeting criteria and the second set of suggested targeting criteria; and   adjusting the trust factors based on the received performance measures.   
     
     
         38 . The system of  claim 37 , wherein receiving performance feedback comprises receiving data specifying a portion of the first set of suggested targeting criteria and a portion of the second set of suggested targeting criteria that have been used by users. 
     
     
         39 . The system of  claim 38 , wherein adjusting the trust factors comprises:
 increasing the trust factor for a source when at least a threshold portion of the suggested targeting criteria from the source have been used by users; and   decreasing the trust factor for a source when less than the threshold portion of the suggested targeting criteria from the source have been used by users.   
     
     
         40 . The system of  claim 37 , wherein receiving performance feedback comprises:
 receiving performance feedback representing performance measures for content that was presented using the first suggested targeting criteria; and   receiving performance feedback representing performance measures for content that was presented using the second suggested targeting criteria.   
     
     
         41 . The system of  claim 40 , wherein adjusting the trust factors comprises:
 increasing the trust factor for a source when performance feedback for the content presented using the suggested targeting criteria from the source exceeds a reference performance; and   decreasing the trust factor for a source when performance feedback for the content presented using the suggested targeting criteria from the source is lower than a reference performance.   
     
     
         42 . The system of  claim 36 , wherein the one or more computers are further configured to perform operations including:
 receiving a seed keyword with which targeting keyword suggestions are generated, wherein:   the first source is a first keyword suggestion tool and the first set of suggested targeting criteria is a first set of suggested targeting keywords that control presentation of advertisements, the first set of suggested targeting keywords being generated by the first keyword suggestion tool using the seed keyword, and   the second source is a second keyword suggestion tool and the second set of suggested targeting criteria is a second set of suggested targeting keywords that control presentation of advertisements, the second set of suggested targeting keywords being generated by the second keyword suggestion tool using the seed keyword.   
     
     
         43 . The system of  claim 42 , wherein the first set of targeting keywords and the second set of targeting keywords each include at least one keyword having a keyword type selected from the set consisting of broadening keyword, expansion keyword, and negative keyword. 
     
     
         44 . The system of  claim 42 , wherein the one or more computers are further configured to perform operations including providing the seed keyword to the first source and the second source. 
     
     
         45 . The system of  claim 36 , wherein generating a third set of suggested targeting criteria comprises generating a third set of suggested targeting criteria in which each suggested targeting criteria in the third set has a final score that exceeds final scores of targeting criteria from the first set and targeting criterion from the second set. 
     
     
         46 . A method performed by a machine, the method comprising
 receiving at least two sets of suggestions for advertisement placement criteria, wherein each of the sets of suggestions includes suggestions that are scored and ranked, and scores for suggestions in one set of suggestions of the at least two sets of suggestions are scaled differently from scores for suggestions in at least one other set of suggestions of the at least two sets of suggestions;   selecting, by one or more processors, a specified quantity of suggestions from each of the at least two sets of suggestions;   for each of the suggestions selected from the at least two sets of suggestions:
 determining, by the one or more processors, a normalized score based on a score associated with the suggestion and a rank of the suggestion in the set from which the suggestion is selected; 
 generating, by the one or more processors, a final score using the normalized score associated with the suggestion and a trust factor of a suggestion tool that was used to generate at least one of the sets of suggestions, the trust factor representing a measure of performance for advertisement placement criteria previously suggested by the suggestion tool; 
   combining, by the one or more processors, the selected suggestions into a combined set of suggestions, wherein the selected suggestions are ordered using the final scores; and   presenting the combined set of suggestions to a user.