Evaluating the ranking quality of a ranked list

Abstract

The ranking quality of a ranked list may be evaluated. In an example embodiment, a method is implemented by a system to access log data, ascertain which entries of a ranked list are skipped, and determine a ranking quality metric from the skipped entries. More specifically, log data that reflects user interactions with a ranked list having multiple entries is accessed. The user interactions include at least indications of which of the multiple entries are selected entries. It is ascertained which entries of the multiple entries of the ranked list are skipped entries based on the selected entries. The ranking quality metric for the ranked list is determined responsive to the skipped entries.

1 - 20 . (canceled) 
     
     
         21 . A system comprising:
 a processing unit; and   a processor-accessible media storing executable instructions which, when executed by the processing unit, cause the processing unit to:   obtain data identifying user interactions by a user with a ranked list having multiple entries; and   provide a characterization of the ranked list based at least on dwell times of the user with different entries in the ranked list.   
     
     
         22 . The system of  claim 21 , wherein the characterization comprises a ranking quality metric for the ranked list. 
     
     
         23 . The system of  claim 21 , wherein the multiple entries include documents. 
     
     
         24 . The system of  claim 21 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 tune an application that generates the ranked list based at least on the characterization.   
     
     
         25 . The system of  claim 24 , wherein the application comprises a search engine. 
     
     
         26 . The system of  claim 21 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 identify viewed entries in the ranked list that are viewed by the user, the viewed entries including the different entries and one or more other entries; and   provide the characterization based at least on the viewed entries.   
     
     
         27 . The system of  claim 26 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 identify a last selected entry in the ranked list; and   exclude, from the viewed entries, other entries occurring in the ranked list after the last selected entry.   
     
     
         28 . A method comprising:
 obtaining data identifying user interactions by a user with a ranked list having multiple entries;   ascertaining, from the data, that the multiple entries include multiple skipped entries that the user has not selected; and   determining a ranking quality metric for the ranked list based at least on the multiple skipped entries.   
     
     
         29 . The method of  claim 28 , further comprising:
 based at least on the ranking quality metric, tuning a search engine that produced the ranked list.   
     
     
         30 . The method of  claim 28 , wherein the data comprises log data reflecting clicks by the user on other entries from the ranked list. 
     
     
         31 . The method of  claim 28 , further comprising:
 identifying a partially skipped entry in the ranked list based at least on a dwell time of the user on the partially skipped entry; and   determining the ranking quality metric for the ranked list based at least on the partially skipped entry.   
     
     
         32 . The method of  claim 31 , further comprising:
 comparing the dwell time of the user on the partially skipped entry to a threshold; and   responsive to a determination that the dwell time falls below the threshold, characterizing the partially skipped entry as partially skipped by the user.   
     
     
         33 . The method of  claim 28 , further comprising:
 determining a first number of the multiple entries of the ranked list that the user has viewed;   determining a second number of the multiple skipped entries that the user has not selected; and   determining the ranking quality metric using the first number and the second number.   
     
     
         34 . The method of  claim 33 , wherein the ranking quality metric reflects a probability of the user skipping an entry in the ranked list. 
     
     
         35 . A system comprising:
 a processing unit; and   a processor-accessible media storing executable instructions which, when executed by the processing unit, cause the processing unit to:   obtain data identifying user interactions by a user with a ranked list having multiple entries;   identify at least two different partially-skipped entries in the ranked list having dwell times below a specified dwell time threshold; and   determine a ranking quality metric for the ranked list based at least on the at least two different partially-skipped entries in the ranked list having the dwell times below the specified dwell time threshold.   
     
     
         36 . The system of  claim 35 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 determine the ranking quality metric for at least two different zones of a web page, including a first zone having the ranked list and a second zone having another ranked list.   
     
     
         37 . The system of  claim 36 , wherein the web page is a search results page. 
     
     
         38 . The system of  claim 35 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 identify viewed entries in the ranked list, the viewed entries including the at least two different partially-skipped entries; and   determine a ranking quality metric for the ranked list using the viewed entries and the at least two different partially-skipped entries.   
     
     
         39 . The system of  claim 38 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 determine a dwell time distribution from various dwell times of the user interactions; and   use the dwell time distribution to characterize the at least two different partially-skipped entries as having been partially skipped by the user.   
     
     
         40 . The system of  claim 39 , wherein the executable instructions, when executed by the processing unit, cause the processing unit to:
 separate the dwell time distribution into a first component for selected entries and a second component for partially skipped entries; and   characterize the at least two different partially-skipped entries as partially skipped using the second component.