Methods and apparatus for assessing web page decay

Abstract

Systems and methods are herein disclosed for assessing the staleness of a web page. In particular, in one method of the present invention, the staleness of a web page is assessed by examining internal date references within the web page. In another method of the present invention, the staleness of a web page is assessed by examining the meta-data associated with the web page. In a further method of the present invention, the staleness of a hyperlinked web page is determined by examining the link status of the hyperlinks. If the web page has a relatively large number of dead links, it is assessed as being a stale web page. In a still further method of the present invention, the link status of web pages in the neighborhood of the web page being assessed is likewise examined.

1 . A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations for assessing the currency of a web page, the operations comprising: 
 establishing a date threshold, wherein web pages older than the date threshold will be assessed at not being current;    accessing a web page;    extracting date information from the web page identifying the age of the web page; and    comparing the date information extracted from the web page to the date threshold.    
   
   
       2 . The signal-bearing medium of  claim 1  further comprising: 
 identifying the web page as lacking currency if the date information identifying the age of the web page is older than the date threshold.    
   
   
       3 . The signal-bearing medium of  claim 1  further comprising: 
 identifying the web page as being current if the date information identifying the age of the web page is younger than the date threshold.    
   
   
       4 . A signal-bearing medium tangibly embodying a program of machine-readable executable by a digital processing apparatus of a computer system to perform operations for assessing the currency of a web page, the operations comprising: 
 receiving a user-specified topicality threshold, where the topicality threshold concerns the topicality of material content of the web page;    accessing a web page;    extracting topicality information from the web page; and    comparing the topicality information extracted from the web page to the topicality threshold.    
   
   
       5 . The signal-bearing medium of  claim 4  further comprising: 
 identifying the web page as lacking currency if the topicality information extracted from the web page lack topicality when compared to the topicality threshold.    
   
   
       6 . The signal-bearing medium of  claim 4  further comprising: 
 identifying the web page as being current if the topicality information extracted from the web page is topical when compared to the topicality threshold.    
   
   
       7 . A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations for assessing the currency of a web page, the operations comprising: 
 establishing a link threshold, wherein a web page will be assessed as lacking currency if a percentage of hyperlinks contained in the web page that link to an active page is less than the link threshold;    accessing a web page containing hyperlinks;    testing the hyperlinks;    calculating the percentage of hyperlinks that return active web pages; and    comparing the percentage of hyperlinks that return active web pages with the link threshold.    
   
   
       8 . The signal-bearing medium of  claim 7  where the operations further comprise: 
 identifying the web page as lacking currency if the percentage of hyperlinks that return active web pages is less than the link threshold.    
   
   
       9 . The signal-bearing medium of  claim 7  where the operations further comprise: 
 identifying the web page as being current if the percentage of hyperlinks that return active web pages is greater than the link threshold.    
   
   
       10 . The signal-bearing medium of  claim 7  where testing the hyperlinks further comprises: 
 establishing a time out limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the time out limit is exceeded;    selecting a hyperlink; and    monitoring the elapsed time until a web page is returned, if at all.    
   
   
       11 . The signal-bearing medium of  claim 10  where testing the hyperlinks further comprises: 
 assessing the hyperlink as linking to a dead page if the time out limit is exceeded.    
   
   
       12 . The signal-bearing medium of  claim 7  where testing the hyperlinks further comprises: 
 selecting a hyperlink; and    assessing the hyperlink as linking to a dead page based on an HTTP code returned in response to an HTTP request targeting the selected hyperlink.    
   
   
       13 . The signal-bearing medium of  claim 7  where testing the hyperlinks further comprises: 
 establishing a redirect limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the redirect limit is exceeded;    selecting a hyperlink; and    monitoring the number of redirects before the desired web page is returned, if at all.    
   
   
       14 . The signal-bearing medium of  claim 13  where testing the hyperlinks further comprises: 
 assessing the hyperlink as linking to a dead page if the redirect limit is exceeded.    
   
   
       15 . The signal-bearing medium of  claim 7  where testing the hyperlinks further comprises: 
 selecting a first hyperlink;    saving the web page returned in response to the selection of the first hyperlink;    formulating a web page request to a host of the first hyperlink, where the web page request is of a form that will not return an active web page with a high degree of probability; and    issuing the web page request.    
   
   
       16 . The signal-bearing medium of  claim 15  where testing the hyperlinks further comprises: 
 assessing the first hyperlink as linking to an active web page if an HTTP error code is returned in response to the web page request.    
   
   
       17 . The signal-bearing medium of  claim 15  where testing the hyperlinks further comprises: 
 saving a web page returned in response to the web page request;    comparing the web page returned in response to the web page request to the web page returned in response to the selection of the first hyperlink; and    assessing the first hyperlink as linking to a dead web page if the web page returned in response to the selection of the first hyperlink is identical to the web page returned in response to the web page request.    
   
   
       18 . The signal-bearing medium of  claim 7  where the currency of a web page is assessed by additionally testing the link status of hyperlinks contained in web pages linked through a chain of at least one hyperlink to the web page whose currency is being tested, and where: 
 establishing a link threshold further comprises applying a sliding scale weighting factor to hyperlinks contained in web pages linked to the web page whose currency is being tested, where the weight given to a dead link decreases with the distance of the web page containing the dead link from the web page whose currency is being tested in terms of intermediate web pages; and where    testing the hyperlinks further comprises testing hyperlinks in web pages linked to the web page whose currency is being tested.    
   
   
       19 . A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations for assessing the decay of a web page, the operations comprising: 
 accessing a subject web page containing hyperlinks;    assessing the decay of the subject web page by following a random walk away from the subject web page, where the random walk consists of a testing of links on the subject web page and web pages linked to the subject web page under test; and    assigning a decay score to the subject web page in dependence on dead links encountered in the random walk, wherein the decay score is a weighted sliding scale, where a dead link encountered relatively close in the random walk to the subject web page in terms of intermediate web pages results in a higher decay score than a dead link encountered relatively farther away from the subject web page.    
   
   
       20 . The signal-bearing medium of  claim 19  where testing of links further comprises: 
 establishing a time out limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the time out limit is exceeded;    selecting a hyperlink; and    monitoring the elapsed time until a web is returned, if at all.    
   
   
       21 . The signal-bearing medium of  claim 20  where testing the links further comprises: 
 assessing the hyperlink as linking to a dead page if the time out limit is exceeded.    
   
   
       22 . The signal-bearing medium of  claim 19  where testing the links further comprises: 
 selecting a hyperlink; and    assessing the hyperlink as linking to a dead page based on an HTTP code returned in response to an HTTP request targeting the selected hyperlink.    
   
   
       23 . The signal-bearing medium of  claim 19  where testing the links further comprises: 
 establishing a redirect limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the redirect limit is exceeded;    selecting a hyperlink; and    monitoring the number of redirects before the desired web page is returned, if at all.    
   
   
       24 . The signal-bearing medium of  claim 23  where testing the links further comprises: 
 assessing the hyperlink as linking to a dead page if the redirect limit is exceeded.    
   
   
       25 . The signal-bearing medium of  claim 19  where testing the links further comprises: 
 selecting a first hyperlink;    saving the web page returned in response to the selection of the first hyperlink;    formulating a web page request to the host of the first hyperlink, where the request is of a form that will not return an active web page with a high degree of probability; and    issuing the web page request.    
   
   
       26 . The signal-bearing medium of  claim 25  where testing the links further comprises: 
 assessing the first hyperlink as linking to an active web page if an HTTP error code is returned in response to the web page request.    
   
   
       27 . The signal-bearing medium of  claim 25  where testing the links further comprises: 
 saving a web page returned in response to the web page request;    comparing the web page returned in response to the web page request to the web page returned in response to the selection of the first hyperlink; and    assessing the first hyperlink as linking to a dead web page if the web page returned in response to the selection of the first hyperlink is identical to the web page returned in response to the web page request.    
   
   
       28 . A computer system for assessing the currency of a web page, the computer system comprising: 
 an internet connection for connecting to the internet and for accessing web pages available on the internet;    at least one memory to store web pages retrieved from the internet and at least one program of machine-readable instructions, where the at least one program performs operations to assess the currency of a web page;    at least one processor coupled to the internet connection and the at least one memory, where the at least one processor performs the following operations when the at least one program is executed: 
 retrieving a date threshold, wherein web pages older than the date threshold will be assessed as not being current;  
 accessing a web page;  
 extracting date information from the web page identifying the age of the web page; and  
 comparing the date information extracted from the web page to the date threshold.  
   
   
   
       29 . The computer system of  claim 28  where the operations further comprise: 
 identifying the web page as lacking currency if the date information identifying the age of the web page is older than the date threshold.    
   
   
       30 . The computer system of  claim 28  where the operations further comprise: 
 identifying the web page as being current if the date information identifying the age of the web page is younger than the date threshold.    
   
   
       31 . A computer system for assessing the currency of a web page, the computer system comprising: 
 an internet connection for connecting to the internet and for accessing web pages available on the internet;    at least one memory to store web pages retrieved from the internet and at least one program of machine-readable instructions, where the at least one program performs operations to assess the currency of a web page;    at least one processor coupled to the internet connection and the at least one memory, where the at least one processor performs the following operations when the at least one program is executed: 
 retrieving a predetermined topicality threshold, where the topicality threshold, where the topicality threshold concerns the topicality of material comprising a web page;  
 extracting topicality information from the web page; and  
 comparing the topicality information extracted from the web page to the topicality threshold.  
   
   
   
       32 . The computer system of  claim 31  where the operations further comprise: 
 identifying the web page as lacking currency if the topicality information extracted from the web page lacks topicality when compared to the topicality threshold.    
   
   
       33 . The computer system of  claim 31  where the operations further comprise: 
 identifying the web page as being current if the topicality information extracted from the web page is topical when compared to the topicality threshold.    
   
   
       34 . A computer system for assessing the currency of a web page, the computer system comprising: 
 an internet connection for connecting to the internet and for accessing web pages available on the internet;    at least one memory to store web pages retrieved from the internet and at least one program of machine-readable instructions, where the at least one program performs operations to assess the currency of a web page;    at least one processor coupled to internet connection and the at least one memory, where the at least processor performs the following operations when the at least one program is executed; 
 establishing a link threshold, wherein a web page will be assessed as lacking currency if a percentage of hyperlinks contained in the web page that link to an active page is less than the link threshold;  
 accessing a web page containing hyperlinks;  
 testing the hyperlinks;  
 calculating the percentage of hyperlinks that return active web pages; and  
 comparing the percentage of hyperlinks that return active web pages with the link threshold.  
   
   
   
       35 . The computer system for assessing the currency of a web page of  claim 34  where the operations further comprise: 
 identifying the web page as lacking currency if the percentage of hyperlinks that return active web pages is less than the link threshold.    
   
   
       36 . The computer system for assessing the currency of a web page of  claim 34  where the operations further comprise: 
 identifying the web page as being current if the percentage of hyperlinks that return active web pages is greater than the link threshold.    
   
   
       37 . The computer system for assessing the currency of a web page of  claim 34  where testing the hyperlinks further comprises: 
 establishing a time out limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page of the time out limit is exceeded;    selecting a hyperlink; and    monitoring the elapsed time until a web page is returned, if at all.    
   
   
       38 . The computer system for assessing the currency of a web page of  claim 37  where testing the hyperlinks further comprises: 
 assessing the hyperlink as linking to a dead page if the time out limit is exceeded.    
   
   
       39 . The computer system for assessing the currency of a web page of  claim 34  where testing the hyperlinks further comprises: 
 selecting a hyperlink; and    assessing the hyperlink as linking to a dead page based on an HTTP code returned in response to an HTTP request targeting the selected hyperlink.    
   
   
       40 . The computer system for assessing the currency of a web page of  claim 34  where testing the hyperlinks further comprises: 
 establishing a redirect limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the redirect limit is exceeded;    selecting a hyperlink; and    monitoring a number of redirects before the desired web page is returned, if at all.    
   
   
       41 . The computer system for assessing the currency of a web page of  claim 40  where testing the hyperlinks further comprises: 
 assessing the hyperlink as linking to a dead web page if the redirect limit is exceeded.    
   
   
       42 . The computer system for assessing the currency of a web page of  claim 34  where testing the hyperlinks further comprises: 
 selecting a first hyperlink;    saving the web page returned in response to the selection of the first hyperlink;    formulating a web page request to the parent directory of the address corresponding to the first hyperlink, where the web page request is of a form that will not return an active web page with a high degree of probability; and    issuing the web page request.    
   
   
       43 . The computer system for assessing the currency of a web page of  claim 42  where testing the hyperlinks further comprises: 
 assessing the first hyperlink as linking to an active web page if an HTTP error code is returned in response to the web page request.    
   
   
       44 . The computer system for assessing the currency of a web page of  claim 42  where testing the hyperlinks further comprises: 
 saving a web page returned in response to the web page request;    comparing the web page returned in response to the web page request to the web page returned in response to the selection of the first hyperlink; and    assessing the first hyperlink as linking to a dead web page if the web page returned in response to the selection of the first hyperlink is identical to the web page returned in response to the web page request.    
   
   
       45 . The computer system for assessing the currency of a web page of  claim 34  where the currency of a web page is assessed by additionally testing the link status of hyperlinks contained in web pages linked through a chain of at least one hyperlink to the web page whose currency is being tested, and where: 
 establishing a link threshold further comprises applying a sliding scale weighting factor to hyperlinks contained in web pages linked to the web page whose currency is being tested, where the weight given to a dead link decreases with the distance of the web page containing the dead link from the web page whose currency is being tested in terms of intermediate web pages; and where    testing the hyperlinks further comprises testing hyperlinks in web pages linked from the web page whose currency is being tested.    
   
   
       46 . A computer system for assessing the decay of a web page comprising: 
 an internet connection for connecting to the internet and for accessing web pages available on the internet;    at least one memory to store web pages retrieved from the internet and at least one program of machine-readable instructions, where the at least one program performs operations to assess the decay of web page;    at least one processor coupled to the internet connection and the at least one memory, where the at least one processor performs the following operations when the at least one program is executed: 
 accessing a subject web page containing hyperlinks;  
 assessing the decay of the subject web page by following a random walk away from the subject web page, where the random walk consists of a testing of links on the subject web page and web pages linked to the subject web page under test; and  
 assigning a decay score to the subject web page in dependence on dead links encountered in the random walk, wherein the decay score is a weighted sliding scale, where a dead link encountered relatively close in the random walk to the subject web page in terms of intermediate web pages results in a higher decay score than a dead link encountered relatively farther away from the subject web page.  
   
   
   
       47 . The computer system for assessing the decay of a web page of  claim 46  where testing of links further comprises: 
 establishing a time out limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the time out limit is exceeded;    selecting a hyperlink; and    monitoring the elapsed time until a web is returned, if at all.    
   
   
       48 . The computer system for assessing the decay of a web page of  claim 47  where testing the links further comprises: 
 assessing the hyperlink as linking to a dead page if the time out limit is exceeded.    
   
   
       49 . The computer system for assessing the decay of a web page of  claim 46  where testing the links further comprises: 
 selecting a hyperlink; and    assessing the hyperlink as linking to a dead page based on the HTTP code returned in response to an HTTP request targeting the selected hyperlink.    
   
   
       50 . The computer system for assessing the decay of a web page of  claim 46  where testing the links further comprises: 
 establishing a redirect limit for testing a hyperlink, where when the hyperlink is tested, the hyperlink will be assessed as linking to a dead web page if the redirect limit is exceeded;    selecting a hyperlink; and    monitoring the number of redirects before the desired web page is returned, if at all.    
   
   
       51 . The computer system for assessing the decay of a web page of  claim 50  where testing the links further comprises: 
 assessing the hyperlink as linking to a dead page if the redirect limit is exceeded.    
   
   
       52 . The computer system for assessing the decay of a web page of  claim 46  where testing the links further comprises: 
 selecting a first hyperlink;    saving the web page returned in response to the selection of the first hyperlink;    formulating a web page request to the host of the first hyperlink, where the request is of a form that will not return an active web page with a high degree of probability; and    issuing the web page request.    
   
   
       53 . The computer system for assessing the decay of a web page of  claim 52  where testing the links further comprises: 
 assessing the first hyperlink as linking to an active web page if an HTTP error code is returned in response to the web page request.    
   
   
       54 . The computer system for assessing the decay of a web page of  claim 52  where testing the links further comprises: 
 saving a web page returned in response to the web page request;    comparing the web page returned in response to the web page request to the web page returned in response to the selection of the first hyperlink; and    assessing the first hyperlink as linking to a dead web page if the web page returned in response to the selection of the first hyperlink is identical to the web page returned in response to the web page request.