Dynamic page similarity measurement

Abstract

A method for determining which web page among multiple candidate web pages is similar to a given web page. For each candidate web page, a set of scoring rules is provided to score the components therein. When the given web page is compared against a candidate web page, each component that is found in both the given web page and the candidate web page under examination is given a score in accordance with the set of scoring rules that is specific to that web page under examination. A composite similarity score is computed for each comparison between the given webpage and a candidate web page. If the composite similarity score exceeds a predefined threshold value for a comparison between the given webpage and a candidate web page, that candidate web page is deemed the web page that is similar.

1 . A computer-implemented method for ascertaining which web page among a plurality of candidate web pages is similar to a given web page, comprising:
 extracting a set of web page components from said given web page;   comparing said given web page against each of said plurality of candidate web pages in turn, said comparing results in a composite similarity score for said set of web page components, said composite similarity score being computed from scores assigned to individual ones of said set of web page components in accordance with a set of scoring rules associated with said web page that is under examination for similarity, wherein a web page component of said set of web page components is associated with a first score if said web page component also exists in said web page that is under examination for similarity, said web page component of said set of web page components is associated with second score different from said first web page component if said web page component does not exists in said web page that is under examination for similarity; and   if said composite similarity score exceeds a predefined threshold, designating said given web page similar to said web page that is under examination for similarity.   
   
   
       2 . The method of  claim 1  wherein said set of web page components includes at least a URL string. 
   
   
       3 . The method of  claim 1  wherein said set of web page components includes an image element. 
   
   
       4 . The method of  claim 1  wherein said web page component represents text. 
   
   
       5 . The method of  claim 4  wherein said web page component is tested for similarity using a regular expression. 
   
   
       6 . The method of  claim 1  wherein said web page component is visible. 
   
   
       7 . The method of  claim 1  wherein said web page component is invisible. 
   
   
       8 . The method of  claim 1  wherein said comparing is performed until a similar web page is found. 
   
   
       9 . The method of  claim 1  further comprising providing a warning indication if multiple web pages of said plurality of web pages are deemed similar to said given web page. 
   
   
       10 . A computer-implemented method for designating a given web page similar or dissimilar with respect to a reference web page, comprising:
 extracting a set of web page components from said given web page;   computing, using a set of scoring rules associated with said reference web page, a composite similarity score for said set of web page components, said composite similarity score being computed from scores assigned to individual ones of said set of web page components, wherein a web page component of said set of web page components is assigned first score if said web page component also exists in said reference web page, said web page component of said set of web page components is assigned second score different from said first web page component if said web page component does not exists in said reference web page;   if said composite similarity score exceeds a predefined threshold, designating said given web page similar to said reference web page.   
   
   
       11 . The method of  claim 10  wherein said set of web page components includes at least a URL string. 
   
   
       12 . The method of  claim 10  wherein said set of web page components includes an image element. 
   
   
       13 . The method of  claim 10  wherein said web page component represents text. 
   
   
       14 . The method of  claim 13  wherein said web page component is tested for similarity using a regular expression. 
   
   
       15 . The method of  claim 10  wherein said web page component is visible. 
   
   
       16 . The method of  claim 10  wherein said web page component is invisible. 
   
   
       17 . An article of manufacture comprising a computer storage medium for storing thereon computer readable code for ascertaining which web page among a plurality of candidate web pages is similar to a given web page, comprising:
 computer readable code for extracting a set of web page components from said given web page;   computer readable code for comparing said given web page against each of said plurality of candidate web pages in turn, said comparing results in a composite similarity score for said set of web page components, said composite similarity score being computed from scores assigned to individual ones of said set of web page components in accordance with a set of scoring rules associated with said web page that is under examination for similarity, wherein a web page component of said set of web page components is associated with a first score if said web page component also exists in said web page that is under examination for similarity, said web page component of said set of web page components is associated with second score different from said first web page component if said web page component does not exists in said web page that is under examination for similarity; and   computer readable code for designating, if said composite similarity score exceeds a predefined threshold, said given web page similar to said web page that is under examination for similarity.   
   
   
       18 . The article of manufacture of  claim 17  wherein said set of web page components includes at least a URL string. 
   
   
       19 . The article of manufacture of  claim 17  wherein said set of web page components includes an image element. 
   
   
       20 . The article of manufacture of  claim 17  wherein said web page component represents text. 
   
   
       21 . The article of manufacture of  claim 20  wherein said web page component is tested for similarity using a regular expression.