Method and system for extracting web data

Abstract

An apparatus for providing an analysis of attitudes expressed in web content, comprising: a collector for collecting attitude-data in relation to a predetermined subject from one or more pre-selected web site, the attitude-data containing attitudes in relation to the predetermined subject; a processor, associated with the collector, for processing the attitude data so as to generate an attitude analysis; and an outputter, associated with the processor, for outputting the attitude analysis, thereby to provide an indication of attitudes being expressed in the web content in relation to the predetermined subject.

1 . Apparatus for providing an analysis of attitudes expressed in web sites, comprising: 
 a collector for collecting attitude-data in relation to a predetermined subject from at least one pre-selected web site, said attitude-data containing attitudes in relation to said predetermined subject;    a processor, associated with said collector, for processing said attitude data so as to generate an attitude analysis; and    an outputter, associated with said processor, for outputting said attitude analysis, thereby to provide an indication of attitudes being expressed in said web content in relation to said predetermined subject.    
     
     
         2 . The apparatus of  claim 1 , wherein said collector further comprises at least one crawler, configured for crawling said at least one respective pre-selected web site for the attitude data in relation to said predetermined subject.  
     
     
         3 . The apparatus of  claim 2 , wherein said crawler is further configured to download relevant pages of the at least one pre-selected web site according to a predetermined schedule.  
     
     
         4 . The apparatus of  claim 1 , wherein said collector is further configured to extract relevant data from said relevant pages of the pre-selected web site.  
     
     
         5 . The apparatus of  claim 1 , wherein said collector is further configured to create a mark-up language format representation of said relevant data.  
     
     
         6 . The apparatus of  claim 5 , wherein said mark-up language is XML.  
     
     
         7 . The apparatus of  claim 1 , wherein said collector further comprises a data integrator, configured for integrating the attitude data into a complete and non-redundant attitude data.  
     
     
         8 . The apparatus of  claim 1 , further comprising a collecting policy definer, associated with the collector, operable for defining specific guidelines for collecting with respect to one of said at least one pre-selected web site.  
     
     
         9 . The apparatus of  claim 1 , wherein said processor is further configured for carrying out an initial preprocessing step for extracting metadata of said attitude-data.  
     
     
         10 . The apparatus of  claim 1 , wherein said processor is further operable for categorizing text of said attitude-data.  
     
     
         11 . The apparatus of  claim 10 , wherein said categorization comprises categorization of said attitude data according to style.  
     
     
         12 . The apparatus of  claim 10 , wherein said categorization comprises categorization of said attitude data according to content.  
     
     
         13 . The apparatus of  claim 1 , wherein said processor further comprises a statistics generator, configured for generating statistics based on said processing attitude-data.  
     
     
         14 . The apparatus of  claim 1 , wherein said processor further comprises a concept analyzer, configured for analyzing said attitude-data and for finding in said attitude-data at least one relevance-relationship between a phrase and a respective concept relating to said subject.  
     
     
         15 . The apparatus of  claim 14 , wherein said concept is an attitude.  
     
     
         16 . The apparatus of  claim 1 , wherein said processor further comprises a correlation measurer, configured for measuring in said attitude-data correlations among phrases having a relevance-relationship with a common concept and between at least one of said phrases and said common concept, said concept relating to said subject.  
     
     
         22 - 25 . (canceled)  
     
     
         26 . The apparatus of  claim 1 , wherein said outputter is implemented using as web browser based application.  
     
     
         27 - 43 . (canceled)