Data refining engine for high performance analysis system and method

Abstract

Price and product attributes from webpages are imported, indexed, analyzed, and made available to be searched in close-to realtime, allowing search for price changes specific to products on individual webpages and for products across all webpages as well as to identify longitudinal correlations between price changes and product attributes. Users may search the data and set alerts.

1 . A computer implemented method of storing information and searching the stored information in close-to realtime, the method comprising:
 at a first computer comprising a processor and a memory, which memory comprises: an attribute datastore for storing price and product attributes for a set of products, which price and product attributes are obtained from webpages accessed via Uniform Resource Identifiers (“URIs”),
 a history datastore for storing historical price and product attributes for the set of products, and 
 an analytics datastore for storing the result of an analysis of the price and product attributes in the attribute datastore and the history datastore; 
   at the first computer receiving a set of price and product attributes obtained from a website for a first product, which attributes comprise a first category in a category taxonomy;   at the first computer determining a geographic area of the website;   at the first computer determining a replica of the attribute datastore and a replica of the analytics datastore, based on the geographic area of the website and the first category;   by the first computer storing the price and product attributes in the determined replica of the attribute datastore;   performing the analysis and storing the result of the analysis in the determined replica of the analytics datastore;   with respect to the price and product attributes stored in the determined replica of the attribute datastore, obtaining a set of values of high cardinality entries in the price and product attributes and hashing each such value to form a set of index key hash values;   forming an inverted bitmap index of a subset of the result of the analysis of the price and product attributes stored in the determined replica of the analytics datastore;   receiving a query;   hashing a high cardinality search term in the query, if any, and comparing the hashed high cardinality search term with the set of index key hash values to determine that the replica of the attribute datastore comprises a set of records responsive to the query or, if the query does not comprise a high cardinality search term, searching for terms in the query according to the inverted bitmap index to identify the corresponding replica in the analytics datastore and the set of records responsive to the query therein;
 in response to the query, returning the set of records responsive to the query. 
   
     
     
         2 - 23 . (canceled)