System and method for gathering ecommerce data

Abstract

The techniques introduced here provide a method of gathering ecommerce data. The techniques described here allow a system to return information about a product from several non-related ecommerce sites in response to a single search query. Using the techniques described here, a data mining system determines from the search query a product ID and retrieves from a database one or more product links that correspond to the product ID. Using the product links retrieved from the database, the data mining system traverses the links and parses the web-pages corresponding each of the links to determine up to date product information. The product information can then be returned to the application that initiated the request.

What is claimed is: 
     
         1 . A method for collecting product information based on a unique link ID, comprising:
 receiving at a data mining system, a product URL;   determining, at the data mining system, a unique link ID associated with the product URL based on a textual pattern of the product URL defined by a regular expression;   identifying, from a database of the data mining system, one or more product links that correspond to a product represented by the unique link ID;   retrieving webpages corresponding to the one or more product links, the webpages presenting the product; and   collecting product information of the product by parsing the webpages.   
     
     
         2 . The method of  claim 1 , further comprising:
 displaying the product information to a user of the data mining system.   
     
     
         3 . The method of  claim 1 , wherein said determining a unique link ID comprises:
 determining from the product URL a domain and retrieving the regular expression that corresponds to the domain from the database;   processing the product URL using the regular expression by identifying the textual pattern of alphanumeric characters, numbers or symbols that is defined by the regular expression;   combining the domain with the processed product URL to create a unique link; and   retrieving the unique link ID associated with the unique link from the database.   
     
     
         4 . The method of  claim 1 , further comprising writing the product information to the database, wherein the product link is used as a key for the product information. 
     
     
         5 . The method of  claim 1 , further comprising writing the product information to a local instance of a shared cache and replicating the product information to other instances of the shared cache. 
     
     
         6 . The method of  claim 1 , wherein the product information includes a screenshot of a web-page referenced by the product link. 
     
     
         7 . The method of  claim 1 , wherein the product information includes at least one of, a SKU, a product name, a manufacturer name, a part number, a physical description, or a sales price. 
     
     
         8 . A data mining system for acquiring product information online, comprising:
 a network server coupled with an application through a network, the network server configured to receive from the application a URL from a domain, the URL associated with a product;   wherein, in operation, the web server identifies a unique product ID from the URL using a regular expression corresponding to the domain, and retrieves from a database one or more product links that are associated with the unique product ID, and creates a job request to gather information from the one or more product links; and   an acquisition engine configured to traverse the one or more product links in response to the job request, wherein the acquisition engine, in operation, parses a web-page corresponding to each of the one or more product links and returns information relating to the product from the one or more product links.   
     
     
         9 . The data mining system of  claim 8 , further comprising a job server configured to queue the job request until the acquisition engine is available to process the job request. 
     
     
         10 . The data mining system of  claim 8 , further comprising a datastore including a product information database, wherein the information relating to the product is written to the product information database. 
     
     
         11 . The data mining system of  claim 8 , further comprising a shared cache, wherein the information relating the product is written to the shared cache. 
     
     
         12 . The data mining system of  claim 8 , further comprising an application programming interface (API) configured to provide an interface with the application to access the information gathered by the data mining system. 
     
     
         13 . A computer server, comprising:
 a processor;   a network interface, coupled to the processor, through which the computer server can communicate with ecommerce sites;   a data storage storing a database; and   a memory storing instructions which, when executed by the processor, cause the computer server to perform a process including:
 receiving, via the network interface, a product URL of a product web page from an ecommerce site; 
 determining a prime URL from the product URL by extracting the alphanumeric characters, numbers or symbols from the product URL using a regular expression corresponding to the ecommerce site, wherein the prime URL is a URL that cannot be expressed in any simpler form and still leads to the product web page; and 
 retrieving, form the database, a product ID associated with the prime URL, wherein the product ID identifies a product being presented in the product web page. 
   
     
     
         14 . The computer server of  claim 13 , wherein said determining a prime URL from the product URL comprises:
 determining, by the processor, a domain of the ecommerce site from the product URL of the product web page;   retrieving, from the database, a base URL and a regular expression associated with the domain; and   creating a prime URL by running the product URL through a regular expression and adding the base URL.   
     
     
         15 . The computer server of  claim 13 , wherein the process further includes:
 associating the product ID to a plurality of product URLs, wherein the plurality of product URLs have the same prime URL.   
     
     
         16 . A method for acquiring product data through a network, comprising:
 receiving, at a network server, a search request from an application for product information, wherein the search request includes a network address pointing to a webpage;   determining a domain hosting the webpage;   identifying a regular expression associated with the domain;   locating a unique link ID from the network address by running the network address through the regular expression;   determining, from a database, one or more network addresses associated with the unique link ID; and   collecting the product information from one or more webpages to which the one or more network addresses point.   
     
     
         17 . The method of  claim 16 , further comprising:
 storing the product information into the database using the unique link ID as a database key.   
     
     
         18 . The method of  claim 16 , further comprising:
 determining whether a copy of a webpage of the one or more webpages is cached; and   collecting the product information from the cached copy of the webpage instead of retrieving the webpage from a server.