Central aggregator architechture for question and answer search engine

Abstract

A question and answer search engine includes a load balancer that receives the request from the user computer system, a plurality of front end systems that receive the requests from the load balancer, including the request from the user computer system, an aggregator and a plurality of retrievers, the aggregator being connected to the front end systems and to the retrievers, the request passing from a respective front end system via the aggregator to at least a first of the retrievers, the first retriever returning a result via the aggregator and the respective front end system to the user computer system in response to the request.

What is claimed: 
     
         1 . A question and answer system for providing results to requests comprising:
 an online system that includes:
 at least one data store; 
 a question and answer search engine that receives a request from a user computer system, determines a result from the data store based on the request and returns the answer to the user computer system, the question and answer search engine including: 
 a load balancer that receives the request from the user computer system; 
 a plurality of front end systems that receive the requests from the load balancer, including the request from the user computer system; 
 an aggregator; and 
 a plurality of retrievers, the aggregator being connected to the front end systems and to the retrievers, the request passing from a respective front end system via the aggregator to at least a first of the retrievers, the first retriever returning a result via the aggregator and the respective front end system to the user computer system in response to the request; and 
   an offline system that includes:
 a file system; 
 a hierarchical database; and 
 an index controller having: 
 at least one reducer that retrieves content from the file system; and 
 at least one writer that maintains the data store with the content retrieved by the reducer, and maintains the hierarchical database with data reflecting the content in the data store. 
   
     
     
         2 . The system of  claim 1 , wherein the request passes from the respective front end system via the aggregator to at least a second of the retrievers, the second retriever returning a result via the aggregator and the respective front end system to the user computer system in response to the request. 
     
     
         3 . The system of  claim 2 , wherein the aggregator aggregates the results received from the first and second retrievers. 
     
     
         4 . The system of  claim 2 , further comprising:
 a cache forming part of the load balancer, wherein the front end system checks whether a cached result is available in the cache, wherein if a cached result is available then the front end system retrieves the cached result, the cached result being the result that is returned; and   if a cached result is not available then processes result extraction to obtain at least one processed result, the processed result being the result that is returned; and   
       updates the cache with the processed result. 
     
     
         5 . The system of  claim 4 , further comprising:
 a metaservice holding a plurality of global question identifiers, wherein the result extraction includes:
 translating parameters of the request into data parameters suitable for determining the answer from the data store; 
 determining a selected one of a plurality of modes based on the request; 
 filling in data parameters defined for the selected mode; 
 removing common words; 
 requesting a global question identifier from the metaservice; 
 processing pre request blocking; 
 blocking of answers based on text of the request and the global question identifier; 
 requesting the aggregator to provide search results; 
 processing post request blocking; 
 processing results for field collapsing; 
 retaining a maximum of predetermined number of results for each field value; 
 removing duplicate results in the form of question and answer pairs that have exactly the same question and answer; and 
 normalizing scores of the results to a common scale. 
   
     
     
         6 . The system of  claim 4 , wherein the front end system processes post request blocking if the cached result is available. 
     
     
         7 . The system of  claim 1 , wherein the offline system includes:
 a crawler that connects over the Internet to remote computer systems to retrieve data for that is placed in the file system.   
     
     
         8 . The system of  claim 7 , further comprising:
 a batch update crawl cluster that includes:
 a crawl database within the file system; and 
 a map reducer within the index controller, the map reducer having a reducer core with a plurality of slow queues that retrieve the content from the crawl database, and a reducer adapter that writes an output of the reducer core into the hierarchical database; and 
   a fast update crawl cluster that includes:
 a crawl database within the file system; and 
 a map reducer within the index controller, the map reducer having a reducer core with a plurality of fast queues that retrieve the content from the crawl database at a faster frequency than the slow queues, and a reducer adapter that writes an output of the reducer core into the hierarchical database. 
   
     
     
         9 . The system of  claim 1 , wherein the offline system includes:
 at least one data store, the writer of the index controller writing to the data store of the offline system and the data store of the online system synchronizing with the data store of the offline system.   
     
     
         10 . A method for providing results to requests comprising:
 receiving, with a question and answer search engine of an online system, a request from a user computer system;   receiving the request from the user computer system at a load balancer of the question and answer search engine;   receiving requests at a plurality of front end systems of the question and answer search engine from the load balancer, including the request from the user computer system;   passing the request from a respective front end system via an aggregator of the question and answer search engine, the aggregator being connected to the front end systems and to the retrievers to at least a first of the retrievers, the first retriever returning a result via the aggregator and the respective front end system to the user computer system in response to the request;   determining, with the question and answer search engine, a result from a data store of the online system based on the request and returns the answer to the user computer system;   returning a result from the respective retriever via the aggregator and the respective front end system to the user computer system in response to the request;   returning, with the question and answer search engine, the answer to the user computer system;   retrieving, with at least one reducer of an index controller of an offline system, content from a file system of the offline system; and   maintaining, with at least one writer of the index controller, the data store with the content retrieved by the reducer, and the hierarchical database with data reflecting the content in the data store.   
     
     
         11 . The method of  claim 10 , wherein the request passes from the respective front end system via the aggregator to at least a second of the retrievers, the second retriever returning a result via the aggregator and the respective front end system to the user computer system in response to the request. 
     
     
         12 . The method of  claim 11 , further comprising:
 aggregating, with the aggregator, the results received from the first and second retrievers.   
     
     
         13 . The method of  claim 11 , further comprising:
 checking whether a cached result is available in a cache of the load balancer;   if a cached result is available then retrieving the cached result, the cached result being the result that is returned; and   if a cached result is not available then processing result extraction to obtain at least one processed result, the processed result being the result that is returned and updating the cache with the processed result.   
     
     
         14 . The method of  claim 13 , wherein the result extraction includes:
 translating parameters of the request into data parameters suitable for determining the answer from the data store;   determining a selected one of a plurality of modes based on the request;   filling in data parameters defined for the selected mode;   removing common words;   requesting a global question identifier from a metaservice;   processing pre request blocking;   blocking of answers based on text of the request and the global question identifier;   requesting the aggregator to provide search results;   processing post request blocking;   processing results for field collapsing;   retaining a maximum of predetermined number of results for each field value;   removing duplicate results in the form of question and answer pairs that have exactly the same question and answer; and   normalizing scores of the results to a common scale.   
     
     
         15 . The method of  claim 13 , further comprising:
 processing post request blocking if the cached result is available.   
     
     
         16 . The method of  claim 10 , further comprising:
 retrieving, with a crawler of the offline system that connects over the Internet to remote computer systems, data that is placed in the file system.   
     
     
         17 . The method of  claim 16 , further comprising:
 retrieving the content from a crawl database of a batch update crawl cluster within a file system of the batch update crawl cluster, the content being retrieved with a map reducer of the batch update crawl cluster within the index controller, the map reducer of the batch update crawl cluster having a reducer core with a plurality of slow queues that retrieve the content from the crawl database, and a reducer adapter that writes an output of the reducer core into the hierarchical database; and   retrieving the content from a crawl database of a fast update crawl cluster within a file system of the fast update crawl cluster, the content being retrieved with a map reducer of the fast update crawl cluster within the index controller, the map reducer of the fast update crawl cluster having a reducer core with a plurality of slow queues that retrieve the content from the crawl database at a faster frequency than the slow queues, and a reducer adapter that writes an output of the reducer core into the hierarchical database.