System and method for data organization, optimization and analytics

Abstract

A system and method for data organization, optimization and analytics includes a web server, thrift server, distributed processing framework, key value store, distributed file system, and relational database. The web server provides a method whereby users issue control actions and query for records via interaction with the thrift server. The thrift server is the center of coordination and communication for the system and interacts with other system elements. The key value store organizes all of the operational data for the system. The key value store runs on a highly scalable distributed system, including a distributed file system for storage of data on disk. The distributed processing framework enables data to be processed in bulk and is used to execute analytical processing on the data. The relational database holds all of the administrative data in the system. Search queries are submitted by end user and results of the search query are sent from the web server to the end user. The web server sends control actions to queue background map reduce jobs. These jobs run in the distributed processing framework and are used to write data and indexes and execute bulk analytics against the key value store.

We claim: 
     
         1 . A method for processing data sets, the method comprising the steps of:
 automatically distributing, using key-value stores, input data records across a plurality of servers, each server of the plurality of servers receiving a respective corresponding portion of the input data records; and   executing code in jobs locally on each server of the plurality of servers to process each respective corresponding portion of the input data records, wherein processing each portion of the input data records produces a set of output data records.   
     
     
         2 . The method of  claim 1 , wherein key-value stores that sort keys or use hashing to locate keys enable lookup of values by keys. 
     
     
         3 . The method of  claim 1 , further comprising the step of chaining of multiple ones of the jobs into a logical analytical workflow in which each individual job executes on the servers participating in a distributed key-value store. 
     
     
         4 . The method of  claim 3 , wherein the output data record of an intermediate job of a logical analytical workflow is written to either a key-value store or to a distributed file system. 
     
     
         5 . The method of  claim 4 , wherein the output data record of a final job of a logical analytical workflow is written to the key-value store. 
     
     
         6 . The method of  claim 3 , further comprising the step of querying in situ the input data records of each logical analytical workflow. 
     
     
         7 . The method of  claim 3 , further comprising the step of querying in situ the output data records of each logical analytical workflow. 
     
     
         8 . The method of  claim 3 , wherein the chaining of multiple ones of jobs produces a chain of jobs, each said chain of jobs comprising at least one job stage, the method further comprising the step of bundling said stages into individual jobs. 
     
     
         9 . The method of  claim 3 , wherein the bundled individual jobs comprise one of a single map stage, map and combine stages, map and reduce stages, and map, combine, and reduce stages. 
     
     
         10 . The method of  claim 9 , further comprising the steps of:
 configuring each bundled individual job to retrieve an input data record from either the key-value store or from a distributed file system storing the results of a previous job, and   configuring an output data record for each bundled individual job to go to either the distributed file system for processing by a later job or to the key-value store for storing final results.