Method and system for data indexing and reporting

Abstract

The present teaching relates to system, method, and programming for indexing user records. A plurality of user records are obtained, wherein each user record is associated with a user identifier and one or more data-segments. For each data-segment, one or more user identifiers associated with the data-segment are determined, a relative distance of each user identifier with respect to a previous user identifier associated with the data-segment is computed, and a variable number of bits to be used to store the relative distance of each user identifier in a bit stream are determined based on a criterion associated with the relative distance. Offset values are generated for the one or more data-segments, wherein an offset value of a data-segment indicates an index of the bit stream where information of a first user identifier associated with the data-segment is stored.

We claim: 
     
         1 . A method for responding to a query, comprising:
 obtaining a query from a user;   assigning the query to an indexing cluster of a plurality of indexing clusters based on a processing workload of the indexing cluster;   obtaining data from one or more data networks based on a triggering condition or a predetermined schedule;   transforming the received data into a column-based format;   transmitting the transformed data to the indexing cluster; and   generating, by the indexing cluster, a response to the query based on the transformed data.   
     
     
         2 . The method of  claim 1 , further comprising:
 storing each column of the data in the column-based format in a binary file.   
     
     
         3 . The method of  claim 2 , further comprising:
 processing only data stored in the binary files to generate the response.   
     
     
         4 . The method of  claim 1 , wherein the data in the column-based format comprises user records, and wherein each user record is associated with one of a plurality of user identifiers and one or more of a plurality of data-segments. 
     
     
         5 . The method of  claim 4 , further comprising:
 generating an offset stream utilized to generate the response, wherein the offset stream includes offset values for the plurality of data-segments, wherein an offset value of a corresponding data-segment indicates an index of a bit stream where information of a first user identifier of the plurality of user identifiers associated with the corresponding data-segment is stored.   
     
     
         6 . The method of  claim 1 , wherein the indexing cluster includes multiple index processing units. 
     
     
         7 . The method of  claim 6 , wherein each of the index processing units processes a respective portion of the transformed data to generate the response. 
     
     
         8 . A non-transitory, computer-readable medium having information recorded thereon for advertising, wherein the information, when read by a machine, causes the machine to perform operations comprising:
 obtaining a query from a user;   assigning the query to an indexing cluster of a plurality of indexing clusters based on a processing workload of the indexing cluster;   obtaining data from one or more data networks based on a triggering condition or a predetermined schedule;   transforming the received data into a column-based format;   transmitting the transformed data to the indexing cluster; and   generating, by the indexing cluster, a response to the query based on the transformed data.   
     
     
         9 . The medium of  claim 8 , wherein the operations further comprise:
 storing each column of the data in the column-based format in a binary file.   
     
     
         10 . The medium of  claim 9 , wherein the operations further comprise:
 processing only data stored in the binary files to generate the response.   
     
     
         11 . The medium of  claim 8 , wherein the data in the column-based format comprises user records, wherein each user record is associated with one of a plurality of user identifiers and one or more of a plurality of data-segments. 
     
     
         12 . The medium of  claim 11 , wherein the operations further comprise:
 generating an offset stream utilized to generate the response, wherein the offset stream includes offset values for the plurality of data-segments, and wherein an offset value of a corresponding data-segment indicates an index of a bit stream where information of a first user identifier of the plurality of user identifiers associated with the corresponding data-segment is stored.   
     
     
         13 . The medium of  claim 8 , wherein the indexing cluster includes multiple index processing units. 
     
     
         14 . The medium of  claim 13 , wherein each of the index processing units processes a respective portion of the transformed data to generate the response. 
     
     
         15 . A system for advertising, the system comprising:
 memory storing computer program instructions; and   one or more processors that, in response to executing the computer program instructions, effectuate operations comprising:   obtaining a query from a user;   assigning the query to an indexing cluster of a plurality of indexing clusters based on a processing workload of the indexing cluster;   obtaining data from one or more data networks based on a triggering condition or a predetermined schedule;   transforming the received data into a column-based format;   transmitting the transformed data to the indexing cluster; and   generating, by the indexing cluster, a response to the query based on the transformed data.   
     
     
         16 . The system of  claim 15 , wherein the operations further comprise:
 storing each column of the data in the column-based format in a binary file.   
     
     
         17 . The system of  claim 16 , wherein the operations further comprise:
 processing only data stored in the binary files to generate the response.   
     
     
         18 . The system of  claim 15 , wherein the data in the column-based format comprises user records, wherein each user record is associated with one of a plurality of user identifiers and one or more of a plurality of data-segments. 
     
     
         19 . The system of  claim 18 , wherein the operations further comprise:
 generating an offset stream utilized to generate the response, wherein the offset stream includes offset values for the plurality of data-segments, and wherein an offset value of a corresponding data-segment indicates an index of a bit stream where information of a first user identifier of the plurality of user identifiers associated with the corresponding data-segment is stored.   
     
     
         20 . The system of  claim 15 , wherein the indexing cluster includes multiple index processing units, wherein each of the index processing units processes a respective portion of the transformed data to generate the response.