Method and system for data pipeline stratification for low latency access

Abstract

A method at a data pipeline in computing system, the method including receiving data and a schema identifier associated with the data; confirming that the schema identifier is valid; and routing the data at the data pipeline based on the schema identifier. Also, a computing system having a processor; a data pipeline; and a communications subsystem, wherein the computing system is configured to: receive data and a schema identifier associated with the data; confirm, using the processor, that the schema identifier is valid; and route the data at the data pipeline based on the schema identifier.

1 . A method at a data pipeline in computing system, the method comprising:
 receiving data and a schema identifier associated with the data;   confirming that the schema identifier is valid; and   routing the data at the data pipeline based on the schema identifier.   
     
     
         2 . The method of  claim 1 , wherein the routing comprises routing the data to a secondary processing function within the data pipeline. 
     
     
         3 . The method of  claim 1 , wherein the routing comprises:
 cloning the data to create cloned data;   sending the data to a destination; and   sending the cloned data to a processing function,   
       wherein the processing function comprises a slower pathway than the sending the data to the destination. 
     
     
         4 . The method of  claim 1 , further comprising:
 receiving a new schema identifier; and   determining a data path for data associated with the new schema identifier.   
     
     
         5 . The method of  claim 4 , further comprising creating a new processing function for data associated with the new schema identifier. 
     
     
         6 . The method of  claim 1 , wherein a parallel data pipeline is created when a volume of data reaches a threshold. 
     
     
         7 . The method of  claim 2 , wherein the secondary processing function is duplicated in the data pipeline when a volume of data reaches a threshold. 
     
     
         8 . The method of  claim 1 , wherein the confirming the schema identifier is valid comprises checking that schema of the data matches schema of the schema identifier. 
     
     
         9 . The method of  claim 1 , wherein the routing the data comprises routing the data to a bus which comprises a real time data cache. 
     
     
         10 . A computing system comprising:
 a processor;   a data pipeline; and   a communications subsystem,   
       wherein the computing system is configured to:
 receive data and a schema identifier associated with the data; 
 confirm, using the processor, that the schema identifier is valid; and 
 route the data at the data pipeline based on the schema identifier. 
 
     
     
         11 . The computing system of  claim 10 , wherein the computing system is configured to route by routing the data to a secondary processing function within the data pipeline. 
     
     
         12 . The computing system of  claim 10 , wherein the computing system is configured to route by:
 cloning the data to create cloned data;   sending the data to a destination; and   sending the cloned data to a processing function,   
       wherein the processing function comprises a slower pathway than the sending the data to the destination. 
     
     
         13 . The computing system of  claim 10 , wherein the computing system is further configured to:
 receive a new schema identifier; and   determine a data path for data associated with the new schema identifier.   
     
     
         14 . The computing system of  claim 13 , wherein the computing system is further configured to create a new processing function for data associated with the new schema identifier. 
     
     
         15 . The computing system of  claim 10 , wherein a parallel data pipeline is created when a volume of data reaches a threshold. 
     
     
         16 . The computing system of  claim 11 , wherein the secondary processing function is duplicated in the data pipeline when a volume of data reaches a threshold. 
     
     
         17 . The computing system of  claim 10 , wherein the computing system is configured to confirm that the schema identifier is valid by checking that schema of the data matches schema of the schema identifier. 
     
     
         18 . The computing system of  claim 10 , wherein the computing system is configured to route the data by routing the data to a bus which comprises a real time data cache. 
     
     
         19 . A computer readable medium for storing instruction code, which, when executed by a processor in a computing system cause the computing system to:
 receive data and a schema identifier associated with the data;   confirm, using the processor, that the schema identifier is valid; and   route the data at the data pipeline based on the schema identifier.