Systems and methods of streaming data

Abstract

Systems and methods of stream technologies are disclosed. In some example embodiments, a method and system for a database stream switchover from one data center to another with notification at each component is provided.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 detecting a switchover event or a failover event for a first database host of a first data center, the first database host storing a first table of data;   in response to detecting the switchover event or the failover event, performing an orchestrated stop of a data extraction process, the orchestrated stop comprising stopping a first extraction of a data stream from the first table of data in the first database host at a specific location within the data stream;   determining an offset value indicating the specific location within the data stream; and   performing an orchestrated start of the extraction process in the second data center using the offset value, the orchestrated start comprising starting a second extraction of the data stream from the first table of data in the second database host at the specific location within the data stream using the offset value.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the data stream comprises change data of the first table, the change data comprising data in the first table that has changed. 
     
     
         3 . The computer-implemented method of  claim 2 , wherein:
 the orchestrated stop further comprises, subsequent to stopping the first extraction of the data stream from the first table of data, stopping a first last change record (LCR) minoring of the change data at the specific location within the data stream, the first LCR mirroring comprising a copying of last change data of the first table from a first LCR cluster of machines in the first data center to a second LCR cluster of machines in a second data center, the second data center having a second database host storing a second table of data; and   the orchestrated start further comprises, subsequent to starting the second extraction of the data stream from the first table of data in the second database host, starting a second LCR mirroring of the change data at the specific location within the data stream, the second LCR mirroring comprising a copying of last change data of the first table from the second LCR cluster of machines in the second data center to another LCR cluster of machines in another data center.   
     
     
         4 . The computer-implemented method of  claim 3 , further comprising:
 extracting the data stream of change data from the first table of data in the first database host;   transforming a format of the change data into a data serialization format based on a schema identification; and   storing the transformed change data in the first LCR cluster of machines.   
     
     
         5 . The computer-implemented method of  claim 3 , wherein:
 the orchestrated stop further comprises, subsequent to stopping the first LCR minoring, stopping a first merging of last change data in the first LCR cluster of machines with a record from the first table that corresponds to the data in the first table that has changed, the first merging forming a reconstructed record; and   the orchestrated stop further comprises, subsequent to starting the second LCR mirroring, starting a second merging of last change data in the second LCR cluster of machines with the reconstructed record stored in a second FCR cluster of machines in the second data center.   
     
     
         6 . The computer-implemented method of  claim 5 , wherein:
 the orchestrated stop further comprises, subsequent to stopping the first merging, stopping a first full change record (FCR) mirroring of the harvested last change data at the specific location within the data stream, the first FCR mirroring comprising a copying of the harvested last change data from a first FCR cluster of machines in the first data center to a second FCR cluster of machines in the second data center; and   subsequent to starting the second merging, starting a second FCR mirroring of the harvested last change data at the specific location within the data stream, the second FCR mirroring comprising a copying of the harvested last change data from the second FCR cluster of machines in the second data center to another FCR cluster of machines in the other data center.   
     
     
         7 . The computer-implemented method of  claim 6 , wherein the other data center is the first data center, the other LCR cluster of machines is the first LCR cluster of machines, and the other FCR cluster of machines is the first FCR duster of machines. 
     
     
         8 . The computer-implemented method of  claim 6 , wherein the other data center is a third data center, the other LCR cluster of machines is a third LCR cluster of machines, and the other FCR cluster of machines is a third FCR cluster of machines. 
     
     
         9 . The computer-implemented method of  claim 5 , wherein the first merging and the second merging are performed automatically based on a configuration. 
     
     
         10 . The computer-implemented method of  claim 3 , further comprising:
 receiving, from an application external to the first data center and the second data center, a request for data; and   transmitting the last change data from the second LCR cluster of machines in the second data center to the application.   
     
     
         11 . A system comprising:
 at least one processor; and   a non-transitory computer-readable medium storing executable instructions that, when executed, cause the at least one processor to perform operations comprising:
 detecting a switchover event or a failover event for a first database host of a first data center, the first database host storing a first table of data; 
 in response to detecting the switchover event or the failover event, performing an orchestrated stop of a data extraction process, the orchestrated stop comprising stopping a first extraction of a data stream from the first table of data in the first database host at a specific location within the data stream; 
 determining an offset value indicating the specific location within the data stream; and 
 performing an orchestrated start of the extraction process in the second data center using the offset value, the orchestrated start comprising starting a second extraction of the data stream from the first table of data in the second database host at the specific location within the data stream using the offset value. 
   
     
     
         12 . The system of  claim 11 , wherein the data stream comprises change data of the first table, the change data comprising data in the first table that has changed. 
     
     
         13 . The system of  claim 12 , wherein:
 the orchestrated stop further comprises, subsequent to stopping the first extraction of the data stream from the first table of data, stopping a first last change record (LCR) mirroring of the change data at the specific location within the data stream, the first LCR mirroring comprising a copying of last change data of the first table from a first LCR cluster of machines in the first data center to a second LCR cluster of machines in a second data center, the second data center having a second database host storing a second table of data; and   the orchestrated start further comprises, subsequent to starting the second extraction of the data stream from the first table of data in the second database host, starting a second LCR mirroring of the change data at the specific location within the data stream, the second LCR minoring comprising a copying of last change data of the first table from the second. LCR cluster of machines in the second data center to another LCR cluster of machines in another data center.   
     
     
         14 . The system of  claim 13 , wherein the operations further comprises:
 extracting the data stream of change data from the first table of data in the first database host;   transforming a format of the change data into a data serialization format based on a schema identification; and   storing the transformed change data in the first LCR cluster of machines.   
     
     
         15 . The system of  claim 13 , wherein:
 the orchestrated stop further comprises, subsequent to stopping the first LCR mirroring, stopping a first merging of last change data in the first LCR cluster of machines with a record from the first table that corresponds to the data in the first table that has changed, the first merging forming a reconstructed record; and   the orchestrated stop further comprises, subsequent to starting the second LCR mirroring, starting a second merging of last change data in the second LCR cluster of machines with the reconstructed record stored in a second FCR cluster of machines in the second data center.   
     
     
         16 . The system of  claim 15 , wherein:
 the orchestrated stop further comprises, subsequent to stopping the first merging, stopping a first full change record (FCR) mirroring of the harvested last change data at the specific location within the data stream, the first FCR mirroring comprising a copying of the harvested last change data from a first FCR cluster of machines in the first data center to a second FCR cluster of machines in the second data center; and   subsequent to starting the second merging, starting a second FCR mirroring of the harvested last change data at the specific location within the data stream, the second FCR mirroring comprising a copying of the harvested last change data from the second FCR cluster of machines in the second data center to another FCR cluster of machines in the other data center.   
     
     
         17 . The system of  claim 16 , wherein the other data center is the first data center, the other LCR cluster of machines is the first LCR cluster of machines, and the other FCR cluster of machines is the first FCR cluster of machines. 
     
     
         18 . The system of  claim 16 , wherein the other data center is a third data center, the other LCR cluster of machines is a third LCR cluster of machines, and the other FCR cluster of machines is a third FCR cluster of machines. 
     
     
         19 . The system of  claim 15 , wherein the first merging and the second merging are performed automatically based on a configuration. 
     
     
         20 . A non-transitory computer-readable medium storing executable instructions that, when executed, cause at least one processor to perform operations comprising:
 detecting a switchover event or a failover event for a first database host of a first data center, the first database host storing a first table of data;   in response to detecting the switchover event or the failover event, performing an orchestrated stop of a data extraction process, the orchestrated stop comprising stopping a first extraction of a data stream from the first table of data in the first database host at a specific location within the data stream;   determining an offset value indicating the specific location within the data stream; and   performing an orchestrated start of the extraction process in the second data center using the offset value, the orchestrated start comprising starting a second extraction of the data stream from the first table of data in the second database host at the specific location within the data stream using the offset value.