Asynchronous replication in a distributed storage environment

Abstract

Embodiments of the present invention relate to asynchronously replicating data in a distributed computing environment. To achieve asynchronous replication, data received at a primary data store may be annotated with information, such as an identifier of the data. The annotated data may then be communicated to a secondary data store, which may then write the data and annotated information to one or more logs for eventual replay and committal at the secondary data store. The primary data store may communicate an acknowledgment of success in committing the data at the primary data store as well as of success in writing the data to the secondary data store. Additional embodiments may include committing the data at the secondary data store in response to receiving an instruction that authorizes committal of data through an identifier.

The invention claimed is: 
     
         1 . A computer-implemented system having one or more processors and instructions, that when executed cause the one or more processors to:
 receive at least one transaction in a geo-message from a primary data store at a secondary data store, the geo-message specifying a destination partition for the at least one transaction;   receive a commit ID from the primary data store, the commit ID specifying a portion of uncommitted transactions that are authorized to be committed at the secondary data store, the portion including the at least one transaction;   map the destination partition specified by the geo-message to a plurality of destination partitions at the secondary data store; and   commit the portion of the uncommitted transactions at the secondary data store based on the committing being authorized by the commit ID, the at least one transaction being committed to at least one of the plurality of destination partitions based on the mapping of the destination partition.   
     
     
         2 . The computer-implemented system of  claim 1 , wherein the mapping is based on analyzing a list of buckets, each bucket representing commit IDs that are within a partition range of a given destination partition of the bucket. 
     
     
         3 . The computer-implemented system of  claim 1 , wherein the one or more processors further update a list of buckets to represent the commit ID in a bucket of the list, each bucket representing commit IDs that are within a partition range of a given destination partition of the bucket. 
     
     
         4 . The computer-implemented system of  claim 1 , wherein the at least one transaction is committed to each of the plurality of destination partitions. 
     
     
         5 . The computer-implemented system of  claim 1 , wherein the one or more processors further split a destination partition into the plurality of destination partitions. 
     
     
         6 . The computer-implemented system of  claim 1 , wherein the one or more processors further split a destination partition into the plurality of destination partitions using ranges received from the primary data store as an indicator to an appropriate split location. 
     
     
         7 . The computer-implemented system of  claim 1 , wherein the commit ID is received at the secondary data store in a same message as the geo-message. 
     
     
         8 . The computer-implemented system of  claim 1 , wherein the commit ID is received at the secondary data store in a separate message from the geo-message. 
     
     
         9 . The computer-implemented system of  claim 1 , wherein the commit ID is an update to a previous version of the commit ID that newly authorize committal of the portion of the data at the secondary data store. 
     
     
         10 . The computer-implemented system of  claim 1 , wherein the commit ID specifies a given epoch number and a given sequence number that authorizes the secondary data store to commit uncommitted transactions up through the given epoch number and the given sequence number at the secondary data store. 
     
     
         11 . The computer-implemented system of  claim 1 , wherein the portion of uncommitted transactions are received at the secondary data store non-sequentially. 
     
     
         12 . A computer-implemented method for geo replicating data in a distributed computing environment, the method comprising:
 receiving at least one transaction in a geo-message from a primary data store at a secondary data store, the geo-message specifying a destination partition for the at least one transaction;   receiving a commit ID from the primary data store, the commit ID specifying a portion of uncommitted transactions authorized to be committed at the secondary data store, the portion including the at least one transaction;   mapping the destination partition specified by the geo-message to a plurality of destination partitions at the secondary data store; and   committing the portion of the uncommitted transactions at the secondary data store based on the committing being authorized by the commit ID, the at least one transaction being committed to at least one of the plurality of destination partitions based on the mapping of the destination partition.   
     
     
         13 . The computer-implemented method of  claim 12 , wherein the mapping is based on analyzing a list of buckets, each bucket representing commit IDs that are within a partition range of a given destination partition of the bucket. 
     
     
         14 . The computer-implemented method of  claim 12 , further comprising updating a list of buckets to represent the commit ID in a bucket of the list, each bucket representing commit IDs that are within a partition range of a given destination partition of the bucket. 
     
     
         15 . The computer-implemented method of  claim 12 , wherein the at least one transaction is committed to each of the plurality of destination partitions. 
     
     
         16 . The computer-implemented method of  claim 12 , further comprising splitting a destination partition into the plurality of destination partitions. 
     
     
         17 . One or more computer storage media devices having computer-executable instructions embodied thereon, that when executed by one or more processors and memory, cause the one or more processors and memory to perform a method for geo replicating data in a distributed computing environment, the method comprising:
 receiving at least one transaction in a geo-message from a primary data store at a secondary data store, the geo-message specifying a destination partition for the at least one transaction;   receiving a commit ID from the primary data store, the commit ID specifying a portion of uncommitted transactions the destination partition is authorized to have committed at the secondary data store, the portion including the at least one transaction;   mapping the destination partition specified by the geo-message to a plurality of destination partitions at the secondary data store based on analyzing a list of buckets, each bucket representing commit IDs that are within a partition range of a given destination partition of the bucket; and   committing the portion of the uncommitted transactions at the secondary data store based on the committing being authorized by the commit ID, the at least one transaction being committed to at least one of the plurality of destination partitions based on the mapping of the destination partition.   
     
     
         18 . The computer-implemented method of  claim 12 , further comprising updating the list of buckets to represent the commit ID in a bucket of the list. 
     
     
         19 . The computer-implemented method of  claim 12 , wherein the at least one transaction is committed to each of the plurality of destination partitions. 
     
     
         20 . The computer-implemented method of  claim 12 , further comprising splitting the destination partition into the plurality of destination partitions.