Global heterogeneous data mirroring

Abstract

A method and apparatus for data minoring are described. In one embodiment, a method for implementing country-specific data locality to cause data related to local transactions to be stored within the country in which the transactions occurred, comprises: capturing a set of transaction data associated with payment processing transactions into a first public cloud storage resource; and performing data mirroring across a heterogeneous set of cloud providers using a pipeline having a plurality of pipeline stages executed by one or more processors.

We claim: 
     
       1. A method for implementing location-specific data locality to cause data related to local transactions to be stored within the location in which the transactions occurred, the method comprising:
 capturing a set of transaction data associated with payment processing transactions into a first public cloud storage resource of a first cloud-based storage provider; 
 determining that the first public cloud storage resource of the first cloud-based storage provider resides in a second location and is not available for use in a first location different than the second location and that a target public cloud storage resource of a second cloud-based storage provider is available for use in the first location; 
 setting up a configuration file of the first public cloud storage resource, the configuration file including (1) a set of filter criteria to create a location-specific subset of data to meet regulatory requirements for a location from the set of transaction data, (2) an output destination which is a remote cloud-based storage location of the target public cloud storage resource of the second cloud-based storage in the first location, and (3) an encryption policy to encrypt the location-specific subset of transaction data; and 
 performing data mirroring across a heterogeneous set of cloud-based storage providers using a pipeline having a plurality of pipeline stages executed by one or more processors, the heterogeneous set of cloud-based storage providers including the first cloud-based storage provider and the second cloud-based storage provider, the performing data mirroring including
 filtering, at a first stage, the set of transaction data stored in the first public cloud storage resource as a regularly occurring batch job according to the configuration file of the first public cloud storage resource and writing the location-specific subset of data to the target public cloud storage resource; 
 monitoring, at a replication stage, the target public cloud storage resource and, in response to new location-specific data of the location-specific subset of data being added to the target public cloud storage resource from the first stage, automatically performing a replication task to mirror the location-specific subset of data to the remote cloud-based storage location defined in the configuration file according to the encryption policy; and 
 providing, at an examination stage, a secure communications protocol access to enable a regulatory entity to access the mirrored data and to decrypt the mirrored data at one of the remote cloud-based storage location or a download location of the regulatory entity using a key that is associated with the regulatory entity. 
 
 
     
     
       2. The method defined in  claim 1  wherein the replication stage wakes to perform the replication task in response to new data being written to the target public cloud storage resource. 
     
     
       3. The method defined in  claim 1  further comprising monitoring message queuing service events related to transactions written into the first public cloud storage resource. 
     
     
       4. The method defined in  claim 1  wherein the replication stage is operable to encrypt data prior to mirroring of the data to the remote cloud-based storage location according to the encryption policy in the configuration file for the target public cloud storage resource. 
     
     
       5. The method defined in  claim 4  further comprising performing a key exchange with the regulatory entity that has access to data in the remote cloud-based storage location, the key exchange to provide the key for decrypting data in the remote cloud-based storage location. 
     
     
       6. The method defined in  claim 5  wherein the plurality of pipeline stages comprises the examination stage to enable the secure communications protocol access by the regulatory entity to the mirrored data at the remote cloud-based storage location. 
     
     
       7. The method defined in  claim 6  wherein the secure communications protocol access is a secure FTP access via an FTP server. 
     
     
       8. The method defined in  claim 7  wherein the FTP server is operable to decrypt the data as the data is being downloaded. 
     
     
       9. The method defined in  claim 1  wherein execution of the first stage and the replication stage is driven by a cron job. 
     
     
       10. The method defined in  claim 1  wherein the replication stage is operable to perform replication by copying chucks of the data in the target public cloud storage resource in parallel to the remote cloud-based storage location. 
     
     
       11. A payment processing system to process transactions of a plurality of merchants, the payment processing system comprising:
 a network interface; 
 a memory to store instructions; 
 one or more processors coupled to the memory and the network interface to execute the stored instructions to:
 capture a set of transaction data associated with payment processing transactions into a first public cloud storage resource of a first cloud-based storage provider via the network interface; 
 determining that the first public cloud storage of the first cloud-based storage provider resource resides in a second location and is not available for use in a first location different than the second location different than the second location and that a target public cloud storage resource of a second cloud-based storage provider is available for use in the first location; 
 setting up a configuration file of the first public cloud storage resource, the configuration file including (1) a set of filter criteria to create a location-specific subset of data to meet regulatory requirements for a location from the set of transaction data, (2) an output destination which is a remote cloud-based storage location of the target public cloud storage resource of the second cloud-based storage provider in the first location, and (3) an encryption policy to encrypt the location-specific subset of transaction data; and 
 perform data mirroring across a heterogeneous set of cloud-based storage providers using a pipeline having a plurality of pipeline stages executed by one or more processors, the heterogeneous set of cloud-based storage providers including the first cloud-based storage provider and the second cloud-based storage provider, the performing data mirroring including
 filtering, at a first stage, the set of transaction data stored in the first public cloud storage resource as a regularly occurring batch job according to the configuration file of the first public cloud storage resource and writing the location-specific subset of data to the target public cloud storage resource; 
 monitoring, at a replication stage, the target public cloud storage resource and, in response to new location-specific data of the location-specific subset of data being added to the target public cloud storage resource from the first stage, automatically performing a replication task to mirror the location-specific subset of data to the remote cloud-based storage location defined in the configuration file according to the encryption policy; and 
 providing, at an examination stage, a secure communications protocol access to enable a regulatory entity to access the mirrored data at the remote cloud-based storage location and to decrypt the mirrored data at one of the remote cloud-based storage location or a download location of the regulatory entity using a key that is associated with the regulatory entity. 
 
 
 
     
     
       12. The payment processing system defined in  claim 11  wherein the replication stage wakes to perform the replication task in response to new data being written to the target public cloud storage resource. 
     
     
       13. The payment processing system defined in  claim 11  wherein the one or more processors is operable to monitor message queuing service events related to transactions written into the first public cloud storage resource. 
     
     
       14. The payment processing system defined in  claim 11  wherein the replication stage is operable to encrypt data prior to mirroring of the data to the remote cloud-based storage location according to the encryption policy in the configuration file for the target public cloud storage resource. 
     
     
       15. The payment processing system defined in  claim 14  wherein the one or more processors is operable to perform a key exchange with the regulatory entity that has access to data in the remote cloud-based storage location, the key exchange to provide the key for decrypting data in the remote cloud-based storage location. 
     
     
       16. The payment processing system defined in  claim 11  wherein the plurality of pipeline stages comprises the examination stage to enable the secure communications protocol access to the mirrored data at the remote cloud-based storage location. 
     
     
       17. The payment processing system defined in  claim 16  wherein the secure communications protocol access is a secure FTP access via an FTP server. 
     
     
       18. The payment processing system defined in  claim 17  wherein the FTP server is operable to decrypt the data as the data is being downloaded. 
     
     
       19. The payment processing system defined in  claim 11  wherein execution of the first stage and the replication stage is driven by a cron job. 
     
     
       20. The payment processing system defined in  claim 11  wherein the replication stage is operable to perform replication by copying chucks of the data in the target public cloud storage resource in parallel to the remote cloud-based storage location. 
     
     
       21. One or more non-transitory computer readable storage media having instructions stored thereupon which, when executed by a payment processing system having at least a processor and a memory therein, cause the payment processing system to perform operations comprising:
 capturing a set of transaction data associated with payment processing transactions into a first public cloud storage resource of a first cloud-based storage provider; 
 determining that the first public cloud storage resource of a first cloud-based storage provider resides in a second location and is not available for use in a first location different than the second location different than the second location and that a target public cloud storage resource of a second cloud-based storage provider is available for use in the first location; 
 in response to determining that the first public cloud storage resource of the first cloud-based storage provider is not available for use in the first location, configuring a configuration file of the first public cloud storage resource, the configuration file including (1) a set of filter criteria to create a location-specific subset of data to meet regulatory requirements for a location from the set of transaction data, (2) an output destination which is a remote cloud-based storage location of the target public cloud storage resource of the second cloud-based storage provider in the first location, and (3) an encryption policy to encrypt the location-specific subset of transaction data; and 
 performing data mirroring across a heterogeneous set of cloud-based storage providers using a pipeline having a plurality of pipeline stages executed by one or more processors, the heterogeneous set of cloud-based storage providers including the first cloud-based storage provider and the second cloud-based storage provider, the performing data mirroring including
 filtering, at a first stage, the set of transaction data stored in the first public cloud storage resource as a regularly occurring batch job according to the configuration file of the first public cloud storage resource and writing the location-specific subset of data to the target public cloud storage resource; 
 monitoring, at a replication stage, the target public cloud storage resource and, in response to new location-specific data of the location-specific subset of data being added to the target public cloud storage resource from the first stage, automatically performing a replication task to mirror the location-specific subset of data to the remote cloud-based storage location defined in the configuration file according to the encryption policy; and 
 providing, at an examination stage, a secure communications protocol access to enable a regulatory entity to access the mirrored data at the remote cloud-based storage location and to decrypt the mirrored data at one of the remote cloud-based storage location or a download location of the regulatory entity using a key that is associated with the regulatory entity. 
 
 
     
     
       22. The computer readable storage media defined in  claim 21  wherein the replication stage wakes to perform the replication task in response to new data being written to the target public cloud storage resource. 
     
     
       23. The computer readable storage media defined in  claim 21  wherein the method further comprises monitoring message queuing service events related to transactions written into the first public cloud storage resource. 
     
     
       24. The computer readable storage media defined in  claim 21  wherein the replication stage is operable to encrypt data prior to mirroring of the data to the remote cloud-based storage location according to the encryption policy in the configuration file for the target public cloud storage resource, and the method further comprises performing a key exchange with the regulatory entity that has access to data in the remote cloud-based storage location, the key exchange to provide the key for decrypting data in the remote cloud-based storage location. 
     
     
       25. The computer readable storage media defined in  claim 21  wherein execution of the first stage and the replication stage is driven by a cron job.