Transfer of data from transactional data sources to partitioned databases in restartable environment

Abstract

Method, system, and computer program product for transferring data from transactional data sources to partitioned databases are provided. One or more messages from a transactional data source are read. Each message includes one or more records to be inserted into a database comprising a plurality of partitions. One of the partitions of the database in which the one or more records of each message are to be inserted is then identified. A transaction is initiated to insert the one or more records of each message into the one partition and to delete the one or more messages from the transactional data source. The transaction is committed only when insertion of the one or more records of each message into the one partition and deletion of the one or more messages from the transactional data source succeed.

What is claimed is: 
     
         1 . A system for transferring data from transactional data sources to partitioned database, the method comprising a feed handler reading a first set of one or more messages from a transactional data source, each message in the first set comprising one or more records to be inserted into a database comprising a plurality of partitions a sorter in communication with the feed handler, the sorter identifying a first of the plurality of partitions of the database in which to insert the one or more records of each message in the first set; and a first flusher in communication with the sorter and the first partition, the first flusher initiating a first transaction to insert the one or more records of each message in the first set into the first partition and to delete the one or more messages in the first set from the transactional data source, and committing the first transaction only when insertion of the one or more records of each message in the first set into the first partition and deletion of the one or more messages in the first set from the transactional data source succeed. 
     
     
         2 . The system of  claim 1 , wherein the first flusher further rolls back the first transaction when insertion of the one or more records of each message in the first set into the first partition or deletion of the one or more messages in the first set from the transactional data source fails. 
     
     
         3 . The system of  claim 1 , wherein the feed handler further reads a second set of one or more messages from the transactional data source, each message in the second set comprising one or more records to be inserted into the database, the sorter further identifies a second of the plurality of partitions of the database in which to insert the one or more records of each message in the second set, and the system further comprises a second flusher in communication with the sorter and the second partition, the second flusher initiating a second transaction to insert the one or more records of each message in the second set into the second partition and to delete the one or more messages in the second set from the transactional data source, and committing the second transaction only when insertion of the one or more records of each message in the second set into the second partition and deletion of the one or more messages in the second set from the transactional data source succeed. 
     
     
         4 . The system of  claim 3 , wherein the second flusher further rolls back the second transaction when insertion of the one or more records of each message in the second set into the second partition or deletion of the one or more messages in the second set from the transactional data source fails 
     
     
         5 . The system of  claim 3 , wherein the first transaction and the second transaction are running in parallel. 
     
     
         6 . The system of  claim 3 , wherein the second flusher is in communication with the second partition via at least one dedicated connection to the second partition. 
     
     
         7 . The system of  claim 1 , wherein the first flusher is in communication with the first partition via at least one dedicated connection to the first partition.