Method for using catch-up logging to time-synchronize object stores during maintenance or recovery operations

Abstract

Provided is a method of synchronizing a distributed object store including a plurality of data stores, the method including bringing a first data store of the data stores online after having been offline, choosing a second data store of the data stores, the second data store being online while the first data store was offline, reading, by the first data store, an entry from a catch-up log, and performing a data modification operation corresponding to the entry on the first data store to synchronize the plurality of data stores.

What is claimed is: 
     
       1. A method of synchronizing a distributed object store comprising a plurality of data stores, the method comprising:
 performing a first operation at a first data store and at a second data store, such that the first data store and the second data store have a first state of an object; 
 performing a second operation on the object at the second data store, such that the second data store has a second state that is different from the first state; 
 generating an entry in a loci based on the second operation, the entry indicating a key associated with the object, and indicating the second state of the second data store as a result of the second operation; 
 performing a third operation on the object at the second data store, such that the second data store has the first state; 
 bringing the first data store having the first state online; 
 reading the entry indicating the second state of the second data store; and 
 based at least in part on:
 accessing the second data store, which is based on the reading of the entry indicating the second state; and 
 determining that the second data store has the first state, 
 
 maintaining the first state of the first data store. 
 
     
     
       2. The method of  claim 1 , further comprising causing the first data store to go offline in accordance with a maintenance operation. 
     
     
       3. The method of  claim 1 , further comprising storing the log on multiple nodes of a cluster underlying the distributed object store. 
     
     
       4. The method of  claim 1 , wherein the log comprises an unordered set of entries corresponding to respective data modification operations. 
     
     
       5. The method of  claim 1 , wherein the entry indicates a type of data modification operation. 
     
     
       6. The method of  claim 1 , wherein:
 the entry comprises a delete entry; 
 the second operation comprises a delete operation; and 
 the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is contained in the second data store. 
 
     
     
       7. The method of  claim 1 , wherein:
 the entry comprises a write entry; 
 the second operation comprises a write operation; and 
 the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is absent from the second data store. 
 
     
     
       8. A distributed object store, as a non-transitory storage medium, comprising a plurality of data stores, the distributed object store being configured to be synchronized by:
 performing a first operation at a first data store and at a second data store, such that the first data store and the second data store have a first state of an object; 
 performing a second operation on the object at the second data store, such that the second data store has a second state that is different from the first state; 
 generating an entry in a loci based on the second operation, the entry indicating a key associated with the object, and indicating the second state of the second data store as a result of the second operation; 
 performing a third operation on the object at the second data store, such that the second data store has the first state; 
 bringing the first data store having the first state online; 
 reading the entry indicating the second state of the second data store; and 
 based at least in part on:
 accessing the second data store, which is based on the reading of the entry indicating the second state; and 
 determining that the second data store has the first state, 
 
 maintaining the first state of the first data store. 
 
     
     
       9. The distributed object store of  claim 8 , wherein the distributed object store is further configured to be synchronized by causing the first data store to go offline in accordance with a maintenance operation. 
     
     
       10. The distributed object store of  claim 8 , wherein the distributed object store is further configured to be synchronized by storing the log on multiple nodes of a cluster underlying the distributed object store. 
     
     
       11. The distributed object store of  claim 8 , wherein the log comprises an unordered set of entries corresponding to respective data modification operations. 
     
     
       12. The distributed object store of  claim 8 , wherein the entry indicates a type of data modification operation. 
     
     
       13. The distributed object store of  claim 8 , wherein:
 the entry comprises a delete entry; 
 the second operation comprises a delete operation; and 
 the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is contained in the second data store. 
 
     
     
       14. The distributed object store of  claim 8 , wherein:
 the entry comprises a write entry 
 the second operation comprises a write operation; and 
 the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is absent from the second data store. 
 
     
     
       15. A distributed object store system comprising a non-transitory computer readable medium having computer code that, when executed on a processor, implements a method of synchronizing a distributed object store comprising a plurality of data stores, the method comprising:
 performing a first operation at a first data store and at a second data store, such that the first data store and the second data store have a first state of an object; 
 performing a second operation on the object at the second data store, such that the second data store has a second state that is different from the first state; 
 generating an entry in a loci based on the second operation, the entry indicating a key associated with the object, and indicating the second state of the second data store as a result of the second operation; 
 performing a third operation on the object at the second data store, such that the second data store has the first state; 
 bringing the first data store having the first state online; 
 reading the entry indicating the second state of the second data store; and 
 based at least in part on:
 accessing the second data store, which is based on the reading of the entry indicating the second state; and 
 determining that the second data store has the first state, 
 
 maintaining the first state of the first data store. 
 
     
     
       16. The distributed object store system of  claim 15 , wherein the computer code, when executed by the processor, further implements the method of synchronizing the distributed object store by causing the first data store to go offline in accordance with a maintenance operation. 
     
     
       17. The distributed object store system of  claim 15 , wherein the computer code, when executed by the processor, further implements the method of synchronizing the distributed object store by storing the log on multiple nodes of a cluster underlying the distributed object store. 
     
     
       18. The distributed object store system of  claim 15 , wherein the log comprises an unordered set of entries corresponding to respective data modification operations. 
     
     
       19. The distributed object store system of  claim 15 , wherein the entry indicates a type of data modification operation. 
     
     
       20. The distributed object store system of  claim 15 , wherein:
 the entry comprises a delete entry 
 the second operation comprises a delete operation; and 
 the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is contained in the second data store.