Out of Order Delivery for Data and Metadata Mirroring in a Cluster Storage System

Abstract

Described herein are a system and method for remote mirroring of data and metadata from a local node to a remote node using out-of-order delivery (OOD), while also providing data integrity at the remote node. OOD may utilize increased throughput of multiple connection paths between nodes. A mirroring layer/engine executing on the local node may receive related groups of data and metadata for storing to the remote node, each related group comprising one or more data sets and one metadata set that describes and is associated with each of the one or more data sets in the related group. The mirroring layer provides data integrity at the remote node by ensuring that the metadata set of a related group is stored to the remote node only after all the data sets in the related group are stored to the remote node, thus ensuring data consistency at the remote node.

We claim: 
     
         1 . A system for storing data and metadata from a local node to a remote node, the system comprising:
 the local node comprising a mirroring engine configured for:
 receiving a plurality of data and metadata sets from a client, the received data and metadata sets comprising at least one related group, each related group comprising one or more data sets and one metadata set that describes each of the one or more data sets in the related group; 
 storing received data and metadata sets to a data structure according to the time order the data and metadata sets are received; 
 transmitting each data set to the remote node through a network that provides multiple connection paths between the local and remote nodes that results in out-of-order delivery (OOD) of data between the local and remote nodes; 
 upon receiving a completion acknowledgement from the remote node indicating that a data or metadata set is successfully stored to the remote node, removing the data or metadata set from the data structure; and 
 transmitting each metadata set, stored in the data structure, to the remote node through the network only if the metadata set is the first set in the data structure. 
   
     
     
         2 . The system of  claim 1 , wherein:
 OOD of data between the local and remote nodes occurs when data is not received at the remote node in the same time order as it was transmitted from the local node.   
     
     
         3 . The system of  claim 1 , wherein:
 a related group comprises one or more data sets and a current metadata set that are received by the mirroring engine after a previous metadata set, that is not in the related group; and   the current metadata set in the related group is received by the mirroring engine after the one or more data sets are received by the mirroring engine.   
     
     
         4 . The system of  claim 1 , wherein:
 the mirroring engine is configured for storing data and metadata to the remote node using OOD of data while data integrity is provided at the remote node; and   data integrity at the remote node exists when, for each related group, the metadata set of the related group is stored to the remote node only after each data set in the related group is stored to the remote node.   
     
     
         5 . The system of  claim 4 , wherein:
 the mirroring engine is configured for providing data integrity at the remote node without requiring use of a processor of the remote node.   
     
     
         6 . The system of  claim 1 , wherein:
 the remote node is a predetermined failover partner node of the local node for resuming functions of the local node upon failure of the local node; and   the plurality of data and metadata sets from the client is stored to a local non-volatile storage device on the local node and stored to a remote non-volatile storage device on the remote node for failover protection.   
     
     
         7 . The system of  claim 1 , wherein data sets of different related groups are transmitted to the remote node in parallel. 
     
     
         8 . A method for storing data and metadata from a local node to a remote node, the method comprising:
 providing the local node comprising a mirroring engine configured for:
 receiving a plurality of data and metadata sets from a client, the received data and metadata sets comprising at least one related group, each related group comprising one or more data sets and one metadata set that describes each of the one or more data sets in the related group; 
 storing received data and metadata sets to a data structure according to the time order the data and metadata sets are received; 
 transmitting each data set to the remote node through a network that provides multiple connection paths between the local and remote nodes that results in out-of-order delivery (OOD) of data between the local and remote nodes; 
 upon receiving a completion acknowledgement from the remote node indicating that a data or metadata set is successfully stored to the remote node, removing the data or metadata set from the data structure; and 
 transmitting each metadata set, stored in the data structure, to the remote node through the network only if the metadata set is the first set in the data structure. 
   
     
     
         9 . The method of  claim 8 , wherein:
 OOD of data between the local and remote nodes occurs when data is not received at the remote node in the same time order as it was transmitted from the local node.   
     
     
         10 . The method of  claim 8 , wherein:
 a related group comprises one or more data sets and a current metadata set that are received by the mirroring engine after a previous metadata set, that is not in the related group; and   the current metadata set in the related group is received by the mirroring engine after the one or more data sets are received by the mirroring engine.   
     
     
         11 . The method of  claim 8 , wherein:
 the mirroring engine is configured for storing data and metadata to the remote node using OOD of data while data integrity is provided at the remote node; and   data integrity at the remote node exists when, for each related group, the metadata set of the related group is stored to the remote node only after each data set in the related group is stored to the remote node.   
     
     
         12 . The method of  claim 11 , wherein:
 the mirroring engine is configured for providing data integrity at the remote node without requiring use of a processor of the remote node.   
     
     
         13 . The method of  claim 8 , wherein:
 the remote node is a predetermined failover partner node of the local node for resuming functions of the local node upon failure of the local node; and   the plurality of data and metadata sets from the client is stored to a local non-volatile storage device on the local node and stored to a remote non-volatile storage device on the remote node for failover protection.   
     
     
         14 . The method of  claim 8 , wherein data sets of different related groups are transmitted to the remote node in parallel. 
     
     
         15 . A non-transitory computer readable medium having instructions stored thereon when executed by a processor, store data and metadata from a local node to a remote node, the non-transitory computer readable medium comprising instructions for:
 on the local node, configuring a mirroring engine for:
 receiving a plurality of data and metadata sets from a client, the received data and metadata sets comprising at least one related group, each related group comprising one or more data sets and one metadata set that describes each of the one or more data sets in the related group; 
 storing received data and metadata sets to a data structure according to the time order the data and metadata sets are received; 
 transmitting each data set to the remote node through a network that provides multiple connection paths between the local and remote nodes that results in out-of-order delivery (OOD) of data between the local and remote nodes; 
 upon receiving a completion acknowledgement from the remote node indicating that a data or metadata set is successfully stored to the remote node, removing the data or metadata set from the data structure; and 
 transmitting each metadata set, stored in the data structure, to the remote node through the network only if the metadata set is the first set in the data structure. 
   
     
     
         16 . The non-transitory computer readable medium of  claim 15 , wherein:
 OOD of data between the local and remote nodes occurs when data is not received at the remote node in the same time order as it was transmitted from the local node.   
     
     
         17 . The non-transitory computer readable medium of  claim 15 , wherein:
 a related group comprises one or more data sets and a current metadata set that are received by the mirroring engine after a previous metadata set, that is not in the related group; and   the current metadata set in the related group is received by the mirroring engine after the one or more data sets are received by the mirroring engine.   
     
     
         18 . The non-transitory computer readable medium of  claim 15 , wherein:
 the mirroring engine is configured for storing data and metadata to the remote node using OOD of data while data integrity is provided at the remote node; and   data integrity at the remote node exists when, for each related group, the metadata set of the related group is stored to the remote node only after each data set in the related group is stored to the remote node.   
     
     
         19 . The non-transitory computer readable medium of  claim 18 , wherein:
 the mirroring engine is configured for providing data integrity at the remote node without requiring use of a processor of the remote node.   
     
     
         20 . The non-transitory computer readable medium of  claim 15 , wherein:
 the remote node is a predetermined failover partner node of the local node for resuming functions of the local node upon failure of the local node; and   the plurality of data and metadata sets from the client is stored to a local non-volatile storage device on the local node and stored to a remote non-volatile storage device on the remote node for failover protection.   
     
     
         21 . The non-transitory computer readable medium of  claim 15 , wherein data sets of different related groups are transmitted to the remote node in parallel.