Data storage architecture for storing metadata with data

Abstract

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

I/we claim: 
     
         1 . A computer-implemented method comprising:
 receiving, at a storage management computer node of a storage management system, a write request including a data object and metadata of the data object from a client system, wherein the metadata includes information specified by the client system and information regarding multiple attributes of the data object;   determining, by the storage management computer node, the metadata of the data object;   encoding, by the storage management computer node, the data object to generate multiple encoded data pieces, the encoded data pieces representing the data object; and   processing, by the storage management computer node, the encoded data pieces and the metadata for storage across multiple storage devices associated with the storage management system.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein processing the encoded data pieces for storage includes:
 transmitting, by the storage management computer node, the encoded data pieces to multiple storage computer nodes associated with the storage management system for storage across the storage devices.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein transmitting the encoded data pieces to the storage computer nodes includes combining, prior to transmitting, one or more of the encoded data pieces with the metadata to generate corresponding multiple combined encoded data pieces. 
     
     
         4 . The computer-implemented method of  claim 2  further including:
 encoding, by a storage computer node of the storage computer nodes, an encoded data piece of the encoded data pieces to generate multiple encoded data sub-pieces representing the encoded data piece; 
 combining one or more of the encoded data sub-pieces with the metadata to generate corresponding multiple combined encoded data sub-pieces; and 
 storing, by the storage computer node, the combined encoded data sub-pieces across a set of the storage devices associated with the storage computer node. 
 
     
     
         5 . The computer-implemented method of  claim 4 , wherein encoding the data object to generate the encoded data pieces includes:
 generating, by the storage computer node, a first specified number of the encoded data sub-pieces, the first specified number of the encoded data sub-pieces including a second specified number of the encoded data sub-pieces using which the encoded data piece can be regenerated, wherein the first specified number and the second specified number are determined as a function of a specified storage resiliency.   
     
     
         6 . The computer-implemented method of  claim 5 , wherein the specified storage resiliency indicates a resistance to at least one of a failure of a specified number of storage devices of the set of the storage devices of the storage computer node or a loss of a specified number of the encoded data sub-pieces before the encoded data piece is lost. 
     
     
         7 . The computer-implemented method of  claim 4 , wherein combining one or more of the encoded data sub-pieces with the metadata includes prefixing the one or more of the encoded data sub-pieces with the metadata. 
     
     
         8 . The computer-implemented method of  claim 4 , wherein combining the metadata with the one or more of the encoded data sub-pieces includes providing a common storage resiliency to the metadata and the data object. 
     
     
         9 . The computer-implemented method of  claim 2 , wherein encoding the data object to generate the encoded data pieces includes generating a first specified number of the encoded data pieces, the first specified number being a function of a number of the storage computer nodes. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein processing the encoded data pieces for storage includes:
 combining one or more of the encoded data pieces with the metadata to generate corresponding multiple combined encoded data pieces, and   storing the combined encoded data pieces across the storage devices.   
     
     
         11 . The computer-implemented method of  claim 10 , wherein combining the one or more of the encoded data pieces with the metadata includes prefixing the one or more of the encoded data pieces with the metadata. 
     
     
         12 . The computer-implemented method of  claim 10 , wherein combining the one or more of the encoded data pieces with the metadata includes providing a common storage resiliency to the metadata and the data object. 
     
     
         13 . The computer-implemented method of  claim 1 , wherein encoding the data object to generate the encoded data pieces includes:
 generating a first specified number of the encoded data pieces, the first specified number of the encoded data pieces including a second specified number of the encoded data pieces using which the data object can be regenerated, wherein the first specified number and the second specified number are determined as a function of a specified storage resiliency.   
     
     
         14 . The computer-implemented method of  claim 13 , wherein the specified storage resiliency indicates a resistance to a failure of a specified number of the storage devices or a loss of specified number of the encoded data pieces before the data object is lost. 
     
     
         15 . The computer-implemented method of  claim 10 , wherein storing the combined encoded data pieces includes:
 storing the combined encoded data pieces across multiple storage shelves, each of the storage shelves storing at least a subset of the combined encoded data pieces, the storage shelf storing the subset of the combined encoded data pieces across a set of the storage devices associated with the storage shelf.   
     
     
         16 . The computer-implemented method of  claim 15 , wherein storing the combined encoded data pieces includes:
 determining, by the storage management computer node, a storage layout of the combined encoded data pieces, the storage layout including a first identification information of the storage shelves at which each of the combined encoded data pieces is stored.   
     
     
         17 . The computer-implemented method of  claim 16 , wherein the storage layout includes a second identification information of the set of the storage devices within a storage shelf of the storage shelves at which the subset of the combined encoded data pieces are stored. 
     
     
         18 . The computer-implemented method of  claim 1 , wherein the encoding includes encoding the data object as a function of an erasure coding technique. 
     
     
         19 . A computer-implemented method comprising:
 receiving, at a storage management computer node of a storage management system, a write request including a data object and metadata of the data object from a client system, wherein the metadata includes information specified by the client system and information regarding multiple attributes of the data object;   determining metadata of the data object;   encoding, by the storage management computer node, the data object to generate multiple encoded data segments, the encoded data segments representing the data object;   transmitting, by the storage management computer node, the encoded data segments to multiple storage computer nodes associated with the storage management system for storage across multiple storage devices associated with the storage management system;   causing, by the storage management computer node a storage computer node of the storage computer nodes, to encode an encoded data segment of the encoded data segments to generate multiple encoded data fragments representing the encoded data segment;   causing, by the storage management computer node the storage computer node, to combine one or more of the encoded data fragments with the metadata to generate corresponding multiple combined encoded data fragments; and   causing, by the storage management computer node the storage computer node, to store the combined encoded data fragments across a set of the storage devices associated with the storage computer node.   
     
     
         20 . The computer-implemented method of  claim 19 , wherein transmitting the encoded data segments to the storage computer nodes includes combining, prior to transmitting, one or more of the encoded data segments with the metadata to generate corresponding multiple combined encoded data segments. 
     
     
         21 . The computer-implemented method of  claim 20 , wherein combining the one or more of the encoded data segments with the metadata includes prefixing the one or more of the encoded data segments with the metadata. 
     
     
         22 . The computer-implemented method of  claim 20 , wherein causing the storage computer node to combine the metadata with the one or more of the encoded data segments includes providing a common storage resiliency to the metadata and the data object. 
     
     
         23 . The computer-implemented method of  claim 19 , wherein encoding the data object to generate the encoded data segments includes:
 generating, by the storage computer node, a first specified number of the encoded data fragments, the first specified number of the encoded data fragments including a second specified number of the encoded data fragments using which the encoded data segment can be regenerated, wherein the first specified number and the second specified number are determined as a function of a specified storage resiliency.   
     
     
         24 . The computer-implemented method of  claim 23 , wherein the specified storage resiliency indicates a resistance to at least one of a failure of a specified number of storage devices of the set of the storage devices of the storage computer node or a loss of a specified number of the encoded data fragments before the encoded data segment is lost. 
     
     
         25 . The computer-implemented method of  claim 19 , wherein causing the storage computer node to store the combined encoded data fragments includes causing the storage computer node to determine the storage devices at which the combined encoded data fragments are to be stored on a random basis. 
     
     
         26 . The computer-implemented method of  claim 19 , wherein storing the combined encoded data fragments includes determining the storage devices at which the combined encoded data fragments are to be stored on a random weighted basis. 
     
     
         27 . The computer-implemented method of  claim 26 , wherein the random weighted basis randomly identifies the storage devices at which the combined encoded data fragments are to be stored as a function of an available storage capacity of the storage devices. 
     
     
         28 . The computer-implemented method of  claim 26 , wherein the random weighted basis distributes the encoded data fragments across the storage devices evenly. 
     
     
         29 . The computer-implemented method of  claim 26 , wherein the random weighted basis randomly identifies the storage computer nodes at which the encoded data segments are to be stored as a function of decreasing the risk of data loss. 
     
     
         30 . The computer-implemented method of  claim 19 , wherein encoding the data object includes:
 associating an object identifier with the data object,   associating segment identifiers with the encoded data segments of the data object,   associating fragment identifiers with the encoded data fragments of the encoded data segment, and   generating a mapping of the segment identifiers to the object identifier and the fragment identifiers to a segment identifier of the encoded data segment, the segment identifier being one of the segment identifiers.   
     
     
         31 . A system comprising:
 a processor;   a first module configured to receive a write request including a data object and metadata of the data object from a client system, wherein the metadata includes information specified by the client system and includes information regarding multiple attributes of the data object;   a second module configured to determine metadata of the data object, wherein the metadata includes information regarding multiple attributes of the data object;   a third module configured to encode the data object to generate multiple encoded data pieces, the encoded data pieces representing the data object; and   a fourth module configured to process the encoded data pieces and the metadata for storage across multiple storage devices associated with the storage management system, the fourth module further configured to combine one or more of the encoded data pieces with the metadata to generate corresponding multiple combined encoded data pieces.   
     
     
         32 . The system of  claim 31 , wherein the fourth module is configured to process the encoded data pieces for storage by transmitting the combined encoded data pieces to multiple storage computer nodes associated with the storage management system for storage across the storage devices. 
     
     
         33 . The system of  claim 32  further comprising:
 a fifth module to cause a storage computer node of the storage computer nodes to encode an encoded data piece of the encoded data pieces to generate multiple encoded data sub-pieces representing the encoded data piece; 
 a sixth module to cause the storage computer node to combine one or more of the encoded data sub-pieces with the metadata to generate corresponding combined encoded data sub-pieces; and 
 a seventh module to cause the storage computer node to store the combined encoded data sub-pieces across a set of the storage devices associated with the storage computer node. 
 
     
     
         34 . The system of  claim 31 , wherein the fourth module is further configured to store the combined encoded data pieces across the storage devices.