Wide spreading data storage architecture

Abstract

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

I/we claim: 
     
         1 . A computer-implemented method comprising:
 receiving, at a storage management computer node of a storage management system, a write request including a data object;   encoding, by the storage management computer node, the data object to generate a first specified number of multiple encoded data fragments, the encoded data fragments representing the data object, the first specified number of encoded data fragments including a second specified number of the encoded data fragments using which the data object can be regenerated;   determining, by the storage management computer node, a storage layout of the encoded data fragments for storing the encoded data fragments at multiple storage devices, the storage devices grouped into multiple storage shelves, wherein a number of the storage devices is equal to or greater than the first specified number of the encoded data fragments; and   transmitting, by the storage management computer node and based on the storage layout, to at least a subset of the storage shelves to store the encoded data fragments at the storage devices.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein encoding the data object to generate the first specified number of the encoded data fragments includes generating the encoded data fragments based on a specified ratio of the first specified number to the second specified number. 
     
     
         3 . The computer-implemented method of  claim 2 , wherein the specified ratio is a function of a specified storage resiliency, the specified storage resiliency indicating resistance to at least one of a failure of a specified number of the storage devices or a loss of a specified number of the encoded data fragments without losing the data object. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein encoding the data object includes:
 associating an object identifier with the data object,   associating fragment identifiers with the encoded data fragments of the data object, and   generating a mapping of the fragment identifiers to the object identifier.   
     
     
         5 . The computer-implemented method of  claim 4 , wherein the object identifier and the fragment identifiers are stored in different namespaces of the storage management computer node. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the encoding includes selecting a fragmentation technique to fragment the data object. 
     
     
         7 . The computer-implemented method of  claim 6 , wherein selecting the fragmentation technique includes selecting the fragmentation technique based on at least one of a deduplication binning requirement or an erasure coding requirement. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein determining the storage layout includes determining the storage layout based on an attribute of the write request. 
     
     
         9 . The computer-implemented method of  claim 8 , wherein the attribute of the write request includes a service level objective (SLO) of the write request. 
     
     
         10 . The computer-implemented method of  claim 8 , wherein the attribute of the write request includes a specified storage resiliency, the specified storage resiliency indicating tolerance to failure of a specified number of the storage devices. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein the storage layout includes a first identification information of the storage shelves at which each of the encoded data fragments are stored. 
     
     
         12 . The computer-implemented method of  claim 11 , wherein the storage layout includes a second identification information of a storage device within a storage shelf of the storage shelves at which each of the encoded data fragments is stored. 
     
     
         13 . The computer-implemented method of  claim 1 , wherein determining the storage layout includes determining the storage devices at which the encoded data fragments are to be stored on a random basis. 
     
     
         14 . The computer-implemented method of  claim 1 , wherein determining the storage layout includes determining the storage devices at which the encoded data fragments are to be stored on a random weighted basis. 
     
     
         15 . The computer-implemented method of  claim 14 , wherein the random weighted basis randomly identifies the storage devices at which the encoded data fragments are to be stored as a function of an available storage capacity at the storage devices. 
     
     
         16 . The computer-implemented method of  claim 14 , wherein determining the storage layout on the random weighted basis includes:
 determining that a first storage device of the storage devices has higher available storage capacity than a second storage device of the storage devices, and   storing data at the first storage device at a higher rate than at the second storage device.   
     
     
         17 . The computer-implemented method of  claim 14 , wherein the random weighted basis distributes the encoded data fragments across the storage devices evenly. 
     
     
         18 . The computer-implemented method of  claim 14 , wherein the random weighted basis randomly identifies the storage devices at which the encoded data fragments are to be stored as a function of decreasing the risk of data loss. 
     
     
         19 . The computer-implemented method of  claim 1 , wherein a number of the storage shelves is at least the first number of the encoded data fragments divided by a largest number of storage devices per storage shelf of the storage shelves. 
     
     
         20 . A computer-readable storage medium storing computer-executable instructions comprising:
 instructions for receiving, at a storage management computer node of a storage management system, a read request for obtaining a data object stored at a storage subsystem, the read request including an object identifier of the data object;   instructions for determining, by the storage management computer node and using the object identifier, multiple encoded data fragments of the data object, wherein the data object is stored at the storage subsystem as “N” number of the encoded data fragments, the “N” number of encoded data fragments including “K” number of the encoded data fragments using which the data object can be regenerated;   instructions for determining, by the storage management computer node, a storage layout of the encoded data fragments, the storage layout including identification information of (a) one or more of multiple storage shelves of the storage subsystem that store the encoded data fragments and (b) multiple storage devices of the storage shelves that store the encoded data fragments, wherein a number of the storage devices is equal to or greater than “N”; and   instructions for obtaining, by the storage management computer node and based on the storage layout, the encoded data fragments from the storage devices.   
     
     
         21 . The computer-readable storage medium of  claim 20  further comprising:
 instructions for decoding the encoded data fragments obtained from the storage devices to regenerate the data object; and 
 instructions for transmitting the data object from the storage management computer node in response to the request. 
 
     
     
         22 . The computer-readable storage medium of  claim 20 , wherein the “K” number of the encoded data fragments is a minimum number of encoded data fragments required to regenerate the data object. 
     
     
         23 . The computer-readable storage medium of  claim 20 , wherein the instructions for obtaining the encoded data fragments includes instructions for obtaining at least the “K” number of the encoded data fragments. 
     
     
         24 . The computer-readable storage medium of  claim 23 , wherein the instructions for obtaining the “K” number of the encoded data fragments includes:
 instructions for selecting a first “K” number of the encoded data fragments that arrive at the storage management computer node from the storage devices. 
 
     
     
         25 . The computer-readable storage medium of  claim 23 , wherein the instructions for obtaining the “K” number of the encoded data fragments includes:
 instructions for selecting a subset of the storage devices as a function of at least one of multiple attributes of a specified storage device of the storage devices, and 
 instructions for obtaining the “K” number of the encoded data fragments from the subset of the storage devices. 
 
     
     
         26 . The computer-readable storage medium of  claim 25 , wherein the attributes of the specified storage device includes a read latency of the specified storage device, a number of pending read requests at the specified storage device, or a number of pending write requests at the specified storage device. 
     
     
         27 . The computer-readable storage medium of  claim 20 , wherein the encoded data fragments are generated from the data object based on a fragmentation technique to fragment the data object. 
     
     
         28 . The computer-readable storage medium of  claim 27 , wherein the fragmentation technique is based on an erasure coding technique. 
     
     
         29 . The computer-implemented method of  claim 20 , wherein a number of the storage shelves is at least the “N” number of the encoded data fragments divided by a largest number of storage devices per storage shelf of the storage shelves. 
     
     
         30 . A system comprising:
 a processor;   a first module configured to receive a write request including a data object;   a second module configured to encode the data object to generate a first specified number of multiple encoded data fragments out of which a second specified number of the encoded data fragments are used regenerate the data object, the encoded data fragments representing the data object;   a third module configured to determine a storage layout for storing the encoded data fragments at multiple storage devices, the storage devices grouped into multiple storage shelves, wherein a number of the storage devices is equal to or greater than the first specified number of the encoded data fragments; and   a fourth module configured to transmit the encoded data fragments to at least a subset of the storage shelves based on the storage layout to store the encoded data fragments at one or more of a set of the storage devices of each of the subset of the storage shelves.   
     
     
         31 . The system of  claim 30 , wherein the second module is further configured to encode the data object as a function of erasure coding technique.