Rebuilding a data object using portions of the data object

Abstract

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

I/we claim: 
     
         1 . A computer-implemented method comprising:
 identifying, at a storage management computer node of a storage management system, a specified storage device, the specified storage device being one of multiple storage devices associated with the storage management system, the storage management system storing a data object of multiple data objects as a first set of encoded data fragments, the first set of encoded data fragments stored across the storage devices;   identifying, by the storage management computer node, one or more of the data objects to which multiple encoded data fragments stored at the specified storage device correspond, the identifying including identifying that a group of the encoded data fragments correspond to the data object, the group of the encoded data fragments being part of the first set of encoded data fragments; and   regenerating, by the storage management computer node, a subset of the encoded data fragments as a function of a second set of encoded fragments representing the data object, the second set of encoded fragments being a difference between the first set of encoded data fragments and the group of the encoded data fragments, the second set of encoded data fragments stored at a first set of the storage devices, the first set of the storage devices excluding the specified storage device.   
     
     
         2 . The computer-implemented method of  claim 1  further comprising:
 storing, by the storage management computer node, the regenerated subset of the encoded data fragments at a second set of the storage devices, the second set of the storage devices excluding the specified storage device. 
 
     
     
         3 . The computer-implemented method of  claim 2 , wherein the first set of the storage devices from which the storage management computer node obtains the subset of the encoded data fragments is same as the second set of the storage devices. 
     
     
         4 . The computer-implemented method of  claim 2 , wherein the first set of the storage devices from which the storage management computer node obtains the subset of the encoded data fragments is different from the second set of the storage devices. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the first set of encoded fragments is generated by encoding the data object, the first set of encoded fragments including a first specified number of encoded data fragments out of which a second specified number of encoded data fragments is required for regenerating the data object. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein regenerating the subset of the encoded data fragments includes:
 obtaining, from the first set of the storage devices, at least the second specified number of encoded fragments from the second set of encoded data fragments,   decoding, by the storage management computer node, the at least the second specified number of encoded data fragments to regenerate the subset of the encoded data fragments, and   storing, by the storage management computer node, the subset of the encoded data fragments at a second set of the storage devices, the second set of the storage devices excluding the specified storage device.   
     
     
         7 . The computer-implemented method of  claim 6 , wherein the decoding is executed as a function of an erasure coding technique. 
     
     
         8 . The computer-implemented method of  claim 5 , wherein the regenerating includes:
 determining a specified storage resiliency of the data object,   determining a current storage resiliency of the data object, and   generating the subset of the encoded data fragments corresponding to the data object if the current storage resiliency is below the specified storage resiliency by a specified value.   
     
     
         9 . The computer-implemented method of  claim 8 , wherein the specified storage resiliency is a function of the first specified number of encoded data fragments and the second specified number of encoded data fragments. 
     
     
         10 . The computer-implemented method of  claim 8 , wherein the current storage resiliency is a function of a number of encoded fragments in the second set of encoded data fragments and the second specified number of encoded data fragments. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein regenerating the group of the encoded data fragments includes regenerating the subset of the encoded fragments as a background process in the storage management computer node. 
     
     
         12 . The computer-implemented method of  claim 1 , wherein regenerating the group of the encoded data fragments includes regenerating the subset of the encoded fragments before the specified storage device is replaced with a replacement storage device. 
     
     
         13 . The computer-implemented method of  claim 12 , wherein the replacement storage device is used for storing a collection of encoded data fragments other than the subset of the encoded data fragments. 
     
     
         14 . The computer-implemented method of  claim 1  further comprising:
 detecting, by the storage management computer node, an addition of a replacement storage device, the replacement storage device replacing the specified storage device; and 
 using, by the storage management computer node, the first storage device to store a collection of encoded data fragments other than the subset of the encoded data fragments. 
 
     
     
         15 . The computer-implemented method of  claim 14 , wherein the replacement storage device has a different storage capacity from that of the specified storage device. 
     
     
         16 . The computer-implemented method of  claim 1 , wherein identifying the one or more of the data objects to which the encoded data fragments stored at the specified storage device correspond includes:
 determining, by the storage management computer node, a storage layout of the encoded data fragments, the storage layout including an identification information of the storage devices at which each of the encoded data fragments is stored.   
     
     
         17 . The computer-implemented method of  claim 16 , wherein identifying that the group of the encoded data fragments correspond to the data object includes:
 determining, by the storage management computer node, the data object based on a mapping in the storage layout, the mapping including a mapping of the data object to the subset of the encoded data fragments of the data object.   
     
     
         18 . The computer-implemented method of  claim 16  further comprising:
 updating, by the storage management computer node, the storage layout to indicate that the subset of the encoded data fragments is stored at a second set of the storage devices, the second set of the storage devices excluding the specified storage device. 
 
     
     
         19 . The computer-implemented method of  claim 1 , wherein the data object is encoded to the first set of encoded data fragments as a function of an erasure coding technique. 
     
     
         20 . The computer-implemented method of  claim 1 , wherein identifying the specified storage device includes identifying at least one of the storage devices that has failed, inaccessible or determined to fail. 
     
     
         21 . A computer-implemented method comprising:
 identifying, at a storage management computer node of a storage management system, a specified storage device of a set of storage devices associated with a storage computer node, the storage management computer node encoding a data object of multiple data objects to generate multiple encoded data segments, the storage computer node storing an encoded data segment of the encoded data segments as a set of encoded data fragments in the set of storage devices, the set of encoded data fragments including a first specified number of encoded data fragments out of which a second specified number of encoded data fragments is required for regenerating the encoded data segment,   wherein the storage computer node is one of multiple storage computer nodes, each of the storage computer nodes encoding at least one of the encoded data segments to generate a corresponding set of encoded data fragments and storing the corresponding set of encoded data fragments in a corresponding set of storage devices;   determining, using the storage computer node, an encoded data fragment of a group of encoded data fragments stored at the specified storage device, the group of encoded data fragments corresponding to one or more encoded data segments of one or more of the data objects;   identifying, by the storage computer node, the encoded data segment to which the encoded data fragment corresponds; and   generating, by the storage computer node, a replacement encoded data fragment as a function of at least the second specified number of encoded data fragments stored at one or more of a remaining set of the set of storage devices.   
     
     
         22 . The computer-implemented method of  claim 21  further comprising:
 storing, by the storage management computer node, the replacement encoded data fragment at one of the one or more of the remaining set of the set of storage devices. 
 
     
     
         23 . The computer-implemented method of  claim 21 , wherein generating the replacement encoded data fragment includes:
 obtaining, the at least the second specified number of encoded data fragments from the one or more of the remaining set of the set of storage devices,   encoding, by the storage computer node, the at least the second specified number of the encoded data fragments to generate the replacement encoded data fragment, and   storing, by the storage computer node, the replacement encoded data fragment at one of the one or more of the remaining set of the set of storage devices.   
     
     
         24 . The computer-implemented method of  claim 23 , wherein the encoding is executed as a function of an erasure coding technique. 
     
     
         25 . The computer-implemented method of  claim 21 , wherein the generating includes:
 determining a specified storage resiliency of the encoded data segment,   determining a current storage resiliency of the encoded data segment, and   generating the replacement encoded data fragment if the current storage resiliency is below the specified storage resiliency by a specified value.   
     
     
         26 . The computer-implemented method of  claim 21  further comprising:
 detecting, by the storage management computer node, a failure of the storage computer node; 
 identifying, by the storage management computer node, the data object to which the encoded data segment stored by the storage computer node corresponds; and 
 generating, by the storage management computer node, a replacement encoded data segment for the data object as a function of at least a third specified number of encoded data segments of the data object stored at a remaining set of the storage computer nodes. 
 
     
     
         27 . The computer-implemented method of  claim 26 , wherein generating the replacement encoded data segment includes:
 obtaining, from the remaining set of the storage computer nodes, at least the third specified number of encoded data segments, wherein the data object is encoded to generate a fourth specified number of encoded data segments, which includes the third specified number of encoded data segments required for generating the encoded data segment,   encoding, by the storage management computer node, the at least the third specified number of encoded data segments to generate the replacement encoded data segment, and   sending, by the storage management computer node, the replacement encoded data segment to one of the remaining set of the storage computer nodes for further storage at a first set of storage devices associated with the one of the remaining set of the storage computer nodes.   
     
     
         28 . The computer-implemented method of  claim 27 , wherein obtaining a first encoded data segment of the at least the third specified number of encoded data segments includes:
 obtaining, from a first storage computer node of the remaining set of the storage computer nodes that stores the first encoded data segment, a first set of encoded data fragments corresponding to the first encoded data segment, and   decoding the first set of encoded data fragments to generate the first encoded data segment.   
     
     
         29 . The computer-implemented method of  claim 26  further comprising:
 storing, by one of the remaining set of the storage computer nodes, the replacement encoded data segment as a first set of encoded data fragments. 
 
     
     
         30 . A system comprising:
 a processor;   a first module configured to identify a specified storage device, the specified storage device being one of multiple storage devices associated with the system, the system storing a data object of multiple data objects as a first set of encoded data fragments, the set of encoded data fragments stored across the storage devices;   a second module configured to identify one or more of the data objects to which multiple encoded data fragments stored at the specified storage device correspond, the identifying including identifying that a subset of the encoded data fragments correspond to the data object, the subset of the encoded data fragments being part of the first set of encoded data fragments; and   a third module configured to regenerate the subset of the encoded data fragments as a function of a second set of encoded fragments representing the data object, the second set of encoded fragments being a difference between the first set of encoded data fragments and the subset of the encoded data fragments, the second set of encoded data fragments stored at a first set of the storage devices, the first set of the storage devices excluding the specified storage device.   
     
     
         31 . A system comprising:
 a processor;   a first module configured to identify a specified storage device of a set of storage devices associated with a storage computer node, the system encoding a data object to generate multiple encoded data segments, the storage computer node storing an encoded data segment of the encoded data segments as a set of encoded data fragments in the set of storage devices, the set of encoded data fragments including a first specified number of encoded data fragments out of which a second specified number of encoded data fragments is required for regenerating the encoded data segment,
 wherein the storage computer node is one of multiple storage computer nodes, each of the storage computer nodes encoding at least one of the encoded data segments to generate a corresponding set of encoded data fragments and storing the corresponding set of encoded data fragments in a corresponding set of storage devices; 
   a second module configured to cause the storage computer node to determine an encoded data fragment of a group of encoded data fragments stored at the specified storage device;   a third module configured to cause the storage computer node to identify the encoded data segment to which the encoded data fragment corresponds; and   a fourth module configured to cause the storage computer node to generate a replacement encoded data fragment as a function of at least the second specified number of encoded data fragments stored at one or more of a remaining set of the set of storage devices.