Deferred rebuilding of a data object in a multi-storage device storage architecture

Abstract

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

I/we claim: 
     
         1 . A computer-implemented method comprising:
 obtaining, at a storage management computer node of a storage management system, historical information regarding a failure rate of a storage device, the storage device being of a type of multiple storage devices associated with the storage management system, the storage management computer node encoding a data object to generate a first specified number of encoded data segments, the encoded data segments stored at multiple storage computer nodes, the first specified number of encoded data segments including a second specified number of redundant encoded data segments, which are generated to provide a storage resiliency to the data object, the first specified number being a function of the second specified number;   determining, using the storage management computer node, predicted information regarding a failure rate of the storage devices based on the historical information;   determining, using the storage management computer node, a lifespan of the storage devices as a function of the historical information and the predicted information;   determining, by the storage management computer node, a statistical probability of a loss of a storage computer node of the storage computer nodes over the lifespan of the storage devices; and   determining, by the storage management computer node, the second specified number of the redundant encoded data segments to be generated based on the statistical probability.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the storage resiliency is a function of the second specified number of redundant encoded data segments. 
     
     
         3 . The computer-implemented method of  claim 1  further comprising:
 triggering a regeneration process to generate one or more replacement encoded data segments if the storage resiliency of the data object drops below a specified threshold. 
 
     
     
         4 . The computer-implemented method of  claim 3 , wherein the storage resiliency of the data object drops below the specified threshold due to loss of a specified number of the encoded data segments. 
     
     
         5 . The computer-implemented method of  claim 4 , wherein the loss of the specified number of the encoded data segments occurs due to a failure of one or more storage computer nodes storing the specified number of the encoded data segments. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein the loss of the specified number of the encoded data segments occurs due to a failure of one or more of the storage devices associated with the one or more storage computer nodes, the one or more storage devices storing portions of the specified number of the encoded data segments. 
     
     
         7 . The computer-implemented method of  claim 3 , wherein determining the second specified number of redundant encoded data segments includes:
 dynamically adjusting the second specified number to restrict the storage resiliency from dropping below the specified threshold during the lifespan of at least a specified number of the storage devices.   
     
     
         8 . The computer-implemented method of  claim 3 , wherein determining the second specified number of redundant encoded data segments includes:
 dynamically adjusting the second specified number to restrict the triggering of the regeneration process during the lifespan of at least a specified number of the storage devices.   
     
     
         9 . The computer-implemented method of  claim 8 , wherein the at least the specified number of the storage devices are replaced with a first set of storage devices at the expiry of the lifespan of the at least the specified number of the storage devices. 
     
     
         10 . The computer-implemented method of  claim 9  further comprising:
 triggering the regeneration process when the at least the specified number of the storage devices are replaced with the first set of storage devices. 
 
     
     
         11 . The computer-implemented method of  claim 1  further comprising:
 detecting a loss of one or more of the encoded data segments; and 
 generating one or more replacement encoded data segments using a remaining set of the encoded data segments. 
 
     
     
         12 . The computer-implemented method of  claim 11 , wherein generating the one or more replacement encoded data segments includes generating the one or more replacement encoded data segments if the storage resiliency of the data object drops below a specified threshold. 
     
     
         13 . The computer-implemented method of  claim 12 , wherein the second specified number is adjusted dynamically to keep the storage resiliency of the data object from dropping below the specified threshold during the lifespan of at least a specified number of the storage devices. 
     
     
         14 . The computer-implemented method of  claim 1  further comprising:
 causing, by the storage management computer node, the storage computer node to store an encoded data segment of the encoded data segments as a set of encoded data fragments at a set of the storage devices associated with the storage computer node. 
 
     
     
         15 . The computer-implemented method of  claim 1 , wherein the encoding of the data object to generate the encoded data segments is based on an erasure coding technique. 
     
     
         16 . A computer-readable storage medium storing computer-executable instructions comprising:
 instructions for determining a lifespan of multiple storage devices associated with a storage management system as a function of historical information and predicted information, the storage management system storing a data object as “n” number of encoded data segments at multiple storage computer nodes, the “n” number of encoded data segments including “m” number of redundant encoded data segments, which are generated to provide a storage resiliency to the data object, wherein “n” is a function of “m,” and wherein the storage resiliency of the data object is a function of “m”;   instructions for determining a statistical probability of a loss of a storage computer node of the storage computer nodes over the lifespan of the storage devices; and   instructions for determining the “m” number of the redundant encoded data segments to be generated based on the statistical probability.   
     
     
         17 . The computer-readable storage medium of  claim 16 , wherein the historical information includes information regarding a failure rate of a storage device of a type of the storage devices associated with the storage management system. 
     
     
         18 . The computer-readable storage medium of  claim 16 , wherein the predicted information includes information regarding a failure rate of the storage devices that is determined based on the historical information. 
     
     
         19 . The computer-readable storage medium of  claim 16  further comprising:
 instructions for triggering a regeneration process to generate one or more replacement encoded data segments if the storage resiliency of the data object drops below a specified threshold. 
 
     
     
         20 . The computer-readable storage medium of  claim 19 , wherein the instructions for triggering the regeneration process includes:
 instructions for adjusting the “m” to restrict the storage resiliency from dropping below the specified threshold during the lifespan of at least a specified number of the storage devices.   
     
     
         21 . The computer-readable storage medium of  claim 19 , wherein the instructions for triggering the regeneration process includes:
 instructions for adjusting the “m” to restrict the triggering of the regeneration process during the lifespan of at least a specified number of the storage devices.   
     
     
         22 . A system comprising:
 a processor;   a first module to determine a lifespan of multiple storage devices associated with a storage management system as a function of historical information and predicted information, the storage management system storing a data object as “n” number of encoded data segments at multiple storage computer nodes, the “n” number of encoded data segments including “m” number of redundant encoded data segments, which are generated to provide a storage resiliency to the data object, wherein “n” is a function of “m,” and wherein the storage resiliency of the data object is a function of “m”;   a second module to determine a statistical probability of a loss of a storage computer node of the storage computer nodes over the lifespan of the storage devices;   a third module to determine the “m” number of the redundant encoded data segments to be generated based on the statistical probability; and   a fourth module to trigger a regeneration process to generate one or more replacement encoded data segments for the data object if the storage resiliency of the data object drops below a specified threshold.   
     
     
         23 . The system of  claim 22 , wherein the historical information includes information regarding a failure rate of a storage device of a type of the storage devices associated with the storage management system. 
     
     
         24 . The system of  claim 22 , wherein the predicted information includes information regarding a failure rate of the storage devices that is determined based on the historical information. 
     
     
         25 . The system of  claim 22 , wherein the third module is further configured to adjust the “m” to restrict the storage resiliency from dropping below the specified threshold during the lifespan of at least a specified number of the storage devices. 
     
     
         26 . The system of  claim 22 , wherein the third module is further configured to adjust the “m” to restrict the fourth module from triggering the regeneration process during the lifespan of at least a specified number of the storage devices.