Erasure coding and redundant replication

Abstract

Disclosed are various embodiments for employing an erasure coding storage scheme and a redundant replication storage scheme in a data storage system. Data objects that are greater than a size threshold and accessed less frequently than an access threshold are stored in an erasure coding scheme, while data objects that are sized less than a size threshold or accessed more often than an access threshold are stored in a redundant replication storage scheme.

Therefore, the following is claimed: 
     
       1. A non-transitory computer-readable medium embodying a program executable in a at least one computing device, wherein the programcomprising, when executed, causes the at least one computing device to at least:
 code that generatesgenerate an object size distribution of a plurality of data objects stored in a data storage system, the data storage system comprising at least one data store; 
 periodically determine a size threshold to maintain a balance between data objects stored in a first data replication scheme and a second data replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution; 
 code that generatesgenerate an access pattern distribution of the plurality of data objects; 
 periodically determine an access frequency threshold to maintain the balance between the data objects stored in the first data replication scheme and the second data replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the access frequency threshold in the access pattern distribution; 
 code that identifies from the object size distributionidentify a first at least one object stored in the data storage system that is greater than athe size threshold, the first at least one object stored in athe first data replication scheme, the first data replication scheme comprising a redundant replication scheme wherein a copy of the first at least one object is stored in a plurality of data stores in the data storage system; 
 code that identifies from the access pattern distributionidentify whether the first at least one object is accessed less often than anthe access frequency thresholdfrequency; 
 code that storesstore the first at least one object in athe second data replication scheme in the data storage system when the first at least one object exceeds the size threshold and the first at least one object is accessed less often than the access frequency threshold frequency over a period of time, the second data replication scheme comprising an erasure coding scheme, wherein the at first least one data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the first at least one object and stored in a respective plurality of data stores in the data storage system; 
 code that identifies from the object size distributionidentify a second at least one object stored in the data storage system that is less than the size threshold, the second at least one object stored in the second data replication scheme; 
 code that identifies from the access pattern distributionidentify whether the second at least one object is accessed more often than the access frequency thresholdfrequency; and 
 code that storesstore the second at least one object in the first data replication scheme in the data storage system when the firstsecond at least one object is either less than the size threshold or the second at least one object is accessed more often than the access frequency threshold frequency over a period of time. 
 
     
     
       2. A system, comprising:
 at least one computing device; and 
 at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, wherein the data storage applicationcomprising, when executed, causes the at least one computing device to at least:
 logic that generatesgenerate an object size distribution of a plurality of data objects stored in a data storage system, the data storage system comprising at least one data store;   periodically determine a size threshold to maintain a balance between data objects stored in a first data replication scheme and a second data replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution;   logic that generatesgenerate an access pattern distribution of the plurality of data objects;   periodically determine an access frequency threshold to maintain the balance between the data objects stored in the first data replication scheme and the second data replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the access frequency threshold in the access pattern distribution;   logic that identifies from the object size distributionidentify at least one data object stored in the data storage system that is greater than athe size threshold, the at least one data object stored in athe first data replication scheme, the first data replication scheme comprising a redundant replication scheme wherein a copy of the at least one data object is stored in a plurality of data stores in the data storage system;   logic that identifies from the access pattern distributionidentify whether the at least one data object is accessed less often than anthe access frequency thresholdfrequency; and   logic that storesstore the at least one data object in athe second replication scheme in the data storage system when the at least one data object exceeds the size threshold and the at least one data object is accessed less often than the access frequency threshold frequency over a period of time, the second replication scheme comprising an erasure coding scheme, wherein the at least one data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the at least one data object and stored in a respective plurality of data stores in the data storage system.   
 
     
     
       3. The system of  claim 2 , wherein the plurality of shards have a total size greater than or equal the at least one data object. 
     
     
       4. The system of  claim 2 , wherein the data storage application, when executed, further comprises logic that stores causes the at least one computing device to at least store one of the plurality of shards at least a subset of the at least one data store. 
     
     
       5. The system of  claim 2 , wherein the data storage application, when executed, further comprises causes the at least one computing device to at least:
 logic that identifiesidentify a location of a subset of the shards in an index accessible to the data storage application; 
 logic that retrievesretrieve the subset of the shards from the at least one data store; and 
 logic that reconstructsreconstruct the at least one data object from the subset of the shards. 
 
     
     
       6. The system of  claim 2 , wherein the logic that generates the access pattern distribution of the plurality of data objects further comprises data storage application, when executed, further causes the at least one computing device to at least:
 logic that scansscan an access log of the data storage system over a specified period of time; and 
 logic that identifiesidentify at least one data object accessed within the specified period of time. 
 
     
     
       7. The system of  claim 2 , wherein the logic that generates the access pattern distribution of the plurality of data objects further comprises data storage application, when executed, further causes the at least one computing device to at least:
 logic that samplessample an access log of the data storage system over a specified period of time; and 
 logic that identifiesidentify at least one data object accessed within the specified period of time. 
 
     
     
       8. The system of  claim 2 , wherein the logic that generates the access pattern distribution of the plurality of data objects further comprises data storage application, when executed, further causes the at least one computing device to at least:
 logic that scansscan an index of a plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the objects and a most recent access of at least one of the objects; and 
 logic that identifiesidentify at least one data object accessed within a specified period of time. 
 
     
     
       9. The system of  claim 2 , wherein the data storage applicationfurther comprises, when executed, further causes the at least one computing device to at least:
 logic that receivesreceive a request to retrieve a data object from the data storage system; 
 logic that determinesdetermine whether a size of the data object is greater than the size threshold; 
 logic that storesstore the data object according to the second replication scheme when the size is greater than the size threshold; and 
 logic that storesstore the data object according to the first data replication scheme when the size is less than the size threshold. 
 
     
     
       10. The system of  claim 2 , wherein the data storage applicationfurther comprises, when executed, further causes the at least one computing device to at least:
 logic that receivesreceive a request to retrieve a data object from the data storage system; 
 logic that determinesdetermine whether the data object has been accessed during the period of time more often than the access threshold frequency; 
 logic that storesstore the data object according to the second replication scheme when the data object has been accessed less often than the access threshold frequency; and 
 logic that storesstore the data object according to the first data replication scheme when the data object has been accessed during the period of time more often than the access threshold frequency. 
 
     
     
       11. A method, comprisingthe steps of:
 receiving, in at least one computing device, a request to retrieve a data object from a data storage system comprising at least one data store; 
 logging, in the at least one computing device, the request in an access log accessible to the at least one computing device; 
 periodically determining, in the at least one computing device, a size threshold to maintain a balance between data objects stored in a first replication scheme and a second replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in an object size distribution for the data storage system; 
 determining, in the at least one computing device, whether that a size of the data object exceeds a the size threshold; 
 determining, in the at least one computing device, whether that the data object is stored in a the first replication scheme in the data storage system, the first replication scheme comprising a redundant replication scheme wherein a copy of the data object is stored in a plurality of data stores in the data storage system; and 
 encoding, in the at least one computing device, the data object in a the second replication scheme when in response to determining that the size exceeds the size threshold, the second replication scheme comprising an erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the data object and stored in a respective plurality of data stores in the data storage system; and 
 storing, in the data storage system, the data object in the second replication scheme. 
 
     
     
       12. The method of  claim 11 , further comprising the steps of:
 determining, in the at least one computing device, whether the size of the data object is less than the size threshold;   determining, in the at least one computing device, whether the data object is stored in the second replication scheme in the data storage system;   encoding, in the at least one computing device, the data object in the first replication scheme when the size is less than the size threshold; and   storing, in the data storage system, the data object in the first replication scheme.   
     
     
       13. The method of  claim 11 , further comprising the steps of:
 determining, in the at least one computing device, whether the data object has been accessed less often than an access threshold over a period of time; and   storing, in the data storage system, the data object in the second replication scheme if the data object has been accessed less often than the access threshold and the size exceeds the size threshold.   
     
     
       14. The method of  claim 11 , wherein the step of encoding the data object in the second replication scheme further comprises further comprising:
 dividing the data object into M fragments; 
 generating N fragments from the M fragments; and 
 storing the N fragments and the M fragments in the at least one data store. 
 
     
     
       15. The method of  claim 14 , wherein the step of encoding the data object in the second replication scheme further comprises generating an index describing a location of the N fragments and the M fragments corresponding to the data object. 
     
     
       16. The method of  claim 14 , further comprising the step of reconstructing, in the at least one computing device, the data object using an erasure coding algorithm, the data object reconstructed using a first number of the N fragments and the M fragments, wherein the first number is at least equal to M. 
     
     
       17. The method of  claim 11 , wherein the step of encoding the data object in the second replication scheme further comprises further comprising generating, in the at least one computing device, N fragments from the data object, a total storage size of the N fragments being greater than or equal to a size of the data object. 
     
     
       18. The method of  claim 17 , wherein the step of storing encoding the data object in the second replication scheme further comprises storing each one of the N fragments in a different one of the respective plurality of data stores in the data storage system. 
     
     
       19. The method of  claim 17 , further comprising the step of reconstructing, in the at least one computing device, the data object from a subset of the N fragments. 
     
     
       20. The method of  claim 19 , wherein the step of reconstructing the data object from the subset of the N fragments further comprises the step of retrieving a fragment from each of a subset of the respective plurality of data stores in the data storage system. 
     
     
       21. A system, comprising:
 at least one computing device; and   at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, the data storage application configured to cause the at least one computing device to at least:
 receive a request to store a data object in a data storage system, the data storage system comprising a plurality of data stores; 
 generate an object size distribution for the data storage system; 
 periodically determine a size threshold to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution; 
 determine whether an object size of the data object meets the size threshold; 
 store the data object in the data storage system using the redundant replication scheme when the object size fails to meet the size threshold, wherein the redundant replication scheme comprises a first scheme in which a copy of the data object is stored in a first at least two of the plurality of data stores; and 
 store the data object using the erasure coding scheme in the data storage system when the object size meets the size threshold, the erasure coding scheme comprising a second scheme in which the data object is divided into a plurality of shards, individual ones of the plurality of shards having a size less than the object size and stored in a second at least two of the plurality of data stores. 
   
     
     
       22. The system of claim 21, wherein the data storage application is further configured to cause the at least one computing device to at least:
 determine whether an access frequency of the data object meets an access frequency threshold;   store the data object in the data storage system using the redundant replication scheme when the access frequency meets the access frequency threshold; and   store the data object in the data storage system using the erasure coding scheme when the access frequency does not meet the access frequency threshold and the object size meets the size threshold.   
     
     
       23. The system of claim 21, wherein the plurality of shards have a total combined size that is greater than or equal to the object size. 
     
     
       24. The system of claim 21, wherein the data storage application is further configured to cause the at least one computing device to at least store one of the plurality of shards in a respective one of the second at least two of the plurality of data stores. 
     
     
       25. The system of claim 21, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify a location of a subset of the plurality of shards in an index accessible to the data storage application;   retrieve the subset of the plurality of shards; and   reconstruct the data object from the subset of the shards.   
     
     
       26. The system of claim 21, wherein the data storage application is further configured to cause the at least one computing device to at least:
 generate an access pattern distribution for the data storage system;   identify another data object stored in the data storage system having another object size that meets the size threshold, the other data object stored using the redundant replication scheme;   identify whether an access frequency associated with the other data object fails to meet an access frequency threshold; and   store the other data object using the erasure coding scheme responsive to identifying that the other object size meets the size threshold and the access frequency does not meet the access frequency threshold.   
     
     
       27. The system of claim 26, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an access log of the data storage system over a specified period of time; and   identifying at least one entry corresponding to the other data object within the access log over the specified period of time.   
     
     
       28. The system of claim 27, wherein the access log is scanned by sampling the access log over the specified period of time. 
     
     
       29. The system of claim 26, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an index of the plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the plurality of data objects and a most recent access of at least one of the data objects; and   identifying at least one entry in the index corresponding to the other data object within a specified period of time.   
     
     
       30. The system of claim 21, wherein the data storage application is further configured to cause the at least one computing device to at least:
 receive a request to retrieve another data object from the data storage system;   determine whether another size of the other data object meets the size threshold;   store the other data object using the erasure coding scheme when the other size meets the size threshold; and   store the data object using the redundant replication scheme when the other size does not meet the size threshold.   
     
     
       31. The system of claim 21, wherein the data storage application is further configured to cause the at least one computing device to at least:
 receive a request to retrieve another data object from the data storage system;   determine whether an access frequency associated with the other data object meets an access frequency threshold;   store the other data object using the erasure coding scheme when the access frequency does not meet the access frequency threshold; and   store the other data object according to the redundant replication scheme when the access frequency meets the access frequency threshold.   
     
     
       32. A method, comprising:
 obtaining, by at least one computing device, a request to store a data object in a data storage system comprising a plurality of data stores;   generating, by the at least one computing device, an object size distribution for the data storage system;   periodically determining, by the at least one computing device, a size threshold to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution;   determining, by the at least one computing device, that an object size of the data object fails to meet the size threshold;   storing, by the at least one computing device, the data object in the data storage system using the redundant replication scheme instead of the erasure coding scheme responsive to determining that the object size fails to meet the size threshold, wherein the redundant replication scheme comprises a scheme in which a copy of the data object is stored in a first at least two of the plurality of data stores; and   storing, by the at least one computing device, another data object in the data storage system using the erasure coding scheme, wherein the other data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the other data object and stored in a respective plurality of data stores in the data storage system.   
     
     
       33. The method of claim 32, further comprising:
 determining, by the at least one computing device, that an access frequency associated with the data object meets an access frequency threshold over a period of time;   determining, by the at least one computing device, that the data object is stored in the erasure coding scheme; and   in response to determining that the access frequency associated with the data object meets the access frequency threshold, the data object is stored in the erasure coding scheme, and the object size meets the size threshold:
 encoding, by the at least one computing device, the data object using the redundant replication scheme; and 
 storing, by the at least one computing device, the data object using the redundant replication scheme in the data storage system. 
   
     
     
       34. A system, comprising:
 at least one computing device; and   at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, the data storage application configured to cause the at least one computing device to at least:
 generate an access pattern distribution of a plurality of data objects stored in a data storage system, the data storage system comprising at least one data store; 
 periodically determine an access frequency threshold to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system based at least in part on a number of data objects distributed above or below the access frequency threshold in the access pattern distribution; 
 identify a data object stored in the data storage system having an access frequency that fails to meet the access frequency threshold, the data object stored in the data storage system using the redundant replication scheme wherein a copy of the data object is stored in a plurality of data stores in the data storage system; and 
 store the data object in the data storage system in the erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size of less than an object size of the data object, each of the plurality of shards being stored in a respective plurality of data stores in the data storage system. 
   
     
     
       35. The system of claim 34, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify another data object from the access pattern distribution that is stored in the data storage system having another access frequency that meets the access frequency threshold, the other data object stored in the data storage system using the erasure coding scheme; and   store the other data object in the data storage system using the redundant replication scheme.   
     
     
       36. The system of claim 34, wherein the plurality of shards have a total size greater than or equal the data object. 
     
     
       37. The system of claim 34, wherein the data storage application is further configured to cause the at least one computing device to at least store one of the plurality of shards at least a subset of the at least one data store. 
     
     
       38. The system of claim 34, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify a location of a subset of the shards in an index accessible to the data storage application;   retrieve the subset of the shards from the at least one data store; and   reconstruct the data object from the subset of the shards.   
     
     
       39. The system of claim 34, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an access log of the data storage system over a specified period of time; and   identifying the data object accessed within the specified period of time.   
     
     
       40. The system of claim 34, wherein the access pattern distribution of the plurality of data objects is generated by:
 sampling an access log of the data storage system over a specified period of time; and   identifying the data object accessed within the specified period of time.   
     
     
       41. The system of claim 34, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an index of a plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the data objects and a most recent access of at least one of the data objects; and   identifying the data object accessed within a specified period of time.   
     
     
       42. A method comprising:
 obtaining, in at least one computing device, a request to retrieve a data object from a data storage system comprising at least one data store;   logging, in the at least one computing device, the request in an access log accessible to the at least one computing device;   determining, in the at least one computing device, that an access frequency of the data object fails to meet an access frequency threshold;   determining, in the at least one computing device, that the data object is stored in a first replication scheme in the data storage system, the first replication scheme comprising a redundant replication scheme wherein a copy of the data object is stored in a plurality of data stores in the data storage system;   generating, by the at least one computing device, an object size distribution for the data storage system;   periodically determining, by the at least one computing device, a size threshold to maintain a balance between data objects stored in the first replication scheme and a second replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution; and   encoding, in the at least one computing device, the data object in the second replication scheme responsive to determining that the access frequency fails to meet the access frequency threshold and that a size of the data object meets the size threshold, the second replication scheme comprising an erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the data object and stored in a respective plurality of data stores in the data storage system.   
     
     
       43. The method of claim 42, wherein encoding the data object in the second replication scheme further comprises:
 dividing the data object into M fragments;   generating N fragments from the M fragments; and   storing the N fragments and the M fragments in the at least one data store.   
     
     
       44. The method of claim 43, wherein the step of encoding the data object in the second replication scheme further comprises generating an index describing a location of the N fragments and the M fragments corresponding to the data object. 
     
     
       45. The method of claim 43, further comprising reconstructing the data object using an erasure coding algorithm, the data object reconstructed using a first number of the N fragments and the M fragments, wherein the first number is at least equal to M. 
     
     
       46. The method of claim 42, wherein encoding the data object in the second replication scheme further comprises generating N fragments from the data object, a total storage size of the N fragments being greater than or equal to a size of the data object. 
     
     
       47. The method of claim 46, wherein encoding the data object in the second replication scheme further comprises storing each one of the N fragments in a different one of the respective plurality of data stores in the data storage system. 
     
     
       48. A system, comprising:
 at least one computing device; and   at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, the data storage application configured to cause the at least one computing device to at least:
 generate an object size distribution of a plurality of data objects stored in a data storage system, the data storage system comprising at least one data store; 
 generate an access pattern distribution of the plurality of data objects; 
 periodically determine a size threshold to maintain a balance between data objects stored in a first replication scheme and a second replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution; 
 identify at least one data object stored in the data storage system that fails to meet the size threshold, the at least one data object stored in the first replication scheme, the first replication scheme comprising an erasure coding scheme in which the at least one data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the at least one data object and stored in a respective plurality of data stores in the data storage system; 
 identify that the at least one data object is accessed more often than an access frequency threshold; and 
 store the at least one data object in the second replication scheme in the data storage system responsive to determining that the at least one data object fails to meet the size threshold and the at least one data object is accessed more often than the access frequency threshold over a period of time, the second replication scheme comprising a redundant replication scheme in which a copy of the at least one data object is stored in another respective plurality of data stores in the data storage system. 
   
     
     
       49. The system of claim 48, wherein the plurality of shards have a total size greater than or equal the at least one data object. 
     
     
       50. The system of claim 48, wherein the data storage application is further configured to cause the at least one computing device to at least store at least one of the plurality of shards in the at least one data store. 
     
     
       51. The system of claim 48, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify a location of a subset of the shards in an index accessible to the data storage application;   retrieve the subset of the shards from the at least one data store; and   reconstruct the at least one data object from the subset of the shards.   
     
     
       52. The system of claim 48, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an access log of the data storage system over a specified period of time; and   identifying at least one data object accessed within the specified period of time.   
     
     
       53. The system of claim 48, wherein the access pattern distribution of the plurality of data objects is generated by:
 sampling an access log of the data storage system over a specified period of time; and   identifying at least one data object accessed within the specified period of time.   
     
     
       54. The system of claim 48, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an index of a plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the objects and a most recent access of at least one of the objects; and   identifying at least one data object accessed within a specified period of time.   
     
     
       55. The system of claim 48, wherein the data storage application is further configured to cause the at least one computing device to at least:
 receive a request to retrieve a data object from the data storage system;   determine whether a size of the data object meets the size threshold;   store the data object according to the first replication scheme responsive to determining that the size of the data object meets the size threshold and the data object was previously stored according to the second replication scheme; and   store the data object according to the second replication scheme responsive to determining that the size of the data object fails to meet the size threshold and the data object was previously stored according to the first replication scheme.   
     
     
       56. The system of claim 48, wherein the data storage application is further configured to cause the at least one computing device to at least:
 receive a request to retrieve a data object from the data storage system;   determine whether an access frequency of the data object meets the access frequency threshold;   store the data object according to the second replication scheme responsive to determining that the access frequency meets the access frequency threshold and the data object was previously stored according to the first replication scheme; and   store the data object according to the first replication scheme when the access frequency fails to meet the access frequency threshold and the data object was previously stored according to the second replication scheme.   
     
     
       57. A method comprising:
 receiving, in at least one computing device, a request to retrieve a data object from a data storage system comprising at least one data store;   logging, in the at least one computing device, the request in an access log accessible to the at least one computing device;   generating, in the at least one computing device, an object size distribution for the data storage system;   periodically determining, in the at least one computing device, a size threshold to maintain a balance between data objects stored in a first replication scheme and a second replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution;   determining, in the at least one computing device, that the object size of the data object fails to meet the size threshold;   determining, in the at least one computing device, that the data object is stored in the first replication scheme in the data storage system, the first replication scheme comprising an erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size less than the object size of the data object and stored in a respective plurality of data stores in the data storage system; and   encoding, in the at least one computing device, the data object in the second replication scheme responsive to determining that the size fails to meet the size threshold and the data object is stored according to the first replication scheme, the second replication scheme comprising a redundant replication scheme wherein a copy of the data object is stored in another respective plurality of data stores in the data storage system.   
     
     
       58. The method of claim 57, further comprising:
 determining, in the at least one computing device, whether an access frequency of the data object meets an access frequency threshold over a period of time; and   encoding, in the data storage system, the data object in the second replication scheme responsive to determining that the access frequency meets the access frequency threshold and the data object was previously stored using the first replication scheme.   
     
     
       59. A non-transitory computer-readable medium embodying a program executable by at least one computing device, the program, when executed, configured to cause at least one computing device to at least:
 receive a request to store a data object in a data storage system, the data storage system comprising a plurality of data stores;   determine whether a size of the data object meets a size threshold that is periodically determined to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system and depends upon a number of data objects distributed above or below the size threshold in an object size distribution of a plurality of data objects stored in the data storage system;   store the data object in the data storage system using the redundant replication scheme in response to the size of the data object not meeting the size threshold, wherein the redundant replication scheme comprises a first scheme in which a copy of the data object is stored in a first at least two of the plurality of data stores; and   store the data object using the erasure coding scheme in the data storage system in response to the size of the data object meeting the size threshold, the erasure coding scheme comprising a second scheme in which the data object is divided into a plurality of shards, individual ones of the plurality of shards having a size less than an object size and stored in a second at least two of the plurality of data stores.   
     
     
       60. The non-transitory computer-readable medium of claim 59, wherein the plurality of shards have a total combined size that is greater than or equal to the object size. 
     
     
       61. The non-transitory computer-readable medium of claim 59, wherein the program is further configured to cause the at least one computing device to at least store one of the plurality of shards in a respective one of the second at least two of the plurality of data stores. 
     
     
       62. The non-transitory computer-readable medium of claim 59, wherein the program is further configured to cause the at least one computing device to at least:
 identify a location of a subset of the plurality of shards in an index accessible to the data storage application;   retrieve the subset of the plurality of shards; and   reconstruct the data object from the subset of the shards.   
     
     
       63. The non-transitory computer-readable medium of claim 59, wherein the program is further configured to cause the at least one computing device to at least:
 generate the object size distribution;   generate an access pattern distribution of the plurality of data objects;   identify another data object stored in the data storage system having another object size that meets the size threshold, the other data object stored using the redundant replication scheme;   identify, from the access pattern distribution, whether an access frequency associated with the other data object fails to meet an access frequency threshold; and   store the other data object using the erasure coding scheme when the other object size meets the size threshold and the access frequency does not meet the access frequency threshold.   
     
     
       64. The non-transitory computer-readable medium of claim 63, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an access log of the data storage system over a specified period of time; and   identifying at least one entry corresponding to the other data object within the access log over the specified period of time.   
     
     
       65. The non-transitory computer-readable medium of claim 64, wherein the access log is scanned by sampling the access log over the specified period of time. 
     
     
       66. The non-transitory computer-readable medium of claim 63, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an index of the plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the plurality of data objects and a most recent access of at least one of the data objects; and   identifying at least one entry in the index corresponding to the other data object within a specified period of time.   
     
     
       67. A system, comprising:
 at least one computing device; and   at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, the data storage application configured to cause the at least one computing device to at least:
 receive a request to store a data object in a data storage system, the data storage system comprising a plurality of data stores; 
 determine whether a size of the data object meets a size threshold that is periodically determined to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system and depends upon a number of data objects distributed above or below the size threshold in an object size distribution of a plurality of data objects stored in the data storage system; 
 store the data object in the data storage system using the redundant replication scheme in response to the size of the data object not meeting the size threshold, wherein the redundant replication scheme comprises a first scheme in which a copy of the data object is stored in a first at least two of the plurality of data stores; and 
 store the data object using the erasure coding scheme in the data storage system in response to the size of the data object meeting the size threshold, the erasure coding scheme comprising a second scheme in which the data object is divided into a plurality of shards, individual ones of the plurality of shards having a size less than an object size and stored in a second at least two of the plurality of data stores. 
   
     
     
       68. The system of claim 67, wherein the plurality of shards have a total combined size that is greater than or equal to the object size. 
     
     
       69. The system of claim 67, wherein the data storage application is further configured to cause the at least one computing device to at least store one of the plurality of shards in a respective one of the second at least two of the plurality of data stores. 
     
     
       70. The system of claim 67, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify a location of a subset of the plurality of shards in an index accessible to the data storage application;   retrieve the subset of the plurality of shards; and   reconstruct the data object from the subset of the shards.   
     
     
       71. The system of claim 67, wherein the data storage application is further configured to cause the at least one computing device to at least:
 generate the object size distribution;   generate an access pattern distribution of the plurality of data objects;   identify, from the object size distribution, another data object stored in the data storage system having another object size that meets the size threshold, the other data object stored using the redundant replication scheme;   identify, from the access pattern distribution, whether an access frequency associated with the other data object fails to meet an access frequency threshold; and   store the other data object using the erasure coding scheme when the other object size meets the size threshold and the access frequency does not meet the access frequency threshold.   
     
     
       72. The system of claim 71, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an access log of the data storage system over a specified period of time; and   identifying at least one entry corresponding to the other data object within the access log over the specified period of time.   
     
     
       73. The system of claim 72, wherein the access log is scanned by sampling the access log over the specified period of time. 
     
     
       74. The system of claim 71, wherein the access pattern distribution of the plurality of data objects is generated by:
 scanning an index of the plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the plurality of data objects and a most recent access of at least one of the data objects; and   identifying at least one entry in the index corresponding to the other data object within a specified period of time.   
     
     
       75. The system of claim 67, wherein the data storage application is further configured to cause the at least one computing device to at least:
 receive a request to retrieve another data object from the data storage system;   determine whether another size of the other data object meets the size threshold;   store the other data object using the erasure coding scheme when the other size meets the size threshold; and   store the data object using the redundant replication scheme when the other size does not meet the size threshold.   
     
     
       76. The system of claim 67, wherein the data storage application is further configured to cause the at least one computing device to at least:
 receive a request to retrieve another data object from the data storage system;   determine whether an access frequency associated with the other data object meets an access frequency threshold;   store the other data object using the erasure coding scheme when the access frequency does not meet the access frequency threshold; and   store the other data object according to the redundant replication scheme when the access frequency meets the access frequency threshold.   
     
     
       77. A method, comprising:
 obtaining, by at least one computing device, a request to store a data object in a data storage system comprising a plurality of data stores;   determining, by the at least one computing device, that a size of the data object meets a size threshold that is periodically determined to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system and depends upon a number of data objects distributed above or below the size threshold in an object size distribution of a plurality of data objects stored in the data storage system;   storing, by the at least one computing device, the data object using the erasure coding scheme in the data storage system instead of the redundant replication scheme in response to the size of the data object meeting the size threshold, the erasure coding scheme comprising a scheme in which the data object is divided into a plurality of shards, individual ones of the plurality of shards having a size less than an object size and stored in at least two of the plurality of data stores; and   storing, by the at least one computing device, another data object in the data storage system using the redundant replication scheme, wherein a copy of the other data object is stored in another at least two of the plurality of data stores in the data storage system.   
     
     
       78. The method of claim 77, further comprising:
 dividing the data object into M fragments;   generating N fragments from the M fragments; and   storing the N fragments and the M fragments in the at least two of the plurality of data stores.   
     
     
       79. The method of claim 78, further comprising reconstructing the data object using a first number of fragments selected from the N fragments and the M fragments, wherein the first number is less than N+M. 
     
     
       80. The method of claim 77, further comprising generating N fragments from the data object, wherein a total size of the N fragments is at least as great as the object size. 
     
     
       81. The method of claim 80, wherein the storing the data object in the erasure coding scheme further comprises storing individual ones of the N fragments in different ones of the at least two of the plurality of data stores in the data storage system. 
     
     
       82. The method of claim 81, further comprising reconstructing the data object from a subset of the N fragments. 
     
     
       83. A system, comprising:
 at least one computing device; and   at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, the data storage application configured to cause the at least one computing device to at least:
 perform an analysis of a data storage system to generate an object size distribution for the data storage system, the data storage system comprising at least one data store; 
 periodically determine a size threshold to maintain a balance between data objects stored in a first data replication scheme and a second data replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution; 
 identify whether a data object is stored in the data storage system in at least one of the first data replication scheme or the second data replication scheme, the first data replication scheme comprising a redundant replication scheme wherein a copy of the data object is stored in a plurality of data stores in the data storage system and the second data replication scheme comprising an erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the data object and stored in a respective plurality of data stores in the data storage system; 
 determine whether an object size of the data object meets the size threshold; and 
 store the data object in one of the first data replication scheme or the second data replication scheme in the data storage system based at least in part upon whether the data object meets the size threshold. 
   
     
     
       84. The system of claim 83, wherein the data storage application is further configured to cause the at least one computing device to store the data object in the one of the first data replication scheme or the second data replication scheme further based at least in part upon an access frequency of the data object. 
     
     
       85. The system of claim 83, wherein the data object is stored in the second data replication scheme by:
 dividing the data object into M fragments;   generating N fragments from the M fragments; and   storing the N fragments and the M fragments in the respective plurality of data stores.   
     
     
       86. The system of claim 85, wherein the data storage application is further configured to reconstruct the data object using a first number of fragments selected from the N fragments and the M fragments, wherein the first number is less than N+M. 
     
     
       87. The system of claim 83, wherein the data object is stored in the second data replication scheme by generating N fragments from the data object, wherein a total size of the N fragments is at least as great as the object size. 
     
     
       88. The system of claim 87, wherein the data object is stored in the second data replication scheme by storing individual ones of the N fragments in different ones of at least two plurality of data stores in the data storage system. 
     
     
       89. The system of claim 88, wherein the data storage application is further configured to reconstruct the data object from a subset of the N fragments. 
     
     
       90. A method, comprising:
 performing, by at least one computing device, an analysis of a data storage system to generate an object size distribution for the data storage system, the data storage system comprising at least one data store;   periodically determining, by the at least one computing device, a size threshold to maintain a balance between data objects stored in a first data replication scheme and a second data replication scheme in the data storage system based at least in part on a number of data objects distributed above or below the size threshold in the object size distribution;   identifying, by the at least one computing device, whether a data object is stored in the data storage system in at least one of the first data replication scheme or the second data replication scheme, the first data replication scheme comprising a redundant replication scheme wherein a copy of the data object is stored in a plurality of data stores in the data storage system and the second data replication scheme comprising an erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size less than an object size of the data object and stored in a respective plurality of data stores in the data storage system;   determining, by the at least one computing device, whether an object size of the data object meets the size threshold; and   storing, by the at least one computing device, the data object in one of the first data replication scheme or the second data replication scheme in the data storage system based at least in part upon whether the data object meets the size threshold.   
     
     
       91. The method of claim 90, wherein the at least one computing device stores the data object in the one of the first data replication scheme or the second data replication scheme further based at least in part upon an access frequency of the data object. 
     
     
       92. The method of claim 90, wherein the data object is stored in the second data replication scheme by:
 dividing the data object into M fragments;   generating N fragments from the M fragments; and   storing the N fragments and the M fragments in at least two data stores.   
     
     
       93. The method of claim 92, further comprising reconstructing the data object using a first number of fragments selected from the N fragments and the M fragments, wherein the first number is less than N+M. 
     
     
       94. The method of claim 90, wherein the data object is stored in the second data replication scheme by generating N fragments from the data object, wherein a total size of the N fragments is at least as great as the object size. 
     
     
       95. The method of claim 94, wherein the data object is stored in the second data replication scheme by storing individual ones of the N fragments in different ones of at least two plurality of data stores in the data storage system. 
     
     
       96. The system of claim 95, further comprising reconstructing the data object from a subset of the N fragments. 
     
     
       97. A system, comprising:
 at least one computing device; and   at least one storage device that is accessible to the at least one computing device, the at least one storage device storing a data storage application executable in the at least one computing device, the data storage application configured to cause the at least one computing device to at least:
 generate an object size distribution of a plurality of data objects stored in a data storage system, the data storage system comprising at least one data store; 
 periodically determine a size threshold to maintain a balance between data objects stored in a redundant replication scheme and an erasure coding scheme in the data storage system based at least in part upon a number of data objects distributed above or below the size threshold in the object size distribution; 
 identify a data object stored in the data storage system having an object size that meets the size threshold, the data object stored in the data storage system using the redundant replication scheme, wherein a copy of the data object is stored in a plurality of data stores in the data storage system; and 
 store the data object in the data storage system in the erasure coding scheme, wherein the data object is divided into a plurality of shards, each of the plurality of shards having a size of less than an object size of the data object, each of the plurality of shards being stored in a respective plurality of data stores in the data storage system. 
   
     
     
       98. The system of claim 97, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify another data object from the object size distribution that is stored in the data storage system having another object size that fails to meet the object size threshold, the other data object stored in the data storage system using the erasure coding scheme; and   store the other data object in the data storage system in the redundant replication scheme.   
     
     
       99. The system of claim 97, wherein the plurality of shards have a total size greater than or equal the data object. 
     
     
       100. The system of claim 97, wherein the data storage application is further configured to cause the at least one computing device to at least store one of the plurality of shards in at least a subset of the at least one data store. 
     
     
       101. The system of claim 97, wherein the data storage application is further configured to cause the at least one computing device to at least:
 identify a location of a subset of the shards in an index accessible to the data storage application;   retrieve the subset of the shards from the at least one data store; and   reconstruct the data object from the subset of the shards.   
     
     
       102. The system of claim 97, wherein the data storage application is further configured to cause the at least one computing device to at least generate an access pattern distribution of the plurality of data objects by:
 sampling an access log of the data storage system over a specified period of time; and   identifying the data object accessed within the specified period of time.   
     
     
       103. The system of claim 97, wherein the data storage application is further configured to cause the at least one computing device to at least generate an access pattern distribution of the plurality of data objects by:
 scanning an index of a plurality of data objects stored in the data storage system, the index specifying a storage location in the data storage system of the data objects and a most recent access of at least one of the data objects; and   identifying the data object accessed within a specified period of time.   
     
     
       104. The system of claim 97, wherein the data storage application is further configured to cause the at least one computing device to at least:
 determine whether an access frequency of the data object meets an access frequency threshold;   store the data object according to the erasure coding scheme when the object size meets the size threshold and the access frequency fails to meet the access frequency threshold; and   store the data object according to the redundant replication scheme when the access frequency meets the access frequency threshold.