Method of storing items of data

Abstract

A computer-implemented method of storing an item of data across a plurality of storage media, the method comprising the steps of: receiving an item of data to be stored; splitting the item of data into N pieces of data; generating M redundancy pieces of data, usable to rebuild the item of data; storing each of the N pieces of data and M redundancy pieces of data on separate storage media of the plurality of storage media at a same offset within a file of the respective storage medium, the files of the storage media containing the N pieces of data and the M redundancy pieces of data being associated as a file set; and storing, separately to the N pieces of data and M redundancy pieces of data, the offset and the file set.

The invention claimed is: 
     
       1. A computer-implemented method of storing an item of data across a plurality of storage media, the method comprising:
 receiving an item of data to be stored; 
 splitting the item of data into N pieces of data; 
 generating M redundancy pieces of data, usable to rebuild the item of data, using the N pieces of data; and 
 storing the N pieces of data and the M redundancy pieces of data in the plurality of storage media, wherein each of the N pieces of data and each of the M redundancy pieces of data are stored on a different storage medium of the plurality of storage media relative to other storage mediums of the plurality of storage media, wherein each storage medium of the plurality of storage media stores a file of a file set, wherein data for each file of the file set is stored at a plurality of offsets within each file of the file set, wherein the N pieces of data and the M redundancy pieces of data are stored at a first offset of the plurality of offsets, the first offset corresponding to a same position within each storage medium of the plurality of storage media. 
 
     
     
       2. The computer-implemented method of  claim 1 , further including, after receiving the item of data to be stored, of determining a value of M and N based on one or more properties of the item of data and/or one or more properties of the plurality of storage media. 
     
     
       3. The computer-implemented method of  claim 1 , wherein the value of M is based on a redundancy level. 
     
     
       4. The computer-implemented method of  claim 3 , wherein the redundancy level is based on a retention schedule associated with the item of data. 
     
     
       5. The computer-implemented method  claim 3 , further comprising:
 capturing, by a video camera, the item of data; and 
 determining whether or not the item of data includes a predetermined feature of interest, wherein the redundancy level is based on the determination of whether or not the item of data includes the predetermined feature of interest. 
 
     
     
       6. The computer-implemented method of  claim 3 , further comprising:
 capturing, by a video camera, the item of data; and 
 deriving an importance level of the item of data, wherein the redundancy level is based on the derivation of the importance level of the item of data. 
 
     
     
       7. The computer-implemented method of  claim 1 , wherein generating the M redundancy pieces of data is performed by passing the N pieces of data to an erasure coding algorithm. 
     
     
       8. The computer-implemented method of  claim 1 , including removing an expired item of data by hole punching the items of data corresponding to the expired item of data. 
     
     
       9. The computer-implemented method of  claim 8 , wherein removing the expired item of data includes compressing the hole left by the hole punching in each file of the respective file set. 
     
     
       10. A computer-implemented method of rebuilding lost pieces of an item of data, comprising the steps of:
 reading a plurality of pieces of data associated with the item of data, each piece of data of the plurality of pieces of data being associated with metadata; 
 determining, based on the metadata associated with each piece of data of the plurality of pieces of data, a number of lost or corrupted pieces of data of the item of data; 
 rebuilding the lost or corrupted pieces of data of the item of data; and 
 storing the rebuilt lost or corrupted pieces of data according to the method of  claim 1 . 
 
     
     
       11. The computer-implemented method of  claim 10 , wherein the rebuilding the lost or corrupted pieces of data of the item of data is based on one or more of:
 a determined importance of the item of data; 
 a redundancy level of the item of data; and 
 a retention policy of the item of data, wherein the retention policy is associated with a requirement time of the item of data. 
 
     
     
       12. A computer-implemented method of reading an item of data, the method comprising:
 determining an offset at which N pieces of data and M redundancy pieces of data associated with the item of data are stored in each storage medium of a plurality of storage media, wherein each of the N pieces of data and each of the M redundancy pieces of data are stored on a different storage medium of the plurality of storage media relative to other storage mediums of the plurality of storage media, wherein each storage medium of the plurality of storage media stores a file of a file set, wherein data for each file of the file set is stored at a plurality of offsets within each file of the file set, wherein the N pieces of data and the M redundancy pieces of data are stored at a first offset of the plurality of offsets, the first offset corresponding to a same position within each storage medium of the plurality of storage media; 
 reading each of the N pieces of data and M redundancy pieces of data, and thereby generating a read item of data, each read piece of data being associated with corresponding metadata; 
 determining, based on the metadata associated with each read piece of data, whether the read item of data is valid; and 
 returning the read item of data to a requester of the item of data when it is determined that the read item of data is valid. 
 
     
     
       13. The computer-implemented method of  claim 12 , further comprising:
 rebuilding any lost or corrupted pieces of the item of data when it is determined that the read item of data is invalid or incomplete. 
 
     
     
       14. The computer-implemented method of  claim 13 , wherein the rebuilding of lost or corrupted pieces of data is based on one or more of:
 a determined importance of the item of data; 
 a redundancy level of the item of data; and 
 a retention policy of the item of data, wherein the retention policy is associated with a requirement time of the item of data. 
 
     
     
       15. The computer-implemented method of  claim 13 , wherein rebuilding the lost pieces of data includes determining whether only redundant pieces of data are missing, and wherein the method includes rebuilding the lost redundancy pieces in parallel to returning the read item of data. 
     
     
       16. A system, including a network adapter, a processor, and a plurality of storage media, wherein the processor is configured to:
 receive, via the network adapter, an item of data to be stored; 
 split the data into N pieces of data; 
 generate M redundancy pieces of data, usable to rebuild the item of data, using the N pieces of data; and 
 store the N pieces of data and the M redundancy pieces of data in the plurality of storage media, wherein each of the N pieces of data and each of the M redundancy pieces of data are stored on a different storage medium of the plurality of storage media relative to other storage mediums of the plurality of storage media, wherein each storage medium of the plurality of storage media stores a file of a file set, wherein data for each file of the file set is stored at a plurality of offsets within each file of the file set, wherein the N pieces of data and the M redundancy pieces of data are stored at a first offset of the plurality of offsets, the first offset corresponding to a same position within each storage medium of the plurality of storage media. 
 
     
     
       17. The system of  claim 16 , wherein the processor is configured to, after receiving the item of data to be stored, determine a value of M and N based on one or more properties of the item of data and/or one or more properties of the plurality of storage media. 
     
     
       18. The computer-implemented method of  claim 6 , wherein the derived importance level of the item of data is determined based on a time instance of capturing the item of data by the video camera.