Distributed Storage System Data Management And Security

Abstract

A system and method for distributing data over a plurality of remote storage nodes. Data are split into segments and each segment is encoded into a number of codeword chunks. None of the codeword chunks contains any of the segments. Each codeword chunk is packaged with at least one encoding parameter and identifier, and metadata are generated for at least one file and for related segments of the at least one file. The metadata contains information to reconstruct from the segments, and information for reconstructing from corresponding packages. Further, metadata are encoded into package(s), and correspond to a respective security level and a protection against storage node failure. A plurality of packages are assigned to remote storage nodes to optimize workload distribution. Each package is transmitted to at least one respective storage node as a function iteratively accessing and retrieving the packages of metadata and file data.

1 - 20 . (canceled) 
     
     
         21 . A system for distributing data of a plurality of files over a plurality of respective remote storage nodes, the method comprising:
 one or more processors in electronic communication with non-transitory processor readable storage media,   one or more software modules comprising executable instructions stored in the storage media, wherein the one or more software modules are executable by the one or more processors and include:   a fragmentation module that configures the one or more processors to split into segments the data of the plurality of files;   an encoding module that configures the one or more processors to encode each segment into a number of codeword chunks, wherein none of the codeword chunks contains any of the segments and to package each codework chunk with at least one encoding parameter and identifier;   a configuration module that configures the one or more processors to generate metadata for at least one file of the plurality of files and metadata for related segments of the at least one file, wherein the metadata for the at least one file contains information to reconstruct the at least one file from the segments, and metadata for the related segments contains information for reconstructing the related segments from corresponding packages;   wherein the metadata is encoded into at least one package, wherein the encoding corresponds to a respective security level and a protection against storage node failure;   a load balancing module that configures the one or more processors to assign a plurality of packages to remote storage nodes, wherein the step of assigning corresponds to optimized workload distribution; including as a function of available network bandwidth;   a control module that configures the one or more processors to transmit each of the packages to at least one respective storage node, and to retrieve at least one of the plurality of files, as a function iteratively accessing and retrieving the packages of metadata and file data.   
     
     
         22 . A method for erasure coding, comprising:
 executing, by one or more processors configured to execute code stored in non-transitory processor readable media, data encoding with an error-correction code C to produce N codeword chunks, wherein the error-correction code C of length N=tn is based on 2h component codes: h outer codes of lengths b i n, 0≤i<h, and h inner codes of length t;   distributing, by the one or more processors, N codeword chunks over a set of storage nodes, wherein mapping of codeword chunks to storage nodes is optimized to balance network load;   reconstructing, by the one or more processors, data chunks from codeword chunks requested from storage nodes; and   repairing, by the one or more processors, data from erased codeword chunks that are reconstructed from other codeword chunks.   
     
     
         23 . The method of  claim 22 , wherein dimensions of outer codes and length multipliers b i  are selected to maximize a minimum distance of code C. 
     
     
         24 . The method of  claim 22 , wherein prior to encoding, data are partitioned into K information chunks and, further comprising encoding, by the one or more processors, metadata into at least one package, as multiplication of vectors having K elements of information chunks by K×N generator matrix of code C, wherein the generator matrix comprises a K×K sparse matrix, such that its inverse matrix is also sparse. 
     
     
         25 . The method of  claim 24 , wherein K×N generator matrix of code C comprises a matrix obtained by column and row permutations from the K×K block-diagonal matrix. 
     
     
         26 . The method of  claim 24 , wherein K×N generator matrix of code C comprises K×K block-diagonal matrix. 
     
     
         27 . The method of  claim 22 , wherein a codeword of code C comprises n groups of t elements, and further wherein any single erased codeword chunk within a group is repairable as a linear combination of other t−1 chunks of the same group. 
     
     
         28 . The method of  claim 22 , further comprising reconstructing erased codeword chunks by multi-stage decoding, wherein a decoding stage comprises decoding in one inner code and one outer code, and further where correction capability of employed inner codes increase with stage index and stages are terminated upon recovering of all erasures within the codeword. 
     
     
         29 . The method of  claim 22 , wherein an inner code in a subsequent stage has a higher minimum distance than an inner code employed in previous stage. 
     
     
         30 . The method of  claim 22 , wherein dimensions of outer codes k i  divided by respective length multipliers b i  constitute non-decreasing sequence, k 0 /b 0 ≤k 1 /b 1 ≤ . . . ≤k h-1 /b h-1 . 
     
     
         31 . The method of  claim 30 , wherein the outer codes are maximum distance separable codes. 
     
     
         32 . The method of  claim 31 , wherein the outer codes are Reed-Solomon codes. 
     
     
         33 . The method of  claim 22 , wherein inner codes are nested codes having a same length and maximized minimum distances. 
     
     
         34 . The method of  claim 33 , wherein inner codes are a maximum distance separable codes. 
     
     
         35 . The method of  claim 22 , wherein inner codes are binary linear block codes with maximum possible minimum distances w i  and length multipliers b i  are such that w 0 <w 1 < . . . <w h-1 . 
     
     
         36 . The method of  claim 22 , wherein updating of several information chunks, corresponding to the same s×s submatrix of the block-diagonal matrix, results in no more than N−K+s updated codeword chunks, where N is the length and K is the dimension of employed error-correction code. 
     
     
         37 . The method of  claim 22 , wherein retrieval of several information chunks, corresponding to the same s×s submatrix of the block-diagonal matrix requires s codeword chunks to be downloaded from storage nodes. 
     
     
         38 . A system for erasure coding, comprising:
 one or more processors in communication with non-transitory processor readable media, wherein the non-transitory processor readable media store instructions that, when executed by the one or more processors, causes the one or more processors to:   execute data encoding with an error-correction code C to produce N codeword chunks, wherein the error-correction code C of length N=tn, is based on 2h component codes: h outer codes of lengths b i n, 0≤i<h, and h, inner codes of length t;   distribute N codeword chunks over a set of storage nodes, wherein mapping of codeword chunks to storage nodes is optimized to balance network load;   reconstruct data chunks from codeword chunks requested from storage nodes; and   repair data from erased codeword chunks that are reconstructed from other codeword chunks.   
     
     
         39 . The system of  claim 38 , wherein dimensions of outer codes and length multipliers b i  are selected to maximize a minimum distance of code C. 
     
     
         40 . The system of  claim 38 , wherein prior to encoding, data are partitioned into K information chunks and, further comprising encoding, by the one or more processors, metadata into at least one package, as multiplication of vectors having K elements of information chunks by K×N generator matrix of code C, wherein the generator matrix comprises a K×K sparse matrix, such that its inverse matrix is also sparse.