Distributed transactional network storage system

Abstract

The present invention provides a highly scalable system for fault tolerant distributed data file storage over a set of functionally equal network servers linked through a local network with network servers and client computers. Data files are represented as a set of transactional records, each record is disassembled into redundant, functionally identical data pieces with original file reassembly dependent only upon the number of data file pieces and not on the presence or absence of any particular data file piece. Local algorithms generate unique data file identifiers upon file creation and disassembly. Changes to the data file storage system are ranked by creation time and stored as separate records with unique transaction identifiers in addition to unique data file identifiers. A transactional data file record is stored by disassembling the transactional file into pieces placed at the network servers. Low-level transactional files are collected to reassemble the data file contents.

What is claimed is:  
     
         1 . A system of distributed file storage, comprising: 
 a local computer network with network servers and client computers;    a plurality of functionally equal network servers linked together by said local network with said network servers and said client computers and ranked according to available capacity and accessibility;    a said plurality of functionally equal network servers organized into a plurality of groups where each said server can participate in a couple of different groups and said servers inside each said group are considered neighbors;    a pre-defined set of network services supported by each of said plurality of functionally equal network servers;    a plurality of client computers utilizing said local network with network servers and client computers;    software constructed and arranged to run on a local client computer to enable distributed data file storage;    software constructed and arranged to run on a network server computer to enable distributed data file storage;    a common file namespace in the form of a tree with a shared root;    directory files and common data files within said common file namespace;    wherein accessibility to stored data does not depend upon dedicated access to any particular member of said plurality of functionally equal network servers, but rather depends only upon access to a pre-defined quantity of network servers from said plurality of functionally equal network servers.    
     
     
         2 . The system of  claim 1 , wherein said software constructed and arranged to run on a local client computer for distributed file storage traces changes to said local data file system, puts said changes into the form of transaction records, and verifies said local data file content to be consistent with the content of the files stored in said distributed data file storage in case of local data file open-and-create requests.  
     
     
         3 . The system of  claim 1 , wherein said software is constructed and arranged to run on a local client computer to be used for said distributed file storage communications with said plurality of functionally equal network servers, records transactions made into said distributed data file storage, and reads and assembles data files into the local file system of a client computer.  
     
     
         4 . A method of providing distributed file storage, comprising the steps of: 
 utilizing a local network with network servers and client computers;    establishing a plurality of functionally equal network servers;    linking said plurality of functionally equal network servers together by said local network with network servers and client computers;    ranking said plurality of functionally equal network servers according to available capacity and accessibility;    supporting a pre-defined set of network services at each of said plurality of functionally equal network servers;    establishing a plurality of client computers utilizing said local network with network servers and client computers;    utilizing software constructed and arranged to run on a local client computer for distributed data file storage;    utilizing software constructed and arranged to run on a network server computer for distributed data file storage;    establishing a common file namespace in the form of a tree with a shared root; and    establishing directory files and common files within said common file namespace;    whereby accessibility to said directory files and said common files does not require dedicated access to any particular member of said plurality of functionally equal network servers, but rather depends only upon access to a pre-defined quantity from said plurality of functionally equal network servers.    
     
     
         5 . The method of  claim 4 , further including the step of: 
 providing client file access through the highest ranked member of said plurality of functionally equal network servers with respect to available capacity and accessibility, where client could be on client computer or initiate request from network server computer.    
     
     
         6 . The method of  claim 5 , wherein the step of providing client file access further includes the steps of: 
 requesting a data file by its full name in said common file namespace from any network server;    requesting information about the availability of a plurality of file transaction record pieces necessary for file data assembling;    collecting a plurality of data file pieces for each transaction record;    checking the presence of said data file pieces in the local cache;    checking the presence of said data file pieces in neighboring servers;    sending said data file pieces from a neighboring server to a requesting network server;    sending said data file pieces from said requesting network server to at least one client computer;    assembling said data file pieces into a requested transactions file on a client computer; and    assembling said transactions into a requested file.    
     
     
         7 . The method of  claim 4 , wherein the step of establishing directory files and common files within a common file namespace, further includes the steps of: 
 assigning a unique data file identifier to each of said directory files and said common files, wherein said unique data file identifier is unique across said local network with network servers and client computers and independent of any particular member of said plurality of functionally equal network servers; and    translation of said common files or directory file full pathname within said common file namespace by traverse procedure using said directory file data along a logical pathname into said unique file identifiers.    
     
     
         8 . The method of  claim 4 , wherein the step of establishing directory files and common files within a common file namespace, further includes the step of: 
 creation of representation of file in the form of transactions records;    creating a procedure to disassemble said transaction records based upon a predetermined fault tolerance level; and    disassembling each of said transaction records according to said procedure.    
     
     
         9 . The method of  claim 8 , wherein the step of creating a procedure to disassemble each of said transaction records based upon a predetermined fault tolerance level further includes the steps of: 
 determining the size of each of said transaction records;    determining the required degree of said file redundancy; and    determining the minimum functional number of said servers based upon said predetermined fault tolerance levels.    
     
     
         10 . The method of  claim 8 , wherein the step of disassembling each of said transaction records according to said procedure further includes the steps of: 
 disassembling each of said transaction records into a plurality of data file pieces;    distributing said plurality of data file pieces to a member of said plurality of functionally equal network servers; and    further distributing said plurality of data file pieces to neighboring members of said plurality of functionally equal network servers;    whereby said predetermined fault tolerance level is achieved and each of said files remains accessible.    
     
     
         11 . The method of  claim 4 , wherein the step of utilizing client local software for distributed data file storage further includes the steps of: 
 tracing changes to said local file system of performed local operation system processes and daemons;    putting said changes into the form of transaction records;    disassembling transaction records into data pieces; and    if requested file does not exist in said local file system, when said local file access is requested from local operation system processes and daemons, performs a search procedure using client network software to find whether this data file is stored in the distributed file storage;    if said search procedure finds said requested file and downloads said requested file into local storage, readdresses said local data file access request to said loaded file and continues work in normal mode;    if said search procedure does not find said requested file, finish said local file access request with appropriate local operation system code denoting that file has not been found;    if said requested file exists in said local file system when said local file access is requested from local operating system processes and daemons, a check is performed of configuration conditions, and after positive response of said check performs the same search procedure using client network software to determine whether said requested file is stored in the distributed file storage and overlaps local files by data stored in distributed storage; after a negative response of said check readdresses said local file access request to said found file and continue work in normal mode.    
     
     
         12 . The method of  claim 4 , wherein the step of utilizing client network software for distributed file storage further includes the steps of: 
 communicating with the network servers;    recording said disassembled transaction pieces created by said software running on said local client computer into the distributed file storage; and    searching for pieces of said transaction records of said data file inside distributed storage;    collecting said pieces of said transaction records of said data file; and    assembling said transaction records and said files and putting them into the local file system of said client computer.    
     
     
         13 . The method of  claim 4 , wherein the step of utilizing client network software for distributed file storage further includes the steps of: 
 requesting a fault tolerance level;    sending transactional file records to the client network software;    disassembling said transactional file records into transactional data file pieces;    distributing said transactional data file pieces to a network server;    further distributing said data file pieces to neighboring servers;    whereby said requested fault tolerance level is achieved.    
     
     
         14 . The method of  claim 13 , wherein said step of requesting a fault tolerance level further includes the steps of: 
 determining the required degree of redundancy of said transactional file record pieces;    determining the required number of said transactional file record pieces;    determining the minimum number of functional servers;    whereby all data files in storage remain accessible.    
     
     
         15 . A method of organizing distributed file storage, comprising the steps of: 
 storing a data file as a series of transactions whereby each of said transactions is an incremental change which is integral and indivisible;    ordering said transactions by time;    storing each of said transactions logically as a special separate file with a unique transaction identifier;    collecting all the transactions for a certain time period;    reassembling said data file from said plurality of transactions.