Method and apparatus for storage and retrieval of very large databases using a direct pipe

Abstract

A method and apparatus for directly connecting very large data streams from an archive command into a backup data system using an “intelligent process.” An output stream is piped into an intelligent pipe-reading process and distributed over a set of temporary data stores built from raw storage resources. A pipe interface process supervises backup of each filled data store, while the remaining output stream continues to be piped into another available data store. The backup system completes archiving of the datastream, keeping a catalog of the datastream storage locations. To retrieve the data, the intelligent process is run in reverse as a pipe-writing process, requesting data from the backup system. Retrieved data traverses the data stores from the backup system and are pumped into the pipe-writing process for delivery to the pipe output file identified by the retrieve or import command.

1 . A method of directly connecting an output of a computer process to a storage device comprising the steps of: 
 creating a FIFO pipe fed by said output;    emptying said FIFO pipe into an intelligent process; and    storing data from said intelligent process into said storage device;    wherein said intelligent process includes at least one FIFO data store and each of said at least one FIFO data stores conveys said data from said FIFO pipe to said storage device.    
   
   
       2 . The method of  claim 1  in which said step of storing data from said intelligent process into said storage device further comprises the steps of: 
 reading data from said FIFO pipe;    selecting one of said at least one FIFO data stores to identify a selected data store;    marking said selected data store as busy;    writing said data read from said FIFO pipe into said selected data store; 
 if said selected data store is filled, and there is additional data in said FIFO pipe, then  
 selecting a next one of said at least one FIFO data stores,  
 writing said additional data in said FIFO pipe into said selected next one of said at least one FIFO data stores; and  
 writing said data from said selected data store into said storage device;  
 if said selected data store is filled, and there is no additional data in said FIFO pipe, then  
 waiting until said FIFO pipe is closed; and  
 writing said data from said selected data store into said storage device.  
   
   
   
       3 . The method of  claim 2  in which said step of selecting one of said at least one FIFO data store further comprises the steps of: 
 reading a list of available data stores;    determining which of said available data stores is not busy; and    identifying the determined available data store as being the selected data store.    
   
   
       4 . The method of  claim 3  in which said step of determining which of said data stores is not busy further comprises the steps of: 
 managing said list of available data stores as a round robin list in which each data store identified as busy is omitted from the round robin list, and each data store that becomes not busy is added to the end of the round robin list.    
   
   
       5 . The method of  claim 1  in which said step of storing data from said intelligent process into said storage device further comprises the steps of: 
 determining a sequence of said at least one data store to be filled with data;    filling said sequence of said at least one data store;    passing information regarding said sequence to said storage device; and    conveying data from said sequence of said at least one data store to said storage device according to said information.    
   
   
       6 . An apparatus for directly storing output from a computer process into a storage device comprising: 
 said computer process generating data to an output;    a FIFO pipe fed by said output;    an intelligent process fed by said FIFO pipe; and    said storage device for storing data from said intelligent process.    
   
   
       7 . The apparatus of  claim 6  in which said intelligent process further includes: 
 a set of physical storage locations arranged as a set of data stores;    a selector for determining which of said data stores should be next filled;    a monitor for determining when said data store is full; and    a selector for determining which of said full data stores should be next fed to said storage device.    
   
   
       8 . The apparatus of  claim 7  in which said selector is adapted to use a round robin list for selecting a next available data store, and each data store is removed from said list while being filled or while being fed to said storage device.  
   
   
       9 . A method of directly connecting a computer processing output to a data storage device comprising the steps of: 
 writing data from said computer processing output into a FIFO pipe;    emptying said FIFO pipe into an intelligent process;    using said intelligent process to distribute said data across a sequence of data stores selected from a plurality of available data stores;    emptying each of said data stores into said data storage device according to said sequence; and    signaling said intelligent process when said sequence is completed.    
   
   
       10 . The method of  claim 9  in which said step of using said intelligent process further comprises the steps of: 
 reading a list of available data stores;    reading data output from said FIFO pipe;    detecting the end of said data;    if said data has ended, then 
 signaling a data manager to empty a final data store;  
 waiting for said data manager to signal completion; and  
 recording a response code for said FIFO pipe;  
   if said data has not ended, then 
 selecting a current data store that is not busy;  
 testing whether the current data store is full;  
 if full, then 
 signaling the data manager to empty the current data store; and  
 marking the current data store as busy;  
 repeating said selecting and testing a current data store until a data store is found to be not full and not busy and then proceeding as if not full;  
 
 if not full, then 
 writing said data into said current data store; and  
 repeating said steps of reading data, detecting the end, selecting, testing, and writing, until said end of data is detected.  
 
   
   
   
       11 . A method of directly reading data from a storage device into a computer process comprising the steps of: 
 activating a FIFO pipe having an input from an intelligent process and an output to a destination file;    reading said data into said intelligent process from said storage device; and    writing said data into said FIFO pipe;    wherein said intelligent process includes at least one FIFO data store and each of said at least one FIFO data stores conveys said data from said storage device to said FIFO pipe.    
   
   
       12 . The method of  claim 11  in which said step of reading said data further comprises the steps of: 
 identifying one of said at least one FIFO data stores to be filled;    filling said identified data store with said data;    signaling said intelligent process to identify a next FIFO data store; and    repeating the steps of filling, signaling, and identifying until said data from said storage device is exhausted; and    signaling said intelligent process that said data is exhausted.    
   
   
       13 . The method of  claim 11  in which said step of writing said data into said FIFO pope further comprises the steps of: 
 obtaining a sequence of said at least one FIFO data stores containing data;    emptying said data into said FIFO pipe; and    repeating said steps of obtaining and emptying until a signal is received.    
   
   
       14 . The method of  claim 11  in which said at least one FIFO data store further comprises a plurality of data stores managed as a round robin list, whereby each one of said plurality of data stores is either identified for filling, waiting to be emptied, or available, as determined by said intelligent process.  
   
   
       15 . A method for communicating between a computer and a storage device comprising the steps of: 
 piping the output of a first computer process into an intelligent process;    storing the output of said intelligent process into a first file in said storage device;    reading data from a second file in a storage device into an intelligent process; and    piping said data into the input of a second computer process;    wherein said intelligent process includes a set of FIFO data stores for conveying said output to said first file and for conveying said data to said second computer process.    
   
   
       16 . The method of  claim 15  in which said step of storing further comprises the steps of: 
 selecting one of said set of FIFO data stores and filling it with said output;    signaling said storage device to empty said output; and    repeating said steps of selecting and signaling until said first computer process is complete.    
   
   
       17 . The method of  claim 15  in which said step of reading data further comprises the steps of: 
 selecting one of said set of FIFO data stores and filling it with said data;    signaling said intelligent process to empty said data; and    repeating said steps of selecting and signaling until said data file is exhausted.    
   
   
       18 . The method of  claim 15  in which said intelligent process further includes: 
 a list of FIFO data stores and a status indicator for each said FIFO data store on said list;    a sequencer for selecting and filling a FIFO data store according to said status indicator; and    a sequencer for selecting and emptying a FIFO data store according to said status indictor.    
   
   
       19 . The method of  claim 15  in which said intelligent process further includes: 
 a means for associating said output with said first file and    a means for associating said data with said second computer process.