Virtual storage system and method

Abstract

A virtual storage system for use in conjunction with a host computer is disclosed. The system features a memory control processor external to the host computer which divides user-defined data sets into blocks of a size convenient for storage on, e.g., magnetic media and individually assigns these blocks to locations determined external to the host. In this way, the extent of a particular data file is not specified by the user; nor is empty space allocated in anticipation of future use. The virtual memory system may additionally comprise a high speed cache memory for receiving data written to the memory system at high speed from the host. Data anticipated to be the subject of future requests can be staged to the cache, so that it can be supplied to the host at high speed, thus improving system performance. Data compression and decompression may be incorporated in the storage system. Numerous data back-up and automated recovery processing operations may additionally be performed by this system without specific instruction from the host.

I claim: 
     
       1. Apparatus for the storage of digital data organized into records comprising: host interface means for receiving data from a host computer;   cache means connected to said host interface means for temporary storage of said data;   storage device interface means for connecting said cache means to one or more memory storage devices of one or more classes of storage devices for comparatively long-term storage of said data; ad   controller means including address recall means, for: (a) determining where said data is to be stored on said .Iadd.one or more .Iaddend.storage devices;   (b) controlling flow of said data from said host interface means through said cache means and said storage device interface means onto said .Iadd.one or more .Iaddend.storage devices; and   (c) storing the addresses at which said data has been stored on said .Iadd.one or more .Iaddend.storage devices;     whereby a given user-defined data set may be stored on a plurality of storage devices, selected from one or more classes of storage devices by said controller means.   
     
     
       2. The apparatus of claim 1 wherein each said data set is divided into pages of length convenient for storage on devices of said .Iadd.one or more .Iaddend.classes by said controller, which operates individually on each of said pages to determine where and on what class of storage devices said page shall be stored. 
     
     
       3. The apparatus of claim 1 wherein said host interface means comprises channel adapter means for compression of said data by replacing successive identical bytes of data by a compressed data word comprised of one of said identical bytes together with a character indicative of the number of times said .Iadd.one or more .Iaddend.identical .[.byte.]. .Iadd.bytes .Iaddend.was repeated in the uncompressed data. 
     
     
       4. The apparatus of claim 3 wherein said compressed data word additionally comprises a character for identification of said compressed data word as such. 
     
     
       5. A data processing and storage system comprising: at least one host computer for performing arithmetic and logic operations on digital data, an extremely high speed, relatively low capacity memory means in said computer, and interface means for transmitting and receiving data from external memory means; and   a virtual storage system, comprising: relatively low high capacity storage devices;   host interface means for receiving data from and transmitting data to said .Iadd.at least one .Iaddend.host;     a high speed, relatively low capacity cache memory mean connected to said host interface means; and   a controller for allocating appropriate storage locations on said relatively low speed, high capacity storage devices for data received from said .Iadd.at least one .Iaddend.host and stored in said cache .Iadd.memory means.Iaddend., for accordingly controlling the flow of data from said cache .Iadd.memory means .Iaddend.to said relatively low speed, high capacity storage device, and for storing said storage locations.   
     
     
       6. The system of claim 5 wherein said relatively low speed, high capacity storage devices include devices of plural classes. 
     
     
       7. The system of either of .[.claims 5 or 6.]. .Iadd.claim 6,.Iaddend. in which said .Iadd.at least one .Iaddend.host computer does not control the location of storage of said data on said relatively low speed, high capacity storage devices, nor .[.the.]. .Iadd.a .Iaddend.choice of class of device on which said data is stored. 
     
     
       8. The system of claim 5 wherein said data is divided by said controller into pages of convenient length for storage on said relatively low speed, high capacity storage devices and the location for storage of each said page is determined individually by said controller. 
     
     
       9. The system of claim 5 wherein said controller operates to store a single user-define data set over plural storage devices as required, whereby a user of said system is not required to define .[.the.]. .Iadd.an .Iaddend.extent of storage locations for the containment of data sets operated on by said .Iadd.at least one .Iaddend.host. 
     
     
       10. The system of claim 5 wherein a single user identified data set may be divided into pages by said controller and stored on differing ones of said relatively low speed storage devices. 
     
     
       11. The method of operation of a data processing and storage system, said system comprising data processing means for outputting results of said processing for storage, and a data storage unit, said data storage unit comprising: plural types of addressable storage devices for long-term storage of said data; and   controller means for allocating long-term storage locations within said long-term storage devices to said data and for controlling flow of data from said data processing means to said long-term storage devices;   wherein the improvement comprises performing the step of determining where and on what type of addressable .[.data.]. storage device(s) the data is to be stored in said controller means, exclusive of the function of said data processing .[.unit.]. .Iadd.means.Iaddend..     
     
     
       12. The method of claim 11, wherein data is stored on relatively higher speed storage devices, and is periodically copied to relatively lower speed storage devices. 
     
     
       13. The method of claim 12 wherein only that data stored on said relatively higher speed storage .[.device.]. .Iadd.devices .Iaddend.in the interim following .[.the.]. .Iadd.a .Iaddend.last previous copying operation is copied. 
     
     
       14. The method of claim 11 wherein said data is simultaneously stored on differing ones of storage devices of the same type upon receipt in said data storage unit. 
     
     
       15. The method of claim 11 wherein said data is supplied by said .Iadd.data .Iaddend.processing .[.unit.]. .Iadd.means .Iaddend.to said .Iadd.data .Iaddend.storage unit in user-defined data sets, and said data sets are divided into pages by said controller .Iadd.means .Iaddend.for storage in said .Iadd.data .Iaddend.storage unit. 
     
     
       16. The method of claim 15 wherein the length of each .[.said.]. page is determined in accordance with the characteristics of the type of storage device on which storage locations for said pages are allocated. 
     
     
       17. The method of claim 11, further comprising the step of temporarily storing said data in .[.said.]. .Iadd.a .Iaddend.short-term memory means within said data storage unit during said step of determining .[.at what long-term storage location.]. .Iadd.where and on what type of addressable storage device(s) .Iaddend.said data is to be stored. 
     
     
       18. The method of claim 11 further comprising the step of compressing said data prior to said step of determining .[.at what long-term storage location.]. .Iadd.where and on what type of addressable storage device(s) .Iaddend.said data is to be stored. 
     
     
       19. The method of claim 18 wherein said data is divided into pages for storage assignment after said compressing step is performed. 
     
     
       20. A magnetic data storage system comprising means for receiving data from a host computer, means for temporary storage of said data means for allocating locations on long-term storage media for said data, and for controlling flow of said data through said system, memory means for recall of said locations for long-term storage of said data, and means for periodically duplicating data stored in a first long-term storage location by copying it to a second long-term storage means, wherein said means for .Iadd.periodically .Iaddend.duplicating data is controlled such that said copying .[.step.]. is performed with respect to only that data which has been operated upon in the interim since .[.the.]. .Iadd.a .Iaddend.last preceding copying.[.step.]., as indicated by said memory means for recall. .[.21. A data storage and recall system, comprising means for receiving data from means for processing data, means for long-term storage of said data, means for temporary storage of said data, means for allocating locations on long-term storage media for said data, means for controlling flow of said data through said system, memory means for recall of said locations for long-term storage of said data, and means for outputting 
     
     
        said data upon instruction..].22. A data storage system for connection to a host computer, said host computer being adapted to indicate to said storage system that a particular data access request is one of a sequence of such requests to be directed to a particular user data set during execution of a particular user program, wherein, said data storage system comprises: long-term, lower speed data storage means and shorter term higher speed data storage means, .[.where upon receipt from said.]. .Iadd.wherein the data storage system is responsive to the .Iaddend.host computer .[.by said data storage system of.]. .Iadd.indicating that .Iaddend.a first data access command request .Iadd.is .Iaddend.directed to a portion of a user data set.[., said data storage system transfers.]. .Iadd.to transfer .Iaddend.a larger portion of said user data set than that requested from said large capacity, slower speed memory means into said smaller capacity, higher speed memory means in anticipation of further host access requests   
     
     
        directed to said user data set. 23. The system of claim 22 wherein said user data set is divided into pages by controller means comprised in said storage system, said pages being stored on plural ones of .Iadd.said .Iaddend.long-term.Iadd., lower speed .Iaddend.data storage .[.devices.]. .Iadd.means.Iaddend., the locations of storage of said pages being retained in address memory means.Iadd., said controller means arranged to determine the location of the larger portion of the user data set in response to the host indication, and transfer the larger portion into the 
     
     
        smaller capacity, higher speed memory means.Iaddend.. 24. .[.Method of operation of a.]. .Iadd.A .Iaddend.data processing and storage system comprising a host computer system and a data storage system, said data storage system comprising: long-term magnetic storage media of relatively high capacity and relatively low speed, and   short-term semi-conductor memory means of relatively low capacity and relatively high speed,   wherein said host .Iadd.computer system .Iaddend.indicates to said storage system that a particular data access request is one of a series of access requests directed to a particular user-defined data set in execution of a user program,   wherein said data storage system comprises processor means for responding to said indication by transferring portions of said user defined data set not called for by said host from said .[.large capacity, slow speed.]. magnetic storage .[.means.]. .Iadd.media .Iaddend.to said .[.low capacity, high speed.]. semi-conductor memory means in anticipation of further requests directed to said data set by said host .Iadd.computer system   
     
     
        .Iaddend.in execution of said program. 25. The .[.method.]. .Iadd.data processing and storage system .Iaddend.of claim 24 wherein said user-defined data set is divided into pages, said pages being individually assigned to storage locations on plural ones of said .[.long-term magnetic memory devices.]. .Iadd.magnetic storage media .Iaddend.by said processor means.Iadd., said processor means determining the location of the user-defined data set not called for by the host, and transferring the portions of the user defined data set not called for by the host into the smaller capacity, higher speed memory means.Iaddend.. .[.26. A data storage subsystem for connection to a host computer by way of a channel comprising: magnetic disk data storage means;   magnetic tape data storage means;   solid-state cache data storage means;   data bus means for transmission of data from said channel to said disk means, from said disk means to said cache means and from said cache means to said tape means; and   controller means for directing the flow of data along said data bus means connecting said cache means, said channel means and said tape means unit, whereby a data set can be copied from a disk memory means into said cache and thence to said tape means..]..[.27. The subsystem of claim 26 further comprising means for determining where and on what type of magnetic storage media said data is to be stored..]..[.28. The subsystem of claim 26 wherein said means for determining further comprises means for division of a user data set into subportions sized in accordance with the physical characteristics of the media on which said subportions are to be stored..]..[.29. The subsystem of claim 28 further comprising means for recall of the location at which said subportions are stored, and for   
     
     
        reassembly of said subportions into a complete data set..]..[.30.  In a data storage subsystem adapted to be connected to a channel interfacing a host computer system with a data storage subsystem, the improvement which comprises: said data storage subsystem comprising: controller means;   disk storage means;   tape storage means; and   solid state cache memory means, said cache means being effectively connected to said magnetic disk storage means and to said magnetic tape storage means, said controller means controlling flow of data between said disk means and said cache means and between said cache means and said tape means, whereby upon indication that a given data set or portion of a data set should be copied to tape media for backup purposes, said data set or portion of a data set may be copied from said disk to said cache and thence to said tape..].