Method and apparatus for large scale data storage

Abstract

A logical apparatus and associated methods provide highly scalable and flexible data storage in a network of computers. The apparatus provides flexible organizational and access control mechanisms and a practical and efficient way to work with smaller portions of a data storage system at a given time to enable sparse population, caching, paging and related functions. A data structure, called a virtual container, comprises references to objects stored in a data storage system such that the same object can be visible from different virtual containers, if such virtual containers hold references to said object. Access controls further enhance the effectiveness of the methods and structures to enable multiple simultaneous organizational schemes and selective sharing of objects. Participating nodes provide access to objects stored on said nodes and their participating peer nodes, employing the data storage apparatus, such that balance in the network is achieved by data placement decisions that may combine common constraints and a node's individual self interest.

1 - 20 . (canceled) 
     
     
         21 . A virtual data storage system comprising:
 at least one computing device operative to execute a virtual storage application, the virtual storage application configured to:   generate at least one virtual storage layer in communication with at least one data storage system storing a plurality of data objects;   arrange at least one virtual container within the at least one virtual storage layer;   store at least one object reference in the at least one virtual container, the at least one object reference configured to reference at least one of the plurality of data objects; and   provide access to the plurality of data objects through the at least one object reference.   
     
     
         22 . The virtual data storage system of  claim 21 , wherein the at least one data storage system comprises at least one logical storage layer. 
     
     
         23 . The virtual data storage system of  claim 21 , wherein the at least one object reference is configured to reference a plurality of data objects. 
     
     
         24 . The virtual data storage system of  claim 23 , wherein the plurality of data objects comprises a plurality of elements associated with at least one multi-element object. 
     
     
         25 . The virtual data storage system of  claim 24 , wherein the at least one multi-element object comprises a multimedia file and wherein the plurality of data objects comprise at least one of the following multimedia file elements: an audio file, a video file, an image file, and a text file. 
     
     
         26 . The virtual data storage system of  claim 21 , wherein the at least one virtual container comprises at least one container layer. 
     
     
         27 . The virtual data storage system of  claim 26 , wherein the at least one container layer comprises a lowest layer and a plurality of higher layers and wherein the lowest layer comprises at least one cell comprising a hash table. 
     
     
         28 . The virtual data storage system of  claim 21 , wherein the at least one computing device comprises a plurality of computing devices and the at least one virtual container is stored across the plurality of computing devices. 
     
     
         29 . The virtual data storage system of  claim 21 , wherein the at least one reference is accessed using a uniform resource identifier. 
     
     
         30 . A computer-implemented method for providing virtual data storage, the method comprising:
 generating at least one virtual storage layer stored on at least one computing device, the at least one virtual storage layer being in communication with at least one data storage system storing a plurality of data objects;   arranging at least one virtual container within the at least one virtual storage layer;   storing at least one object reference in the at least one virtual container;   referencing at least one of the plurality of data objects through the at least one object reference; and   providing access to the plurality of data objects through the at least one object reference.   
     
     
         31 . The computer-implemented method  claim 30 , wherein the at least one computing device comprises a plurality of computing devices and arranging the at least one virtual container within the at least one virtual storage layer comprises storing the at least one virtual container across the plurality of computing devices. 
     
     
         32 . The computer-implemented method of  claim 30 , wherein providing access to the plurality of data objects comprises providing access to the plurality of data objects using a uniform resource identifier. 
     
     
         33 . The computer-implemented method of  claim 30 , further comprising obtaining information describing the plurality of data objects. 
     
     
         34 . The computer-implemented method of  claim 33 , wherein the information describing the plurality of data objects comprises metadata. 
     
     
         35 . The computer-implemented method of  claim 33 , wherein the information describing the plurality of data objects comprises a state of at least one of the plurality of data objects. 
     
     
         36 . The computer-implemented method of  claim 35 , wherein the state comprises at least one of the following: a data object is known to exist state, a data object is known to possibly exist state, a data object is not known to exist state, and a data object is known to exist state. 
     
     
         37 . The computer-implemented method of  claim 36 , further comprising:
 receiving, at the at least one computing device, a request to perform a search associated with at least one data object;   checking the state of the at least one data object; and   processing the request based on the state of the at least one data object.   
     
     
         38 . The computer-implemented method of  claim 37 , wherein processing the search request comprises terminating the search responsive to the state of the at least one data object comprising the data object is not known to exist state. 
     
     
         39 . The computer-implemented method of  claim 37 , wherein processing the search request comprises deferring the search responsive to the state of the at least one data object comprising the data object is known to possibly exist state. 
     
     
         40 . The computer-implemented method of  claim 37 , wherein processing the search request comprises performing the search responsive to the state of the at least one data object comprising the data object is known to exist state.