Multi-layer merge in a storage virtualization system

Abstract

The state of a file may be a combination of local state, typically small (e.g., a placeholder file), and some external source state such as that maintained in a read-only namespace managed by a cloud provider or by another local file system, typically large. A file system component responsible for overlaying (i.e., merging) the partial local state and the external source state into a single file system view that can be used by an application of a container as if the full state exists locally. Overlays that comprise the file system state may be referred to as “layers”. A tombstone mechanism may be provided to record delete or rename modifications in the top layer.

What is claimed is: 
     
         1 . In a computing device comprising a processor, memory, and a local storage unit, the memory storing computer-executable instructions that, when executed by the processor, implement a file system for managing the storage of files on the local storage unit, a method comprising:
 storing, on the local storage unit, as part of a top layer of the file system, a placeholder for a file, the file comprising data and state which is stored remotely from the local storage unit in one or more source layers of the file system, the placeholder comprising at least partial state of the file;   exposing the top layer of the file system to one or more applications, such that the top layer appears to the one or more applications as if it stores the entire state of the file;   merging, in response to a request from one of said one or more applications to access the file, the at least partial state of the file stored in the placeholder in the top layer of the file system with the remaining state of the file stored remotely in the one or more source layers to present a single view of the file to the requesting application.   
     
     
         2 . The method recited in  claim 1 , wherein the top layer is part of a namespace of a container implemented on the computing device. 
     
     
         3 . The method recited in  claim 1 , at least some of said one or more source layers also containing partial state of the file, said merging comprising merging the at least partial state of the file in the top layer with the remaining state in the one or more source layers, successively, until the complete state of the file is obtained. 
     
     
         4 . The method recited in  claim 3 , further comprising:
 determining whether the one or more source layers contain the complete state of the file; and   return an error to the requesting application, if it is determined that the one or more source layers do not contain the complete state of the file.   
     
     
         5 . The method recited in  claim 1 , further comprising:
 receiving a request from one of said one or more applications to delete or rename the file;   deleting the placeholder for the file from the top layer of the file system on the local storage unit;   creating a tombstone that provides a record of the deletion or renaming of the file; and   storing the tombstone in the top layer of the file system on the local storage unit.   
     
     
         6 . The method recited in  claim 5 , further comprising:
 preventing any file system operation from accessing, via the top layer, the file that was requested to be deleted or renamed until the tombstone has been created and stored in the top layer.   
     
     
         7 . The method recited in  claim 6 , the preventing comprising:
 recording, on the local storage unit, a file name of the file prior to its deletion or renaming; and   causing any attempted file system operations on that file name to wait behind the creation and storage of the tombstone.   
     
     
         8 . The method recited in  claim 5 , the tombstone comprising an empty file containing an indication that it represents a file that has been deleted or renamed. 
     
     
         9 . The method recited in  claim 5 , the tombstone superseding the existence of the file in the one or more remotely stored source layers. 
     
     
         10 . The method recited in  claim 5 , further comprising:
 receiving, from one of the one or more applications, a request to enumerate a directory of the file system;   enumerating the directory and determining that it contains the tombstone;   filtering out the tombstone from a result set for the enumeration; and   returning the filtered result set to the requesting application.   
     
     
         11 . A computing device comprising a processor, memory, and a local storage unit, the memory storing computer-executable instructions that, when executed by the processor, cause the device to:
 store, on the local storage unit, as part of a top layer of a file system of the computing device, a placeholder for a file, the file comprising data and state which is stored remotely from the local storage unit in one or more source layers of the file system, the placeholder comprising at least partial state of the file;   expose the top layer of the file system to one or more applications, such that the top layer appears to the one or more applications as if it stores the entire state of the file;   merge, in response to a request from one of said one or more applications to access the file, the at least partial state of the file stored in the placeholder in the top layer of the file system with the remaining state of the file stored remotely in the one or more source layers to present a single view of the file to the requesting application.   
     
     
         12 . The computing device recited in  claim 11 , wherein the top layer is part of a namespace of a container implemented on the computing device. 
     
     
         13 . The computing device recited in  claim 11 , at least some of said one or more source layers also containing partial state of the file, said merging comprising merging the at least partial state of the file in the top layer with the remaining state in the one or more source layers, successively, until the complete state of the file is obtained. 
     
     
         14 . The computing device recited in  claim 13 , the computer-executable instructions further causing the computing device to:
 determine whether the one or more source layers contain the complete state of the file; and   return an error to the requesting application, if it is determined that the one or more source layers do not contain the complete state of the file.   
     
     
         15 . The computing device recited in  claim 11 , the computer-executable instructions further causing the device to:
 receive a request from one of said one or more applications to delete or rename the file;   delete the placeholder for the file from the top layer of the file system on the local storage unit;   create a tombstone that provides a record of the deletion or renaming of the file; and   store the tombstone in the top layer of the file system on the local storage unit.   
     
     
         16 . The computing device recited in  claim 15 , the computer-executable instructions further causing the device to:
 prevent any file system operation from accessing, via the top layer, the file that was requested to be deleted or renamed until the tombstone has been created and stored in the top layer.   
     
     
         17 . The computing device recited in  claim 16 , the preventing comprising:
 recording, on the local storage unit, a file name of the file prior to its deletion or renaming; and   causing any attempted file system operations on that file name to wait behind the creation and storage of the tombstone.   
     
     
         18 . The computing device recited in  claim 15 , the tombstone comprising an empty file containing an indication that it represents a file that has been deleted or renamed. 
     
     
         19 . The computing device recited in  claim 15 , the tombstone superseding the existence of the file in the one or more remotely stored source layers. 
     
     
         20 . The computing device recited in  claim 15 , the computer-executable instructions further causing the device to:
 receive, from one of the one or more applications, a request to enumerate a directory of the file system;   enumerate the directory and determine that it contains the tombstone;   filter out the tombstone from a result set for the enumeration; and   return the filtered result set to the requesting application.