Wan-optimized local and cloud spanning deduplicated storage system

Abstract

A spanning storage interface facilitates the use of cloud storage services by storage clients. The spanning storage interface presents one or more data interfaces to storage clients at a network location, such as file, object, data backup, archival, and storage block based interfaces. The data interfaces allows storage clients to store and retrieve data using non-cloud based protocols. The spanning storage interface may perform data deduplication on data received from storage clients. The spanning storage interface may transfer the deduplicated version of the data to the cloud storage service. The spanning storage interface may include local storage for storing a copy or all or a portion of the data from storage clients. The local storage may be used as a local cache of frequently accessed data, which may be stored data in its deduplicated form.

1 . A local and cloud spanning storage interface comprising:
 a front-end interface adapted to communicate with at least one storage client via a local-area network;   a back-end interface adapted to communicate with at least one cloud storage service via a wide-area network;   a data deduplication module adapted to reduce data redundancy in data received from the storage client to produce deduplicated data; and   a replication module adapted to transfer deduplicated data to the cloud storage service using the back-end interface.   
     
     
         2 . The local and cloud spanning storage interface of  claim 1 , wherein:
 the data deduplication module includes:   slab cache adapted to store a subset of labels and associated data segments in memory;   a reverse map cache adapted to store associations in memory between each of the subset of labels in the slab cache with a portion of a data stream; and   an anchor cache adapted to store associations in memory between portions of the data stream and a second subset of labels and data segments not stored in the slab cache; and   a local data storage adapted to store a copy of at least a portion of the data received from the storage client, wherein the local data storage includes:   a copy of a data segment slab file adapted to store in non-volatile storage the set of labels and associated data segments; and   a copy of a label map container file adapted to store in non-volatile storage at least one label map specifying an arrangement of labels corresponding with the data stream.   
     
     
         3 . The local and cloud spanning storage interface of  claim 2 , wherein the data segment slab file and the label map container file are stored using the cloud storage service. 
     
     
         4 . The local and cloud spanning storage interface of  claim 2 , wherein the copy of the data segment slab file and the copy of the label map container file are selected from a plurality of data segment slab files and label map container files stored using the cloud storage service. 
     
     
         5 . The local and cloud spanning storage interface of  claim 4 , wherein the copy of the data segment slab file and the copy of the label map container file are selected from a plurality of data segment slab files and label map container files based on a cache criteria. 
     
     
         6 . The local and cloud spanning storage interface of  claim 1 , comprising:
 a shell file system module adapted to present a shell file system representing the data to the storage client.   
     
     
         7 . The local and cloud spanning storage interface of  claim 1 , wherein the front-end interface includes a file system interface adapted to communicate with the storage client using a file system protocol. 
     
     
         8 . The local and cloud spanning storage interface of  claim 1 , wherein the front-end interface includes a backup system interface adapted to communicate with the storage client using a backup system protocol. 
     
     
         9 . The local and cloud spanning storage interface of  claim 1 , wherein the front-end interface includes a cloud storage interface adapted to communicate with the storage client using a cloud storage protocol. 
     
     
         10 . The local and cloud spanning storage interface of  claim 1 , wherein the front-end interface includes an archival interface adapted to communicate with the storage client using an archival protocol. 
     
     
         11 . The local and cloud spanning storage interface of  claim 1 , wherein the front-end interface includes an object interface adapted to communicate with the storage client using a binary large object protocol. 
     
     
         12 . The local and cloud spanning storage interface of  claim 1 , wherein the front-end interface includes a block storage interface adapted to communicate with the storage client using a block storage protocol. 
     
     
         13 . The local and cloud spanning storage interface of  claim 12 , wherein the block storage protocol includes iSCSI. 
     
     
         14 . The local and cloud spanning storage interface of  claim 1 , wherein the replication module is adapted to communicate a first portion of the deduplicated data to a first cloud storage service and a second portion of the deduplicated data to a second cloud storage service. 
     
     
         15 . The local and cloud spanning storage interface of  claim 14 , wherein the first and second portions of the deduplicated data determined from a specification provided by a user. 
     
     
         16 . The local and cloud spanning storage interface of  claim 14 , wherein the first and second portions of the deduplicated data are determined at least in part on a file path. 
     
     
         17 . The local and cloud spanning storage interface of  claim 1 , wherein the replication module is adapted to abandon the transfer of the deduplicated data to the cloud storage service in response to a quota being exceeded. 
     
     
         18 . The local and cloud spanning storage interface of  claim 17 , wherein the quota is associated with a storage client. 
     
     
         19 . A method of deduplicating a data stream, the method comprising:
 receiving a data stream;   generating new data segments from the data stream;   determining if the new data segments match previously generated data segments stored only on a cloud storage service;   in response to the determination that at least one of new data segments matches one of the previously generated data segments stored only on the cloud storage service, retrieving a set of data segments including the matching previously generated data segment from the cloud storage service;   assigning provisional labels to at least a portion of the new data segments not matching locally cached data segments; and   adding the provisional labels to the label map associated with the data stream.   
     
     
         20 . The method of  claim 19 , comprising:
 comparing the new data segments assigned to the provisional labels with the set of data segments retrieved from the cloud storage service;   in response to the determination that the new data segment matches one of the set of data segments, discarding the provisional label assigned to the new data segment and assigning a previously generated label associated with the matching one of the set of data segments to the new data segment.   
     
     
         21 . The method of  claim 19 , comprising:
 determining a set of changes to a locally stored label map container file and a locally stored slab file; and   transferring the set of changes to the cloud storage service to update a full and authoritative set of data segments and label maps.   
     
     
         22 . The method of  claim 21 , wherein transferring the set of changes includes performing an atomic operation to update the full and authoritative set of data segments and label maps.