Object Storage in Cloud with Reference Counting Using Versions

Abstract

A data storage apparatus includes an interface and one or more processors. The interface is configured for communicating with a cloud-based object storage system 5 having a built-in versioning mechanism that assigns version numbers to objects stored therein. The one or more processors are configured to receive data for storage from one or more workloads, to store the data as objects in the cloud-based object storage system, and to 10 update and record reference counts for at least some of the objects, by forcing the built-in versioning mechanism of the cloud-based object storage system to update the version numbers so as to match the reference counts.

What is claimed is: 
     
         1 . A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
 receiving data for storage from a workload;   determining that an existing object that corresponds to the data received for storage is stored in a cloud-based object storage system, the cloud-based object storage system generating version numbers for object metadata of objects stored in the cloud-based object storage system and the existing object comprising corresponding object metadata comprising a first version number, wherein each object stored in the cloud-based object storage system comprises a corresponding value and corresponding object metadata;   in response to determining that the existing object is stored in the cloud-based object storage system, communicating an update metadata command for the existing object to the cloud-based object storage system, the update metadata command, when received by the cloud-based object storage system, causing the cloud-based object storage system to:
 update the corresponding object metadata of the existing object; and 
 generate a second version number for the existing object; and 
   recording, as a reference count corresponding to the data for storage from the workload, the second version number.   
     
     
         2 . The method of  claim 1 , wherein the operations further comprise:
 determining a hash value for an object of the data received for storage; and   matching the hash value of the object of the data received for storage to the existing object to determine that the existing object corresponds to the data received for storage.   
     
     
         3 . The method of  claim 2 , wherein the hash value serves as a key for objects stored in the cloud-based object storage system. 
     
     
         4 . The method of  claim 1 , wherein the corresponding value of the existing object is not updated in response to receiving the update metadata command. 
     
     
         5 . The method of  claim 1 , wherein the update metadata command further causes the cloud-based object storage system to update the corresponding value of the existing obj ect. 
     
     
         6 . The method of  claim 1 , wherein the operations further comprise:
 receiving additional data for storage from the workload;   determining that an object corresponding to the additional data is not stored in the cloud-based object storage system; and   in response to determining that the object corresponding to the additional data is not stored in the cloud-based object storage system, communicating the object corresponding to the additional data to the cloud-based object storage system for storage.   
     
     
         7 . The method of  claim 6 , wherein the operations further comprise recording, as a reference count corresponding to the additional data, a third version number generated by the cloud-based object storage system for the corresponding metadata of the object corresponding to the additional data. 
     
     
         8 . The method of  claim 1 , wherein the cloud-based object storage system comprises a versioning mechanism that generates the version numbers. 
     
     
         9 . The method of  claim 1 , wherein the existing object that corresponds to the data received for storage comprises a dummy object. 
     
     
         10 . The method of  claim 9 , wherein the dummy object comprises a minimum size supported by the cloud-based object storage system. 
     
     
         11 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions executed on the data processing hardware that cause the data processing hardware to perform operations comprising:
 receiving data for storage from a workload; 
 determining that an existing object that corresponds to the data received for storage is stored in a cloud-based object storage system, the cloud-based object storage system generating version numbers for object metadata of objects stored in the cloud-based object storage system and the existing object comprises corresponding object metadata comprising a first version number, wherein each object stored in the cloud-based object storage system comprises a corresponding value and corresponding object metadata; 
 in response to determining that the existing object is stored in the cloud-based object storage system, communicating an update metadata command for the existing object to the cloud-based object storage system, the update metadata command, when received by the cloud-based object storage system, causing the cloud-based object storage system to:
 update the corresponding object metadata of the existing object; and 
 generate a second version number for the existing object; and 
 
 recording, as a reference count corresponding to the data for storage from the workload, the second version number. 
   
     
     
         12 . The system of  claim 11 , wherein the operations further comprise:
 determining a hash value for an object of the data received for storage; and   matching the hash value of the object of the data received for storage to the existing object to determine that the existing object corresponds to the data received for storage.   
     
     
         13 . The system of  claim 12 , wherein the hash value serves as a key for objects stored in the cloud-based object storage system. 
     
     
         14 . The system of  claim 11 , wherein the corresponding value of the existing object is not updated in response to receiving the update metadata command. 
     
     
         15 . The system of  claim 11 , wherein the update metadata command further causes the cloud-based object storage system, to update the corresponding value of the existing obj ect. 
     
     
         16 . The system of  claim 11 , wherein the operations further comprise:
 receiving additional data for storage from the workload;   determining that an object corresponding to the additional data is not stored in the cloud-based object storage system; and   in response to determining that the object corresponding to the additional data is not stored in the cloud-based object storage system, communicating the object corresponding to the additional data to the cloud-based object storage system for storage.   
     
     
         17 . The system of  claim 16 , wherein the operations further comprise recording, as a reference count corresponding to the additional data, a third version number generated by the cloud-based object storage system for the corresponding metadata of the object corresponding to the additional data. 
     
     
         18 . The system of  claim 11 , wherein the cloud-based object storage system comprises a versioning mechanism that generates the version numbers. 
     
     
         19 . The system of  claim 11 , wherein the existing object that corresponds to the data received for storage comprises a dummy object. 
     
     
         20 . The system of  claim 19 , wherein the dummy object comprises a minimum size supported by the cloud-based object storage system.