Selective deduplication

Abstract

Methods and apparatuses for performing selective deduplication in a storage system are introduced here. Techniques are provided for determining a probability of deduplication for a data object based on a characteristic of the data object and performing a deduplication operation on the data object in the storage system prior to the data object being stored in persistent storage of the storage system if the probability of deduplication for the data object has a specified relationship to a specified threshold.

1 - 20 . (canceled). 
     
     
         21 . A method, comprising:
 evaluating characteristics of a data object to determine a deduplication probability of the data object based upon a projected likelihood that deduplication of the data object will reduce storage consumption within a storage device of a data storage system;   evaluating the deduplication probability against a deduplication probability threshold to determine whether to perform inline deduplication, post-processing deduplication, or no deduplication based upon the deduplication probability; and   adjusting the deduplication probability threshold based upon a performance metric of the data storage system.   
     
     
         22 . The method of  claim 21 , comprising:
 measuring system resource availability of the data storage system; and   adjusting the deduplication probability threshold based upon the system resource availability of the data storage system.   
     
     
         23 . The method of  claim 21 , comprising:
 measuring the performance metric as a function of a service level objective.   
     
     
         24 . The method of  claim 21 , comprising:
 measuring read and write response times of the data storage system as the performance metric.   
     
     
         25 . The method of  claim 21 , comprising:
 determining the performance metric as a function of read and write throughout rates of the data storage system.   
     
     
         26 . The method of  claim 21 , comprising:
 adjusting the deduplication probability threshold such that inline deduplication is perform to an extent that the performance metric is still satisfied.   
     
     
         27 . The method of  claim 21 , comprising:
 adjusting the deduplication probability based upon characteristic-based statistics.   
     
     
         28 . A computing device comprising:
 a memory having stored thereon instructions for performing a method; and   a processor coupled to the memory, the processor configured to execute the instructions to cause the processor to:
 evaluate characteristics of a data object to determine a deduplication probability of the data object based upon a projected likelihood that deduplication of the data object will reduce storage consumption within a storage device of a data storage system; 
 evaluate the deduplication probability against a deduplication probability threshold to determine whether to perform inline deduplication, post-processing deduplication, or no deduplication based upon the deduplication probability; and 
 measure system resource availability of the data storage system; and 
 adjust the deduplication probability threshold based upon the system resource availability of the data storage system. 
   
     
     
         29 . The computing device of  claim 28 , wherein the instructions cause the processor to:
 in response to determining that a performance requirement of the data storage system can be met if the deduplication probability is lowered, lower the deduplication probability; and   in response to determining that the performance requirement of the data storage system will not be met if the deduplication probability is lowered, refrain from lowering the deduplication probability.   
     
     
         30 . The computing device of  claim 28 , wherein the instructions cause the processor to:
 in response to determining that a performance metric is not being satisfied, increase the deduplication probability.   
     
     
         31 . The computing device of  claim 28 , wherein the instructions cause the processor to:
 preemptively increase the deduplication probability based upon a determination that a workload of the data storage system will be increasing at a subsequent point in time.   
     
     
         32 . The computing device of  claim 28 , wherein the instructions cause the processor to:
 preemptively increase the deduplication probability based upon a determination that available resources of the data storage system will decreased at a subsequent point in time.   
     
     
         33 . The computing device of  claim 28 , wherein the instructions cause the processor to:
 adjust the deduplication probability threshold to compensate for a change in a characteristic of a data object.   
     
     
         34 . The computing device of  claim 28 , wherein the instructions cause the processor to:
 assign a higher priority to a first data object having a higher deduplication probability than a second data object having a lower deduplication probability.   
     
     
         35 . A non-transitory machine-readable storage media having stored thereon instructions, for performing a method, which causes a processor to:
 evaluate characteristics of a data object to determine a deduplication probability of the data object based upon a projected likelihood that deduplication of the data object will reduce storage consumption within a storage device of a data storage system;   evaluate the deduplication probability against a deduplication probability threshold to determine whether to perform inline deduplication, post-processing deduplication, or no deduplication based upon the deduplication probability; and   adjust the deduplication probability threshold based upon system resource availability of the data storage system.   
     
     
         36 . The non-transitory machine-readable storage media of  claim 35 , wherein the instructions cause the processor to:
 utilize a multi-level deduplication probability process to prioritize data objects for deduplication.   
     
     
         37 . The non-transitory machine-readable storage media of  claim 35 , wherein the instructions cause the processor to:
 assign differing levels of priority to data objects for deduplication processing based upon which multiple levels of deduplication probability thresholds are satisfied.   
     
     
         38 . The non-transitory machine-readable storage media of  claim 37 , wherein the multiple levels deduplication probability thresholds have hierarchical relationships with respect to one another. 
     
     
         39 . The non-transitory machine-readable storage media of  claim 35 , wherein the instructions cause the processor to:
 adjust the deduplication probability threshold to maintain a target rate of object inline deduplication.   
     
     
         40 . The non-transitory machine-readable storage media of  claim 39 , wherein the instructions cause the processor to:
 measure a current rate of data object inline deduplication based upon a quantity of data being inline deduplicated.