Asynchronous Distributed De-Duplication for Replicated Content Addressable Storage Clusters

Abstract

A method is performed by a device of a group of devices in a distributed data replication system. The method includes storing an index of objects in the distributed data replication system, the index being replicated while the objects are stored locally by the plurality of devices in the distributed data replication system. The method also includes conducting a scan of at least a portion of the index and identifying a redundant replica(s) of the at least one of the objects based on the scan of the index. The method further includes de-duplicating the redundant replica(s), and updating the index to reflect the status of the redundant replica.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 at a first device of a plurality of devices of a distributed storage system, wherein the first device includes one or more processors and memory storing programs for execution by the one or more processors:
 accessing an index of objects stored in the distributed storage system, wherein the index is stored at the first device; 
 scanning a first portion of the index; 
 identifying in response to the scanning a redundant replica of one of the objects; 
 writing a new record to the first portion of the index corresponding to the redundant replica, wherein the new record includes a de-duplication designator; and 
   subsequent to the writing, replicating the first portion of the index to the plurality of devices to designate the redundant replica for de-duplication.