Block-level single instancing

Abstract

Systems and methods for single instancing blocks of data in a data storage system. For example, the data storage system may include multiple computing devices (e.g., client computing devices) that store primary data. The data storage system may also include a secondary storage computing device, a single instance database, and one or more storage devices that store copies of the primary data (e.g., secondary copies, tertiary copies, etc.). The secondary storage computing device receives blocks of data from the computing devices and accesses the single instance database to determine whether the blocks of data are unique (meaning that no instances of the blocks of data are stored on the storage devices). If a block of data is unique, the single instance database stores it on a storage device. If not, the secondary storage computing device can avoid storing the block of data on the storage devices.

1 . A system for storing a single instance of a data block in a data storage network, the system comprising:
 one or more computing devices, the one or more computing devices configured to:   determine a first set of one or more files that are not eligible for single instancing and a second set of one or more files that are eligible for single instancing;   divide the first and second set of one or more files into multiple data blocks;   transmit at least one data block of the first set of the multiple data blocks to one or more secondary storage computing devices; and   transmit at least one data block of the second set of the multiple data blocks to one or more secondary storage computing devices.   
     
     
         2 . The system of  claim 1 , wherein the one or more secondary storage computing devices comprises one or more memory buffers,
 wherein each of the one or more memory buffers has a size that is greater than a size of a data block but is less than ten times the size of the data block, and   wherein the one or more secondary storage computing devices is further configured to store multiple data blocks indicated to be eligible for single instancing in the one or more memory buffers.   
     
     
         3 . The system of  claim 1 , wherein the one or more computing devices is further configured to, for at least some of the multiple data blocks indicated to be eligible for single instancing, generate a unique identifier for each data block. 
     
     
         4 . The system of  claim 3 , wherein the transmitted multiple data blocks are transmitted via in one or more data streams,
 wherein the one or more data streams comprises multiple stream header items,   wherein each stream header item containing an indication of whether associated data block is eligible for single instancing.   
     
     
         5 . The system of  claim 1 , wherein the one or more secondary storage computing devices is further configured to, for at least some of the multiple data blocks indicated to be eligible for single instancing, generate a unique identifier for each data block. 
     
     
         6 . The system of  claim 1 , wherein the first and second set of one or more files are determined according to one or more parameters provided in an information management policy assigned to a storage device storing the first and second set of one or more files. 
     
     
         7 . The system of  claim 1 , further comprising:
 one or more storage devices storing multiple data blocks in one or more container files;   one or more single instance databases storing, for at least some of the multiple data blocks,   an identifier of a stored data block, and   a location of the stored data block in a container file; and   one or more index files storing, for at least some of the multiple data blocks, an indication whether the stored data block is referred to in one or more metadata files on the one or more storage devices.   
     
     
         8 . The system of  claim 7 , wherein the one or more single instance databases maintain:
 a data structure storing, for at least some of the multiple data blocks, a location of a reference to the data block.   
     
     
         9 . The system of  claim 1 , wherein a file is determined to be ineligible for single instancing when the file comprises metadata. 
     
     
         10 . The system of  claim 9 , wherein the metadata is one of:
 access control lists (ACLs),   disk partition information,   Master File Table (MFT), and   File Allocation Table (FAT) information.   
     
     
         11 . A method for storing a single instance of a data block in a data storage network, the method comprising:
 determining a first set of one or more files that are not eligible for single instancing and a second set of one or more files that are eligible for single instancing,   dividing the first and second set of one or more files into multiple data blocks;   transmitting at least one data block of the first set of the multiple data blocks to one or more secondary storage computing devices; and   transmitting at least one data block of the second set of the multiple data blocks to one or more secondary storage computing devices.   
     
     
         12 . The method of  claim 11 , wherein the method further comprises generating a unique identifier for each data block. 
     
     
         13 . The method of  claim 12 , wherein the transmitted multiple data blocks are transmitted via in one or more data streams,
 wherein the one or more data streams comprises multiple stream header items,   wherein each stream header item containing an indication of whether associated data block is eligible for single instancing.   
     
     
         14 . The method of  claim 11 , wherein the first and second set of one or more files are determined according to one or more parameters provided in an information management policy assigned to a storage device storing the first and second set of one or more files. 
     
     
         15 . The method of  claim 11 , wherein the method further comprises:
 storing multiple data blocks in one or more container files;   for at least some of the multiple data blocks storing in one or more single instance databases:
 an identifier of a stored data block, and 
 a location of the stored data block in a container file; and 
 one or more index files storing, for at least some of the multiple data blocks, an indication whether the stored data block is referred to in one or more metadata files on the one or more secondary storage computing devices. 
   
     
     
         16 . The method of  claim 15 , wherein the method further comprises:
 storing, for at least some of the multiple data blocks, a location of a reference to the data block.   
     
     
         17 . The method of  claim 11 , wherein a file is determined to be ineligible for single instancing when the file comprises metadata. 
     
     
         18 . A method for storing a single instance of a data block in a data storage network, the method comprising:
 determining a first set of one or more files that are not eligible for single instancing and a second set of one or more files that are eligible for single instancing,   dividing the first and second set of one or more files into multiple data blocks;   for at least some of the multiple data blocks indicated to be eligible for single instancing, generate a unique identifier for each data block; and   transmitting at least one data block of the first set of the multiple data blocks to one or more secondary storage computing devices; and   transmitting at least one data block of the second set of the multiple data blocks to one or more secondary storage computing devices.