Directory aggregation for files distributed over a plurality of servers in a switched file system

Abstract

A file switch, logically positioned between client computers and file servers in a computer network, distributes user files among multiple file servers using an aggregated directory mechanism. A hierarchical directory structure is created on the file servers and used to store metadata files, which store metadata for each user file to indicate where data files, containing portions of the user file, are stored. The file switch automatically spreads the data files and metadata files over a large number of distinct directories on multiple file servers, preventing large number of data files from being stored in a single directory on a single file server. In response to a directory enumeration request from a client computer, one or more directories of metadata files on one or more of the file servers is enumerated, instead of enumerating the data file that store the user file portions.

What is claimed is:  
     
         1 . A method of generating a directory listing of user files, where each user file of at least a plurality of the user files is divided into portions stored on a plurality of file servers, among a group of file servers in a computer network, comprising: 
 receiving a user request for a directory listing of user files for a specified directory;    accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored; and    constructing a directory listing of the user files from the directory of metafiles.    
     
     
         2 . The method of  claim 1 , wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.  
     
     
         3 . The method of  claim 2 , wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.  
     
     
         4 . The method of  claim 1 , including: 
 whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, updating the directory of metafiles to indicate the changed aggregated file size of the user file.    
     
     
         5 . The method of  claim 1 , wherein 
 the specified directory has an associated file path; and    the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         6 . A file switch for use in a computer network having a group of file servers and a plurality of client computers, the file switch comprising: 
 at least one processing unit for executing computer programs;    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    a file aggregation module including one or more computer programs, the computer programs including instructions for: 
 receiving a user request for a directory listing of user files for a specified directory;  
 accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored;  
 constructing a directory listing of the user files from the directory of metafiles; and  
 sending the directory listing to a particular client computer of the plurality of client computers.  
   
     
     
         7 . The file switch of  claim 6 , wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.  
     
     
         8 . The file switch of  claim 7 , wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.  
     
     
         9 . The file switch of  claim 6 , wherein the computer programs of the file aggregation module include instructions for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.  
     
     
         10 . The file switch of  claim 6 , wherein the specified directory has an associated file path; and 
 the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         11 . A system for use in a computer network having a plurality of client computers, the system comprising: 
 a group of file servers;    a file switch, the file switch including:    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    means for receiving a user request for a directory listing of user files for a specified directory;    means for accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored;    means for constructing a directory listing of the user files from the directory of metafiles; and    means for sending the directory listing to a particular client computer of the plurality of client computers.    
     
     
         12 . The system of  claim 11 , wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.  
     
     
         13 . The system of  claim 11 , wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.  
     
     
         14 . The system of  claim 11 , including means for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.  
     
     
         15 . The system of  claim 11 , wherein the specified directory has an associated file path; and 
 the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         16 . A system for use in a computer network having a plurality of client computers, the system comprising: 
 a group of file servers;    a file switch, the file switch including:    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    at least one processing unit for executing computer programs;    a file aggregation module including one or more computer programs, the computer programs including instructions for: 
 receiving a user request for a directory listing of user files for a specified directory;  
 accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored;  
 constructing a directory listing of the user files from the directory of metafiles; and  
 sending the directory listing to a particular client computer of the plurality of client computers.  
   
     
     
         17 . The system of  claim 16 , wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.  
     
     
         18 . The system of  claim 17 , wherein the aggregated file size is stored in a date field of the directory listing of the metafiles.  
     
     
         19 . The system of  claim 16 , wherein the computer programs of the file aggregation module include instructions for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.  
     
     
         20 . The system of  claim 16 , wherein the specified directory has an associated file path; and 
 the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         21 . A method of generating a directory listing of user files, where each user file of at least a plurality of the user files is divided into portions stored on a plurality of file servers, among a group of file servers in a computer network, comprising: 
 receiving a user request for a directory listing of user files for a specified directory;    accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file; and    aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files.    
     
     
         22 . The method of  claim 21 , wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.  
     
     
         23 . The method of  claim 22 , wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.  
     
     
         24 . The method of  claim 21 , including: 
 whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.    
     
     
         25 . The method of  claim 21 , wherein 
 the specified directory has an associated file path; and    each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         26 . A file switch for use in a computer network having a group of file servers and a plurality of client computers, the file switch comprising: 
 at least one processing unit for executing computer programs;    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    a file aggregation module including one or more computer programs, the computer programs including instructions for: 
 receiving a user request for a directory listing of user files for a specified directory;  
 accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file;  
 aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files; and  
 sending the aggregated directory listing to a particular client computer of the plurality of client computers.  
   
     
     
         27 . The file switch of  claim 26 , wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.  
     
     
         28 . The file switch of  claim 27 , wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.  
     
     
         29 . The file switch of  claim 26 , the computer programs of the file aggregation module including instructions, activated when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, for updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.  
     
     
         30 . The file switch of  claim 26 , wherein 
 the specified directory has an associated file path; and    each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         31 . A system for use in a computer network having a plurality of client computers, the system comprising: 
 a group of file servers;    a file switch, the file switch including:    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    means for receiving a user request for a directory listing of user files for a specified directory;    means for accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file;    means for aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files; and    means for sending the aggregated directory listing to a particular client computer of the plurality of client computers.    
     
     
         32 . The system of  claim 31 , wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.  
     
     
         33 . The system of  claim 32 , wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.  
     
     
         34 . The system of  claim 31 , including means for updating the corresponding directory of metafiles, when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, to indicate the changed aggregated file size of the user file.  
     
     
         35 . The system of  claim 31 , wherein 
 the specified directory has an associated file path; and    each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         36 . A system for use in a computer network having a plurality of client computers, the system comprising: 
 a group of file servers;    a file switch, the file switch including:    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    at least one processing unit for executing computer programs;    a file aggregation module including one or more computer programs, the computer programs including instructions for: 
 receiving a user request for a directory listing of user files for a specified directory;  
 accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined portions of the specified user file;  
 aggregating the accessed directories of metafiles to construct an aggregated directory listing of the user files; and  
 sending the aggregated directory listing to a particular client computer of the plurality of client computers.  
   
     
     
         37 . The system of  claim 36 , wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.  
     
     
         38 . The system of  claim 37 , wherein the aggregated file size is stored in a date field of the respective directory listing of the metafiles.  
     
     
         39 . The system of  claim 36 , the computer programs of the file aggregation module including instructions, activated when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, for updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.  
     
     
         40 . The system of  claim 36 , wherein 
 the specified directory has an associated file path; and    each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.    
     
     
         41 . A method for determining storage locations of a user file in a group of file servers in a computer network, comprising the steps of: 
 in accordance with a predefined set of aggregation rules, determining a set of file servers from the group of file servers for storing the user file;    for each portion of the user file to be stored in a respective one of the determined set of servers, generating a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and    storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.    
     
     
         42 . The method of  claim 41 , wherein the file path for each portion of the user file is generated by generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.  
     
     
         43 . The method of  claim 41 , including generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.  
     
     
         44 . A file switch for use in a computer network having a group of file servers and a plurality of client computers, the file switch comprising: 
 at least one processing unit for executing computer programs;    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    a file aggregation module including one or more computer programs, the computer programs including instructions for: 
 determining, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;  
 for each portion of the user file to be stored in a respective one of the determined set of servers, generating a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and  
 storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.  
   
     
     
         45 . The file switch of  claim 44 , wherein the computer programs of the file aggregation module include instructions for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.  
     
     
         46 . The file switch of  claim 44 , wherein the computer programs of the file aggregation module include instructions for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.  
     
     
         47 . A system for use in a computer network having a plurality of client computers, the system comprising: 
 a group of file servers;    a file switch, the file switch including:    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;    means for determining, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;    means for generating, for each portion of the user file to be stored in a respective one of the determined set of servers, a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and    means for storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.    
     
     
         48 . The system of  claim 47 , including means for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.  
     
     
         49 . The system of  claim 47 , including means for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.  
     
     
         50 . A system for use in a computer network having a plurality of client computers, the system comprising: 
 a group of file servers;    a file switch, the file switch including:    at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file; and    a file aggregation module including one or more computer programs, the computer programs including instructions for: 
 determining, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;  
 for each portion of the user file to be stored in a respective one of the determined set of servers, generating a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server; and  
 storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.  
   
     
     
         51 . The system of  claim 50 , wherein the computer programs of the file aggregation module include instructions for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.  
     
     
         52 . The system of  claim 50 , wherein the computer programs of the file aggregation module include instructions for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.