Method and apparatus for improving the integration between a search engine and one or more file servers

Abstract

In one aspect of the invention, a search engine parses files stored among one or more file servers in order to create and maintain index information used by the search engine to perform searches. For a given file server, the population of files presented to the search engine is reduced in size to facilitate the process of updating the index. In another aspect of the invention, the file server limits the files presented in a directory list request made by a search engine. This reduces the number of file s that need to be considered when performing an index update.

1 . A method for accessing data comprising: 
 storing a plurality of files in a file server;    monitoring operations on one or more of the files in the file server;    if a file in the file server is modified, then adding information representative of the file in an update list, wherein the update list contains information representative of files that have been modified;    providing an index, the index comprising information produced from an analysis of one or more of the files in the file server, the index being accessed by a first computer other than the file server;    obtaining information from the update list, thus identifying each file contained in the update list; and    for each file contained in the update list, updating the index with information produced from an analysis of the file, whereby the updating is performed only on those files which have been modified.    
   
   
       2 . The method of  claim 1  wherein the step of obtaining information from the update list includes communicating to the first computer first information representative of one or more files referenced in the update list.  
   
   
       3 . The method of  claim 2  wherein the update list is stored in the file server.  
   
   
       4 . The method of  claim 2  wherein the first information comprises file references contained in the update list.  
   
   
       5 . The method of  claim 2  wherein the first information comprises copies of the files referenced in the update list.  
   
   
       6 . The method of  claim 1  wherein the update list is stored in a first file and the step of obtaining information from the update list includes communicating a copy of the first file to the first computer.  
   
   
       7 . The method of  claim 6  wherein the first file is stored in the file server.  
   
   
       8 . The method of  claim 1  further comprising clearing the update list when the index is updated, wherein contents of the update list are deleted.  
   
   
       9 . The method of  claim 8  wherein the step of updating is performed by the first computer.  
   
   
       10 . The method of  claim 1  wherein the index is stored in the first computer.  
   
   
       11 . The method of  claim 10  wherein the computer is a search engine server, wherein the index facilitates performing a search of files stored in the file server  
   
   
       12 . A method for accessing data comprising: 
 storing a plurality of files;    receiving a request for a file operation to be performed on a first file;    if the file operation is a write operation, then storing a reference into an update list which identifies the first file, whereby the update list comprises references of only those files whose content have been modified;    receiving a request from a first computer for the update list and in response thereto, communicating information contained in the update list to the first computer; and    subsequent to the step of communicating information, removing the information contained in the update list.    
   
   
       13 . The method of  claim 12  wherein the one or more file operations comprises a plurality of write operations, wherein the step of storing identification information is performed only upon detecting a first of the write operations.  
   
   
       14 . The method of  claim 13  wherein the first write operation is received subsequent to receiving a clear request.  
   
   
       15 . The method of  claim 12  wherein the step of communicating information includes communicating content of the update list to the first computer.  
   
   
       16 . The method of  claim 15  wherein a copy of the update list is communicated to the first computer.  
   
   
       17 . The method of  claim 16  further comprising receiving from the first computer file operation requests for reading files identified in the update list, and in response to each such request communicating the requested file to the first computer.  
   
   
       18 . The method of  claim 17  wherein the first computer is a search engine.  
   
   
       19 . The method of  claim 12  wherein the step of communicating information includes providing a copy of each file that is referenced in the update list to the first computer.  
   
   
       20 . A file server comprising: 
 storage for storing a plurality of files;    an update list; and    a file server controller,    the file server controller configured to perform the method steps of: 
 receiving a request for a file operation to be performed on a first file;  
 if the file operation is a write operation, then storing a reference to the first file into the update list, whereby the update list comprises references of only those files whose content have been modified;  
 receiving a request from a first computer for the update list and in response thereto, communicating information contained in the update list to the first computer; and  
 subsequent to the step of communicating information, removing the information contained in the update list.  
   
   
   
       21 . The file server of  claim 20  wherein the first computer is a search engine.  
   
   
       22 . A method for accessing files from a file server comprising: 
 receiving a search request and in response thereto accessing an index using search criteria associated with the search request to obtain information which identifies any files that match the search criteria and communicating the information in the form of a search result, the index comprising information based on files stored among one or more file servers; and    updating the index comprising:    receiving file information from a first file server, the file information representative of only those files contained in the first file server that have been modified subsequent to a first point in time; and    for each file: 
 accessing the file;  
 parsing the file to produce index information; and  
 updating the index with the index information,  
 wherein only those files that have been modified since the first point in time are accessed and parsed.  
   
   
   
       23 . The method of  claim 22  wherein the first point in time is a time when a previous update of the index with files from the first file server was being performed.  
   
   
       24 . The method of  claim 22  further including creating the index, wherein the first point in time is a time subsequent to creating the index.  
   
   
       25 . The method of  claim 22  further comprising creating an index including: 
 accessing a plurality of first files from the first file server;    parsing one of the first files to produce index information; and    adding the index information into the index, thereby indexing one of the first files,    wherein the steps of parsing and adding are repeated for each of the first files,    wherein the first point in time is a time subsequent to indexing all of the first files.    
   
   
       26 . The method of  claim 25  further comprising communicating a first request to the first file server upon indexing the plurality of first files, whereby the first point in time is determined based on the file server receiving the first request.  
   
   
       27 . The method of  claim 25  wherein creating an index further comprises: 
 accessing a plurality of second files from a second file server;    parsing one of the second files to produce index information; and    adding the index information into the index, thereby indexing one of the second files,    wherein the steps of parsing and adding are repeated for each of the second files.    
   
   
       28 . The method of  claim 22  wherein the step of updating the index is performed for a plurality of file servers, wherein each file server is associated its own first point in time which is a point time subsequent to when the index was previously updated with files from the file server.  
   
   
       29 . The method of  claim 28  further including creating the index, wherein the first point in time is a time subsequent to creating the index.  
   
   
       30 . The method of  claim 28  wherein the first point in time is a time subsequent to a previous updating of the index.  
   
   
       31 . The method of  claim 22  wherein the step of updating the index is repeated for a plurality of additional file servers, wherein only those files in each additional file server which have been modified since the first point in time are accessed and parsed.  
   
   
       32 . A computer for accessing files comprising: 
 a file access controller;    an index accessible by the file access controller; and    computer program code configured to control the file access controller to perform the method steps of  claim 22 .    
   
   
       33 . A search engine server comprising: 
 a search engine controller;    an index accessible by the search engine controller; and    computer program code configured to control the search engine controller to perform the method steps of  claim 22 .    
   
   
       34 . A system for data access comprising: 
 a first file server;    a second server configured to communicate with the first file server;    an index file accessible by the second server, the index file comprising index information obtained from files stored in the first file server; and    a first update file accessible by the first file server,    the first file server configured to add file references to the first update list for files in the first file server whose contents have changed since a first point in time, and further configured to provide first update information contained in the first update list to the second server,    the second server configured to: 
 receive the first update information;  
 access files referenced in the first update information;  
 analyze each of the files to produce index information; and  
 update the index with the index information,  
 whereby updating the index for files stored on the first file server includes accessing only those files which are referenced in the first update list.  
   
   
   
       35 . The system of  claim 34  wherein the second server is a search engine server.  
   
   
       36 . The system of  claim 34  wherein the first point in time is a time subsequent to when the index was created.  
   
   
       37 . The system of  claim 36  wherein the second server is further configured to create the index and to send a first request to the first file server after the index is created, the first point in time being a time subsequent to the first file server receiving the first request.  
   
   
       38 . The system of  claim 36  wherein the second server is further configured to send a first request to the first file server after the index is updated, the first file server further configured to clear the first update list in response to receiving the first request, the first point in time being a time subsequent to a time when the first update list is cleared.  
   
   
       39 . The system of  claim 34  further comprising a second file server and a second update list accessible by the second file server, the second server further being configured for communication with the second file server, 
 the index further comprising index information obtained from files stored in the second file server,    the second file server configured to add file references to the second update list for files in the second file server whose contents have changed since a second point in time and further configured to provide second update information contained in the second update list to the second server,    the second server further configured to update the index based on files referenced in the second update list.    
   
   
       40 . The system of  claim 39  wherein the second point in time is a point in time subsequent to when the index created.  
   
   
       41 . The system of  claim 39  wherein the second server is further configured to create the index and to send a first request to the first file server and to the second file server after the index is created, wherein the first point in time is a time subsequent to the first file server receiving the first request, wherein the second point in time is a time subsequent to the second file server receiving the first request.  
   
   
       42 . A method for accessing data comprising: 
 storing one or more files in a file server;    receiving a first directory list request for a first directory at the file server, the first directory list request originating from a first computer;    in response to receiving the first directory list request from the first computer, producing a first directory listing that is representative of contents of the first directory;    receiving a second directory list request for the first directory at the file server, the second directory list request originating from a second computer;    in response to receiving the second directory list request from the second computer, producing a second directory listing that is representative of contents of the first directory, files represented in the second directory listing being based on one or more criteria contained in a file filter table; and    in the second computer, updating an index based on the second directory listing.    
   
   
       43 . The method of  claim 42  wherein the second computer is a search engine server.  
   
   
       44 . The method of  claim 42  wherein the one or more criteria are based on one or more of: file types; file owner information; file creation dates; and file sizes.  
   
   
       45 . The method of  claim 42  wherein the file filtering table comprises one or more file types which indicate whether files are to be excluded from the second directory listing.  
   
   
       46 . The method of  claim 45  wherein the file filtering table further comprises one or more of file owner information, file creation dates, file sizes.  
   
   
       47 . The method of  claim 42  wherein the file filter specifies which files are to be included in the second directory listing.  
   
   
       48 . The method of  claim 42  wherein the file filter specifies which files are to be excluded from the second directory listing.  
   
   
       49 . The method of  claim 42  wherein the file filter specifies which files are to be included in the second directory listing and which files are to be excluded from the second directory listing.  
   
   
       50 . A method for accessing data comprising: 
 storing one or more files in a file system on a file server;    providing a plurality exports of the file system to a plurality of computer systems;    receiving from a first computer system a directory list request for a first directory stored on the file server;    producing a first directory listing that is representative of contents of the first directory if the first computer system has not mounted a predetermined one of the exports; and    producing a second directory listing that is representative of contents of the first directory if the first computer system has mounted a predetermined one of the exports, wherein files represented in the second directory listing are determined based on one or more criteria contained in a file filter table, wherein an index in the first computer system is updated based on information in the second directory listing.    
   
   
       51 . The method of  claim 50  wherein the first computer system is a search engine server.  
   
   
       52 . The method of  claim 50  wherein the file filter specifies which files are to be included in the second directory listing.  
   
   
       53 . The method of  claim 50  wherein the file filter specifies which files are to be excluded from the second directory listing.  
   
   
       54 . A method for accessing data comprising: 
 storing one or more files in a file system on a file server;    receiving from a first computer system a directory list request for a first directory contained on the file server, the directory list request including source information comprising an identifier of the first computer system;    producing a first directory listing that is representative of contents of the first directory if the identifier of the first computer system is different from a predetermined identifier; and    producing a second directory listing that is representative of contents of the first directory if the identifier of the first computer system is the same as the predetermined identifier, wherein files represented in the second directory listing are determined based on one or more criteria contained in a file filter table, wherein an index in the first computer system is updated based on information in the second directory listing.    
   
   
       55 . The method of  claim 54  wherein the identifier is an internet protocol (IP) address.  
   
   
       56 . In a file server, a method for providing access to files contained in the file server comprising: 
 organizing the files in a file system;    providing access to the file system to a plurality of computer systems;    storing information representative of one or more predetermined computer systems;    receiving from a first computer system a directory list request for a first directory stored on the file server;    producing a first directory listing that is representative of contents of the first directory if the first computer system is not one of the predetermined computer systems; and    producing a second directory listing that is representative of contents of the first directory if the first computer system is one of the predetermined computer systems, wherein files represented in the second directory listing are determined based on one or more criteria contained in a file filter table.    
   
   
       57 . The method of  claim 56  wherein the file filtering table comprises one or more file types which indicate, by file type, whether files are to be excluded from the second directory listing.  
   
   
       58 . The method of  claim 57  wherein the file filtering table further comprises one or more of file owner information, file creation dates, file sizes.  
   
   
       59 . The method of  claim 56  wherein the file filtering table comprises one or more criteria which indicate whether a file is to be excluded form the second directory listing.  
   
   
       60 . The method of  claim 56  wherein the file filtering table comprises one or more criteria which indicate whether a file is to be included in the second directory listing.  
   
   
       61 . The method of  claim 56  wherein the file filtering table comprises one or more first criteria which indicate whether a file is to be included in the second directory listing and one or more second criteria which indicate whether a file is to be included in the second directory listing.  
   
   
       62 . The method of  claim 56  further comprising providing one or more exports to the one or more computer systems, wherein the predetermined one or more computer systems are identified by the exports they have mounted, whereby the steps of producing are based on which of the one or more exports the first computer system has mounted.  
   
   
       63 . The method of  claim 56  wherein the predetermined one or more computer systems are identified by source addresses, whereby the steps of producing are based on a source address of the first computer system.  
   
   
       64 . The method of  claim 63  wherein the source address is an IP address.  
   
   
       65 . A file server comprising: 
 storage for storing a plurality of files;    a file filter table; and    a file server controller,    the file server controller configured to perform the method steps of  claim 56 .    
   
   
       66 . A method for accessing files comprising: 
 detecting a write operation to a first file in a file server;    selectively adding a representation of the first file into an update list based on one or more file filter criteria, wherein the detecting step and the selectively adding step is repeated for additional files in the file server;    communicating update information relating to content of the update list to a first computer,    subsequent to communicating the update information, clearing the update list; and    in the first computer updating a search index including accessing files contained in the file server based on the update information.    
   
   
       67 . The method of  claim 66  wherein file filter criteria specify one or more of: a file type; file ownership; file creation date; and file size.  
   
   
       68 . The method of  claim 66  wherein the update information comprises file references contained in the update list.  
   
   
       69 . The method of  claim 66  wherein the update information comprises copies of the files referenced in the update list.  
   
   
       70 . The method of  claim 66  wherein the update list is stored in a first file and the step of obtaining information from the update list includes communicating a copy of the first file to the first computer.  
   
   
       71 . The method of  claim 66  wherein the first computer is a search engine.  
   
   
       72 . A method for accessing data comprising: 
 detecting write operations on first files in a file server;    for each first file, selectively adding a reference to the first file into an update list based on one or more filtering criteria;    receiving a first request from a first computer, and in response thereto communicating update information relating to the update list; and    subsequent to communicating the update information, clearing the update list.    
   
   
       73 . The method of  claim 72  wherein the filtering criteria include at least one of: a file type; file ownership; file creation date; and file size.  
   
   
       74 . The method of  claim 73  wherein the filtering criteria specify whether to add a file to the update list.  
   
   
       75 . The method of  claim 73  wherein the filtering criteria specify whether to exclude a file from the update list.  
   
   
       76 . The method of  claim 72  wherein the update information is a copy of the update list that is transferred to the first computer.  
   
   
       77 . The method of  claim 72  wherein the update information comprises copies of files referenced in the update list.  
   
   
       78 . The method of  claim 72  wherein the first computer is a search engine.  
   
   
       79 . A file server for providing access to data comprising: 
 storage for storing a plurality of files;    an update list;    a file filter table; and    a file server controller,    the file server controller configured to perform the method steps of  claim 72.