Systems and methods to facilitate prioritization of documents in electronic discovery

Abstract

A method performed by at least one computing system and including performing a document identifying operation on a corpus of documents. The documents are associated one each with a plurality of numeric tier scores. The operation identifies results including one or more of the documents. The method includes calculating each tier score in a portion of the numeric tier scores and organizing the documents into tiers based at least in part on the numeric tier scores. The portion of the numeric tier scores is identified based on the results. The tiers are ranked from most to least relevant and include relevant and non-relevant portions. The method includes sending any of the tiers in the relevant portion one at a time to one or more reviewer computing devices in an order determined by the ranking. Any of the tiers in the non-relevant portion are not sent to reviewer computing device(s).

The invention claimed is: 
     
         1 . A computer-implemented method comprising:
 performing, by at least one computing system, a document identifying operation on a document corpus comprising a plurality of documents, the document identifying operation identifying results comprising one or more of the plurality of documents, the plurality of documents being associated one each with a plurality of numeric tier scores;   calculating, by the at least one computing system, each tier score in a portion of the plurality of numeric tier scores, the portion of the plurality of numeric tier scores being identified based on the results;   organizing, by the at least one computing system, the plurality of documents into tiers based at least in part on the plurality of numeric tier scores, the tiers being ranked from most relevant to least relevant, the tiers comprising a relevant portion and a non-relevant portion, the relevant portion comprising the most relevant of the tiers, the non-relevant portion comprising the least relevant of the tiers; and   sending, by the at least one computing system, any of the tiers in the relevant portion one at a time to one or more reviewer computing devices in an order determined by the ranking, the order sending the most relevant of the tiers to the one or more reviewer computing devices first, any of the tiers in the non-relevant portion not being sent to one or more reviewer computing devices.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 displaying, by the at least one computing system, information associated with the tiers in a graphical user interface.   
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 receiving, by the at least one computing system, a selection of a stopping point; and   using, by the at least one computing system, the stopping point to identify which of the tiers are in the relevant portion and which of the tiers are in the non-relevant portion.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein calculating each tier score in the portion of the plurality of numeric tier scores comprises:
 adding a relevance weight to each tier score in the portion of the plurality of numeric tier scores.   
     
     
         5 . The computer-implemented method of  claim 4 , further comprising:
 receiving, by the at least one computing system, the relevance weight as user input.   
     
     
         6 . The computer-implemented method of  claim 1 , wherein the document identifying operation is a relevance operation, the results are relevance results, and the method further comprises:
 setting, by the at least one computing system, the plurality of numeric tier scores equal to identical default numerical values before the document identifying operation is performed;   performing, by the at least one computing system, a non-relevance operation on the document corpus that identifies, as non-relevance results, at least one of the plurality of documents; and   setting, by the at least one computing system, each of the plurality of numeric tier scores associated with the at least one document equal to the identical default numerical values before the plurality of documents are organized into the tiers.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein the document identifying operation is a relevance operation, the results are relevance results, and the method further comprises:
 performing, by the at least one computing system, a non-relevance operation on the document corpus that identifies, as non-relevance results, at least one of the plurality of documents; and   reducing, by the at least one computing system, each of the plurality of numeric tier scores associated with the at least one document before the plurality of documents are organized into the tiers.   
     
     
         8 . The computer-implemented method of  claim 1 , wherein the document identifying operation is a document search and the method further comprises:
 receiving, by the at least one computing system, search criteria used to perform the document search.   
     
     
         9 . The computer-implemented method of  claim 1 , wherein the at least one computing system stores the document corpus as a data set,
 the data set is structured or unstructured,   the document identifying operation comprises at least one query, and   performing the document identifying operation comprises performing the at least one query on the data set.   
     
     
         10 . The computer-implemented method of  claim 1 , wherein the document identifying operation is a cluster analysis and the method further comprises:
 receiving, by the at least one computing system, a selection of at least one cluster identified by the cluster analysis, the at least one cluster comprising the one or more documents that are identified as the results.   
     
     
         11 . The computer-implemented method of  claim 1 , wherein the document identifying operation is an automated machine learning method that automatically selects the one or more documents that are identified as the results. 
     
     
         12 . A system comprising at least one processor and memory storing processor executable instructions that when executed by the at least one processor perform a method comprising:
 performing a plurality of document identifying operations on a document corpus comprising a plurality of documents, each of the plurality of document identifying operations identifying corresponding results comprising one or more of the plurality of documents, the plurality of documents being associated one each with a plurality of numeric tier scores;   after each of the plurality of document identifying operations, adding a relevance weight to any of the plurality of numeric tier scores associated with the one or more documents of the corresponding results;   organizing the plurality of documents into tiers based on the plurality of numeric tier scores, the tiers being ranked from a highest one of the plurality of numeric tier scores to a lowest one of the plurality of numeric tier scores to thereby define a review order; and   sending the tiers one at time and in accordance with the review order to one or more reviewer computing devices until a stopping point is reached to thereby avoid sending any of the plurality of documents associated with lower tier scores to the one or more reviewer computing devices.   
     
     
         13 . The system of  claim 12 , wherein the method further comprises:
 receiving the stopping point from a client computing device, the stopping point having been entered into the client computing device as user input.   
     
     
         14 . The system of  claim 12 , wherein the method further comprises:
 after sending each of the tiers in accordance with the review order to the one or more reviewer computing devices, (a) determining whether information related to any of the plurality of documents in the tier has been received from any of the one or more reviewer computing devices, and (b) determining the stopping point has been reached when no information related to any of the plurality of documents in the tier has been received from any of the one or more reviewer computing devices.   
     
     
         15 . The system of  claim 12 , wherein the method further comprises:
 receiving the relevance weight from a client computing device after each of the plurality of document identifying operations, the relevance weight having been entered into the client computing device as user input.   
     
     
         16 . The system of  claim 12 , wherein the method further comprises:
 setting the plurality of numeric tier scores equal to identical default numerical values before the plurality of document identifying operations are performed;   performing a non-relevance operation on the document corpus before the plurality of documents are organized into the tiers, the non-relevance operation identifying, as non-relevance results, at least one of the plurality of documents; and   setting each of the plurality of numeric tier scores associated with the at least one document equal to the identical default numerical values before the plurality of documents are organized into the tiers.   
     
     
         17 . The system of  claim 12 , wherein the method further comprises:
 performing a non-relevance operation on the document corpus before the plurality of documents are organized into the tiers, the non-relevance operation identifying, as non-relevance results, at least one of the plurality of documents; and   reducing each of the plurality of numeric tier scores associated with the at least one document before the plurality of documents are organized into the tiers.   
     
     
         18 . The system of  claim 12 , wherein the method further comprises:
 generating a graphical user interface with information associated with the tiers; and   transmitting the graphical user interface to a client computing device for display thereby.   
     
     
         19 . The system of  claim 12 , wherein the method further comprises:
 performing a statistical validation method configured to determine whether a reasonably high percentage of relevant documents are included in those of the plurality of documents sent to the one or more reviewer computing devices.   
     
     
         20 . A computer-implemented method for use with a document corpus comprising a plurality of documents, the method comprising:
 associating, by at least one computing system, each the plurality of documents with a tier score to thereby define a plurality of numeric tier scores;   repeatedly (a) receiving, by the at least one computing system, user-defined criteria related to at least one document identifying operation, (b) performing, by the at least one computing system, the at least one document identifying operation on the document corpus, the at least one document identifying operation identifying results comprising one or more of the plurality of documents, (c) displaying, by the at least one computing system, a graphical user interface allowing a user to demote or promote the results, (d) receiving, by the at least one computing system, an indication from the graphical user interface indicating that the user is demoting or promoting the results, (e) when the indication indicates that the user is promoting the results, increasing, by the at least one computing system, any of the plurality of numeric tier scores associated with the one or more documents of the results, and (f) when the indication indicates that the user is demoting the results, decreasing, by the at least one computing system, any of the plurality of numeric tier scores associated with the one or more documents of the results;   organizing the plurality of documents into tiers based on the plurality of numeric tier scores having a review order; and   sending the tiers one at time and in accordance with the review order to one or more reviewer computing devices until a stopping point is reached.