System to organize search and display unstructured data

Abstract

A system to organize, search and display unstructured data comprising a token retrieval module, a document indexing engine, a subspace search module and a user interface module has been devised. The system retrieves a plurality of tokens and associates them with coordinates in subspace. It also retrieves documents and creates a multidimensional matrix of documents and tokens where each cell contains the number of times the token occurs in each document. That matrix is employed in a search using user specified search terms. The search results are displayed such that the search tokens occupy specific spatial coordinates and documents spatial coordinates are dictated by the relative preponderance of each search term in each document.

What is claimed is: 
     
         1 . A system to organize, search and display unstructured data comprising:
 a token retrieval module stored in a memory of and operating on a processor of a computing device and configured to:
 retrieve a plurality of tokens from a source; 
 create a rank basis matrix for a subspace of the space of tokens; 
 associate retrieved tokens with multidimensional coordinates in subspace; 
 persistently store the subspace coordinate associated tokens; 
   a document indexing engine stored in a memory of and operating on a processor of a computing device and configured to:
 retrieve documents from a document source; 
 create a raw multidimensional matrix of documents and subspace coordinate associated tokens where each cell contains the number of times the single token occurs in the corresponding document; 
 create a normalized matrix of documents and subspace coordinate associated tokens from the raw multidimensional matrix; 
 and create from the rank basis matrix an associated subspace coordinate matrix of the normalized matrix; 
   a subspace search module stored in a memory of and operating on a processor of computing device and configured to:
 receive selected subspace coordinate associated tokens as search tokens from the user interface module; 
 create a rank basis matrix for search from the selected tokens; 
 perform a search of documents using the search tokens and using the basis matrix for the search; and 
 calculate document display coordinates from search results; 
   a user interface module stored in a memory of and operating on a processor of a computing device and configured to:
 allow a user to select document source for an analysis; 
 allow the user to select token source for analysis; 
 accept selected subspace coordinate associated search tokens from the user; 
 display search tokens and documents at view space coordinates calculated from document display coordinates retrieved from the subspace search module. 
   
     
     
         2 . The system of  claim 1 , wherein search tokens are designated through selection from a list of tokens. 
     
     
         3 . The system of  claim 1 , wherein at least at least one set of documents is retrieved from a corporate specific source. 
     
     
         4 . The system of  claim 1 , wherein the rank basis matrix for the subspace is retrieved from memory. 
     
     
         5 . The system of  claim 1 , wherein the rank basis matrix for the subspace is calculated from the normalized matrix for a sample of documents. 
     
     
         6 . The system of  claim 1 , wherein the coordinates for the display search tokens are optimized to minimize the information entropy. 
     
     
         7 . The system of  claim 1 , wherein at least one set of documents consists of network flow data from transmission into and out of a computer network. 
     
     
         8 . The system of  claim 4 , wherein at least one set of tokens are network addresses associated with a computer network. 
     
     
         9 . The system of  claim 4 , wherein at least one token is a network port number. 
     
     
         10 . The system of  claim 4 , wherein at least at least one token is an identifier of a process associated with computer network traffic. 
     
     
         11 . The system of  claim 4 , wherein at least one token is an identifier of a protocol used in network data transmission. 
     
     
         12 . The system of  claim 1 , wherein display coordinates of at least one search token on a search result view field is modified by a user. 
     
     
         13 . The system of  claim 1 , wherein the source of at least one set of tokens is from a lexicon of corporate field specific terms. 
     
     
         14 . A method for organization, search and display unstructured data comprising the steps of:
 a) retrieving a plurality of tokens and assigning subspace coordinates to them using a token retrieval module;   b) retrieving a plurality of documents and indexing them into a multidimensional matrix with the retrieved tokens using a document indexing engine;   c) normalizing and centering the rows and columns of the multidimensional matrix using the document indexing engine;   d) searching the normalized multidimensional matrix with user specified tokens and creating display coordinate for each token and each searched document using a subspace search module;   e) displaying search result dictated tokens and documents at coordinates that reflect the relative preponderance of each search token in each analyzed document using a user controlled display module.   
     
     
         15 . The method of  claim 11 , wherein search tokens are designated through selection from a list of tokens. 
     
     
         16 . The method of  claim 11 , wherein at least at least one set of documents is retrieved from a corporate specific source. 
     
     
         17 . The method of  claim 11 , wherein at least one set of documents consists of network flow data from transmission into and out of a computer network. 
     
     
         18 . The method of  claim 17 , wherein at least one set of tokens are network addresses associated with a computer network. 
     
     
         19 . The method of  claim 14 , wherein at least one token is a network port number. 
     
     
         20 . The method of  claim 14 , wherein at least at least one token is an identifier of a process associated with computer network traffic. 
     
     
         21 . The method of  claim 14 , wherein at least one token is an identifier of a protocol used in network data transmission. 
     
     
         22 . The method of  claim 11 , wherein display coordinates of at least one search token on a search result view field is be modified by a user. 
     
     
         23 . The method of  claim 11 , wherein the source of at least one set of tokens is from a lexicon of corporate field specific terms.