Search-on-the-fly/sort-on-the-fly by a search engine directed to a plurality of disparate data sources

Abstract

A Sort-on-the-Fly/Search-on-the-Fly search engine provides an intuitive means for searching databases, allowing a user to access data in the database without having to know anything about the database structure. A user selects a desired search term, and the search engine searches the database for all instances of the desired term, even if a specific file or table does not contain the instance. The user may specify other criteria, or constraints to narrow the search results, or for other reasons. The search engine then conducts a further search using these criteria and produces a second search result. Further narrowing or broadening of the search are permitted, with the search-on-the-fly search engine returning results based on any new constraints. If the returned data would be too large to be conveniently displayed at a terminal, the search engine executes a truncation routine so that the returned data is easily displayed.

The invention claimed is: 
     
       1. A method, executed on a suitably programmed computing device, for managing data, the method comprising:
 extracting, by a search engine, a plurality of data field descriptors from at least one database from each of a plurality of disparate data sources, comprising the search engine:
 performing a preliminary database access function to determine access to each of the at least one database, wherein for each database:
 determining a schema of the database, and 
 determining available data fields of the database; 
 
 
 using the plurality of data field descriptors from each of the at least one database, developing a data field result list upon which a query may be run, by:
 providing a display of the plurality of data field descriptors and control data from the at least one database, the control data comprising one or more of constraints, fields, keywords, and truncations used for extracting the plurality of data field descriptors from the at least one database, and 
 saving the plurality of data field descriptors and the control data as the data field result list for each of the at least one database; 
 
 receiving, from a graphical user interface, a user search request directed to the data field result list; 
 transforming the user search request into a dataset query based on the data field result list by combining the user search request with an API request; 
 communicating the dataset query to the at least one database; 
 searching the at least one database using the dataset query; and 
 managing, based on the data field result list, data from the at least one database comprising providing a response to the dataset query of the at least one database. 
 
     
     
       2. The method of  claim 1 , wherein developing the data field result list comprises gathering and structuring metadata of underlying databases. 
     
     
       3. The method of  claim 2 , wherein the gathered metadata is stored in a metadata database. 
     
     
       4. The method of  claim 1 , wherein developing the data field result list comprises importing a dataset. 
     
     
       5. The method of  claim 1 , wherein developing the data field result list comprises developing a dataset tree. 
     
     
       6. The method of  claim 1 , wherein the user search request is transformed into a system search request and the system search request is transformed into the dataset query. 
     
     
       7. The method of  claim 1 , wherein managing data from the at least one database comprises retrieving data from the at least one database to create the response. 
     
     
       8. The method of  claim 7 , further comprising transforming the response into a search result display. 
     
     
       9. The method of  claim 1 , wherein managing the data sources comprises writing data to the at least one database. 
     
     
       10. The method of  claim 1 , wherein each of the disparate data sources communicates in a different language and the user search request is processed into a plurality of dataset queries, each dataset query being configured for communicating with a different disparate data source. 
     
     
       11. The method of  claim 1 , further comprising communicating with a different data source using an adapter configured to communicate in the language of the different data source. 
     
     
       12. The method of  claim 1 , wherein the at least one database is sorted/searched by a user using the dataset query. 
     
     
       13. A system, including a non-transitory, computer-readable storage medium comprising machine instructions stored on and executed by a suitable computing device, for managing data comprising:
 a setup portion configured for building a data field result list, developed by extracting, using a search engine, a plurality of data field descriptors from at least one database of each of a plurality of disparate data sources, wherein the data field result list comprises the plurality of data field descriptors and control data from each of the at least one database, wherein the search engine:
 performs a preliminary database access function to determine access to each of the at least one database, wherein for each database:
 determining a schema of the database, and 
 determining available data fields of the database; 
 
 develops a data field result list from the plurality of data field descriptors from each of the at least one database; 
 stores the data field result list in a metadata database; and 
 
 a user portion for receiving and processing a user request, the user portion performed by a communication system and a data processing component, comprising:
 the data processing component that:
 provides a display of the plurality of data field descriptors and control data from the at least one database, the control data comprising one or more of constraints, fields, keywords, and truncations used for extracting the plurality of data field descriptors of the at least one database, and 
 saves the plurality of data field descriptors and the control data as the data field result list for each of the at least one database; 
 receives, from a user graphical interface, a user search request directed to the data field result list; 
 transforms the user search request into a dataset query based on the data field result list by combining the user search request with an API request; 
 communicates the dataset query to the at least one database; 
 searches the at least one database using the dataset query; and 
 manages, based on the data field result list, data from the at least one database comprising providing a response to the dataset query of the at least one database. 
 
 
 
     
     
       14. The system of  claim 13 , wherein the setup portion comprises a configuration tool. 
     
     
       15. The system of  claim 13 , wherein the system receives the user search request and generates a response via interaction with the data processing component. 
     
     
       16. The system of  claim 13 , wherein the processing component comprises at least one adapter, and wherein the at least one adapter is configured for communicating with a specific type of data source. 
     
     
       17. The system of  16 , wherein the at least one adapter retrieves data from the specific type of data source. 
     
     
       18. The system of  claim 17 , further comprising a transformation manager for transforming the retrieved data into a response. 
     
     
       19. The system of  claim 16 , wherein the at least one adapter comprises a query builder. 
     
     
       20. The system of  claim 13 , wherein the data processing component further comprises an adapter factory and wherein the adapter factory receives the request and returns an adapter from the at least one adapter associated with a data source for the request. 
     
     
       21. The system of  claim 13 , wherein the at least one database is sorted/searched by a user using the dataset query.