Use of virtual database technology for internet search and data integration

Abstract

This invention discloses how Virtual Database Technology can be used to make disparate data appear to be (or act as) the sort of uniform data one expects to find within a single relational database. In particular, we show how to process queries similar to those one might use in a database, even though the underlying data may be missing some of the capabilities that are required by normal databases. Whereas traditional databases require that all the tuples in a table be stored, our approach allows queries over tables where the tuples are generated as required from the data sources, and may not be stored anywhere. We show how such facilities can be used as a new foundation for Internet search.

1 . A method to integrate data available from diverse sources such as static and dynamic web pages, files and documents in different formats, databases, and APIs (Application Program Interfaces) comprising the step of using a logical query language to integrate data access across the diverse sources. 
     
     
         2 . The method of  claim 1 , comprising the further steps of viewing a data source as a plurality of relations; extracting tuples of the relation from the data source using an automated process; the relation name, number of relation columns, column names, the location of the data source, and the automated process together defined as the metadata specification for each relation. 
     
     
         3 . The method of  claim 1 , comprising the further step of accessing data available from incomplete tables stored for an indeterminate period of time. 
     
     
         4 . The method of  claim 3 , wherein the only data access possible from a data source is output specifically related to provided inputs. 
     
     
         5 . The method of  claim 1 , wherein the data is under transactional control of some process other than the process accessing the data. 
     
     
         6 . The methods of  claim 1 , wherein the location of a metadata specification is different from the location of the data source, and no cooperation from either the data source provider and owner is required to apply the methods of  claim 1 . 
     
     
         7 . The method of  claim 1 , comprising the further step of describing a single data source by a plurality of different metadata specifications. 
     
     
         8 . The method of  claim 7 , wherein the plurality of metadata specifications can be supplied by a plurality of different authors operating independently, and stored at different locations on the Internet. 
     
     
         9 . The method of  claim 2 , further comprising the step of storing metadata on the Internet in a manner that such metadata specifications can be indexed by search engines and discovered by users when they supply relevant keywords. 
     
     
         11 . The method of  claim 1 , further comprising the step of storing search queries on the Internet in a manner that such queries can be indexed by search engines, discovered by search engine users, and then reused by those users. The method of claim  10  further comprising the step of creating a reusable search query that can be simplified for end users by restricting the presentation of the search query to just the user inputs required to execute the query. 
     
     
         12 . The method of claim  10  further comprising the step of customizing any discovered search query by the user who retrieved the search query. 
     
     
         13 . The method of  claim 2  further comprising the step of limiting the maximum time period for which the metadata specification for each relation from a data source remains valid. 
     
     
         14 . The methods of  claim 1  further comprising the step of storing data from a data source in a local cache; and retrieving the data from the cache if the data has been in the cache for less time than the maximum time period specified in the metadata specification. 
     
     
         15 . The method of  claim 2 , wherein the location of a metadata specification is different from the location of the data source, and no cooperation from either the data source provider and owner is required to apply the methods of  claim 2 . 
     
     
         16 . The method of  claim 13  further comprising the step of storing data from a data source in a local cache; and retrieving the data from the cache if the data has been in the cache for less time than the maximum time period specified in the metadata specification.