Method and system for monitoring data quality and dependency

Abstract

The present teaching relates to monitoring data in a plurality of data sources of heterogeneous types. In one example, a request is received for monitoring data in the data sources of heterogeneous types. One or more metrics are determined based on the request. The request is converted into one or more queries based on the one or more metrics. Each of the one or more queries is directed to at least one of the data sources of heterogeneous types. A monitoring task is created for monitoring the data in the data sources based on the one or more queries in response to the request.

We claim: 
     
         1 . A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for monitoring data in a plurality of data sources of heterogeneous types, the method comprising:
 receiving a request for monitoring data in the data sources of heterogeneous types;   determining one or more metrics based on the request;   converting the request into one or more queries based on the one or more metrics, wherein each of the one or more queries is directed to at least one of the data sources of heterogeneous types; and   creating a monitoring task for monitoring the data in the data sources based on the one or more queries in response to the request.   
     
     
         2 . The method of  claim 1 , further comprising:
 storing the monitoring task associated with the user into a monitoring task database; and/or   executing the monitoring task according to a pre-determined schedule.   
     
     
         3 . The method of  claim 1 , further comprising:
 obtaining one or more inputs from the user with respect to the monitoring task;   generating one or more alert conditions associated with the monitoring task based on the one or more inputs; and   storing the one or more alert conditions, wherein an alert will be generated and sent to the user if one of the one or more alert conditions is met after the monitoring task is executed.   
     
     
         4 . The method of  claim 1 , wherein the determining comprises:
 determining one or more collections of data in the data source;   providing the one or more collections of data to the user;   receiving one or more inputs associated with at least one of the one or more collections of data from the user; and   determining one or more metrics based on the one or more inputs.   
     
     
         5 . The method of  claim 1 , wherein the determining comprises:
 determining one or more existing monitoring tasks associated with the user;   providing the one or more existing monitoring tasks to the user;   receiving one or more inputs from the user; and   determining one or more metrics based on the one or more inputs, wherein at least one of the one or more metrics is determined based on at least one of the one or more existing monitoring tasks.   
     
     
         6 . The method of  claim 1 , wherein the determining comprises:
 determining one or more other users associated with the user;   determining one or more shared monitoring tasks that are associated with the one or more other users and shared with the user by the one or more other users;   providing the one or more shared monitoring tasks to the user;   receiving one or more inputs from the user; and   determining one or more metrics based on the one or more inputs, wherein at least one of the one or more metrics is determined based on at least one of the one or more shared monitoring tasks.   
     
     
         7 . The method of  claim 1 , wherein the creating further comprises customizing the one or more queries based on structure of the data source, and the structure includes at least one of: Hive, HBase, Oozie, and HDFS (Hadoop Distributed File System). 
     
     
         8 . The method of  claim 1 , further comprising:
 collecting first information related to the data sources;   collecting second information related to one or more jobs associated with the data sources;   determining one or more relationships among the data sources and the one or more jobs, based on the first and second information; and   generating a data dependency graph based on the one or more relationships.   
     
     
         9 . The method of  claim 8 , wherein the data dependency graph comprises:
 a first set of nodes each of which representing one of the data sources;   a second set of nodes each of which representing one of the one or more jobs; and   a set of directed links each of which connecting two nodes and representing a dependency relationship between the two nodes.   
     
     
         10 . A system, having at least one processor, storage, and a communication platform connected to a network for monitoring data in a plurality of data sources of heterogeneous types, the system comprising:
 a user request receiver configured for receiving a request for monitoring data in the data sources of heterogeneous types;   a metrics determiner configured for determining one or more metrics based on the request;   a query generator configured for converting the request into one or more queries based on the one or more metrics, wherein each of the one or more queries is directed to at least one of the data sources of heterogeneous types; and   a monitoring task generator configured for creating a monitoring task for monitoring the data in the data sources based on the one or more queries in response to the request.   
     
     
         11 . The system of  claim 10 , further comprising:
 a monitoring task managing unit configured for storing the monitoring task associated with the user into a monitoring task database; and/or   a monitoring task executor configured for executing the monitoring task according to a pre-determined schedule.   
     
     
         12 . The system of  claim 10 , further comprising:
 a user input analyzer configured for obtaining one or more inputs from the user with respect to the monitoring task; and   an alert condition generator configured for generating one or more alert conditions associated with the monitoring task based on the one or more inputs and storing the one or more alert conditions, wherein an alert will be generated and sent to the user if one of the one or more alert conditions is met after the monitoring task is executed.   
     
     
         13 . The system of  claim 10 , further comprising:
 a data source identifier configured for determining one or more collections of data in the data source;   a user interface generator configured for providing the one or more collections of data to the user; and   a user input analyzer configured for receiving one or more inputs associated with at least one of the one or more collections of data from the user, wherein the one or more metrics are determined based on the one or more inputs.   
     
     
         14 . The system of  claim 10 , further comprising:
 an existing task extractor configured for determining one or more existing monitoring tasks associated with the user;   a user interface generator configured for providing the one or more existing monitoring tasks to the user; and   a user input analyzer configured for receiving one or more inputs from the user, wherein the one or more metrics are determined based on the one or more inputs, and wherein at least one of the one or more metrics is determined based on at least one of the one or more existing monitoring tasks.   
     
     
         15 . The system of  claim 10 , further comprising:
 a data source crawler configured for collecting first information related to the data sources;   a pipeline crawler configured for collecting second information related to one or more jobs associated with the data sources;   a data/job relationship determiner configured for determining one or more relationships among the data sources and the one or more jobs, based on the first and second information; and   a dependency graph generator configured for generating a data dependency graph based on the one or more relationships.   
     
     
         16 . The system of  claim 15 , wherein the data dependency graph comprises:
 a first set of nodes each of which representing one of the data sources;   a second set of nodes each of which representing one of the one or more jobs; and   a set of directed links each of which connecting two nodes and representing a dependency relationship between the two nodes.   
     
     
         17 . A machine-readable, non-transitory and tangible medium having information recorded thereon for monitoring data in a plurality of data sources of heterogeneous types, wherein the information, when read by the machine, causes the machine to perform the following:
 receiving a request for monitoring data in the data sources of heterogeneous types;   determining one or more metrics based on the request;   converting the request into one or more queries based on the one or more metrics, wherein each of the one or more queries is directed to at least one of the data sources of heterogeneous types; and   creating a monitoring task for monitoring the data in the data sources based on the one or more queries in response to the request.   
     
     
         18 . The medium of  claim 17 , when read by the machine, further causes the machine to perform the following:
 obtaining one or more inputs from the user with respect to the monitoring task;   generating one or more alert conditions associated with the monitoring task based on the one or more inputs; and   storing the one or more alert conditions, wherein an alert will be generated and sent to the user if one of the one or more alert conditions is met after the monitoring task is executed.   
     
     
         19 . The medium of  claim 17 , when read by the machine, further causes the machine to perform the following:
 collecting first information related to the data sources;   collecting second information related to one or more jobs associated with the data sources;   determining one or more relationships among the data sources and the one or more jobs, based on the first and second information; and   generating a data dependency graph based on the one or more relationships.   
     
     
         20 . The medium of  claim 19 , wherein the data dependency graph comprises:
 a first set of nodes each of which representing one of the data sources;   a second set of nodes each of which representing one of the one or more jobs; and   a set of directed links each of which connecting two nodes and representing a dependency relationship between the two nodes.