Data management platform, intelligent defect analysis system, intelligent defect analysis method, computer-program product, and method for defect analysis

Abstract

A data management platform for intelligently managing data is provided. The data management platform includes an ETL module configured to extract, cleanse, transform, or load data; a data lake configured to store a first group of data formed by extracting raw data from a plurality of data sources by the ETL module; a data warehouse configured to store a second group of data formed by cleansing and standardizing on the first group of data; a general data layer configured to store a third group of data formed by subjecting the second group of data to data fusion; and a data mart configured to store a fourth group of data formed by transforming the third group of data by the ETL module. The general data layer is a distributed data storage storing information available for querying. The data mart is a database of NoSQL type storing information available for computational processing.

What is claimed is: 
     
         1 . A data management platform for intelligently managing data, comprising:
 an ETL module configured to extract, cleanse, transform, or load data;   a data lake configured to store a first group of data formed by extracting raw data from a plurality of data sources by the ETL module, the first group of data having same contents as the raw data;   a data warehouse configured to store a second group of data formed by cleansing and standardizing on the first group of data by the ETL module;   a general data layer configured to store a third group of data formed by subjecting the second group of data to data fusion by the ETL module, wherein the general data layer is a distributed data storage storing information available for querying; and   a data mart configured to store a fourth group of data formed by transforming the third group of data by the ETL module, wherein the data mart is a database of NoSQL type storing information available for computational processing.   
     
     
         2 . The data management platform of  claim 1 , wherein the fourth group of data comprising a plurality of sub-tables having index relationship in which data are categorized based on types and/or rules, the fourth group of data available for querying or computational processing. 
     
     
         3 . The data management platform of  claim 1 , wherein the first group of data stored in the data lake is dynamically updated; and
 the first group of data comprises a real-time updated data stored in a Kudu™-based database, or a periodically updated data stored in a Hadoop distributed file system.   
     
     
         4 . The data management platform of  claim 1 , wherein at least a portion of the raw data in the plurality of data sources is in a binary large object format;
 at least a portion of the first group of data extracted onto and stored in the data lake is in a compressed hexadecimal format; and   at least a portion of the second group of data extracted onto and stored in the data warehouse is obtained by decompressing and processing extracted data.   
     
     
         5 . The data management platform of  claim 1 , wherein the ETL module is further configured to generate a dynamically updated table that is automatically updated periodically; and
 the general data layer is configured to store the dynamically updated table comprising information on defects of interest.   
     
     
         6 . The data management platform of  claim 1 , wherein the data mart is configured to store a plurality of sub-tables split from a respective one of data tables stored in the general data layer;
 wherein the plurality of sub-tables comprises:   a first sub-table comprising a plurality of environmental factors corresponding to user-defined analysis criteria in an interactive task sub-interface in communication with the data management platform;   a second sub-table comprising product serial numbers; and   a third sub-table comprising values in the third group of data that correspond to at least one of the product serial numbers.   
     
     
         7 . The data management platform of  claim 1 , wherein the general data layer is configured to performing data fusion on at least two of parameter feature information, biographical information of a manufacturing process, and defect information associated therewith. 
     
     
         8 . An intelligent defect analysis system, comprising:
 a distributed computing system comprising one or more networked computers configured to execute in parallel to perform at least one common task; and   one or more computer readable storage mediums storing instructions that, when executed by the distributed computing system, cause the distributed computing system to execute software modules;   wherein the software modules comprise:   a data management platform configured to intelligently extract, transform, or load raw data from a plurality of data sources into a managed data, wherein the raw data and the managed data comprise defect information, and the managed data is stored in a distributed manner;   an analyzer configured to perform defect analysis upon receiving a task request, the analyzer comprising a plurality of algorithm servers configured to obtain the managed data from the data management platform and perform algorithm analysis on the managed data to derive a result data on underlying reasons for defects; and   a data visualization and interaction interface configured to generate the task requests and display the result data.   
     
     
         9 . The intelligent defect analysis system of  claim 8 , wherein the data management platform comprises an ETL module configured to extract, transform, or load data from the plurality of data sources onto a data mart that is a database of NoSQL type; and
 upon receiving an assigned task, a respective one of the plurality of algorithm servers is configured to obtain a first data from the data mart.   
     
     
         10 . The intelligent defect analysis system of  claim 9 , wherein the ETL module is further configured to extract, transform, or load data from the plurality of data sources onto a general data layer that is a distributed data storage storing information;
 upon performing defect analysis, the respective one of the plurality of algorithm servers is configured to transmit a second data to the general data layer;   the ETL module is configured to generate a dynamically updated table that is automatically updated periodically; and   the general data layer is configured to store the dynamically updated table.   
     
     
         11 . The intelligent defect analysis system of  claim 10 , wherein the software modules further comprise a query engine connected to the data management platform and configured to obtain the managed data from the data management platform. 
     
     
         12 . The intelligent defect analysis system of  claim 11 , wherein the data visualization and interaction interface is configured to generate a task request;
 the analyzer further comprises a plurality of business servers;   upon receiving the task request, the one or more of the plurality of business servers are configured to transmit a query task request to the query engine;   the query engine, upon receiving the query task request from the one or more of the plurality of business servers, is configured to query the dynamically updated table to obtain information on defects of interest, and transmit the information on defects of interest to one or more of the plurality of business servers;   upon receiving defect analysis tasks, the one or more of the plurality of algorithm servers are configured to obtain the first data from the data mart to perform defect analysis; and   upon completion of the defect analysis, the one or more of the plurality of algorithm servers are configured to transmit results of the defect analysis to the general data layer.   
     
     
         13 . The intelligent defect analysis system of  claim 12 , wherein the task request is an automatically recurring task request, the automatically recurring task request defining a recurring period for which the defect analysis is to be performed;
 the query engine is configured to query the dynamically updated table to obtain the information on defects of interest limited to the recurring period; and   upon receiving the information on defects of interest during the recurring period, the one or more of the plurality of business servers are configured to generate the defect analysis tasks based on the information on defects of interest during the recurring period.   
     
     
         14 . The intelligent defect analysis system of  claim 12 , wherein the task request is an interactive task request;
 the data visualization and interaction interface is configured to receive a user-defined analysis criteria, and configured to generate the interactive task request based on the user-defined analysis criteria;   upon receiving the information on defects of interest, the one or more of the plurality of business servers are configured to transmit the information to the data visualization and interaction interface;   the data visualization and interaction interface is configured to display the information on defects of interest and a plurality of environmental factors associated with the defects of interest, and configured to receive a user-defined selection of one or more environmental factors from the plurality of environmental factors, and transmit the user-defined selection to the one or more of the plurality of business servers; and   the one or more of the plurality of business servers are configured to generate the defect analysis tasks based on the information and the user-defined selection.   
     
     
         15 . The intelligent defect analysis system of  claim 11 , wherein the analyzer further comprises a plurality of business servers; and
 the software modules further comprise a load balancer connected to the analyzer, the load balancer configured to receive task requests and configured to assign the task requests to one or more of the plurality of business servers to achieve load balance among the plurality of business servers, and configured to assign tasks from the plurality of business servers to one or more of the plurality of algorithm servers to achieve load balance among the plurality of algorithm servers.   
     
     
         16 . The intelligent defect analysis system of  claim 15 , wherein the task requests are assigned to each of the one or more of the plurality of business servers based on a number of tasks currently performed by each of the plurality of business servers and an amount of computational burden required by each task currently performed by each of the plurality of business servers, to achieve an optimal load balance among the plurality of business servers. 
     
     
         17 . The intelligent defect analysis system of  claim 15 , wherein the data visualization and interaction interface is configured to generate a task request;
 the load balancer is configured to receive the task request and configured to assign the task request to one or more of the plurality of business servers to achieve load balance among the plurality of business servers;   the one or more of the plurality of business servers are configured to transmit a query task request to the query engine;   the query engine, upon receiving the query task request from the one or more of the plurality of business servers, is configured to query the dynamically updated table to obtain information on defects of interest, and transmit the information on defects of interest to one or more of the plurality of business servers;   the one or more of the plurality of business servers are configured to transmit defect analysis tasks to the load balancer for assigning the defect analysis tasks to the one or more of the plurality of algorithm servers to achieve load balance among the plurality of algorithm servers;   upon receiving the defect analysis tasks, the one or more of the plurality of algorithm servers are configured to query the first data from the data mart to perform defect analysis; and   upon completion of the defect analysis, the one or more of the plurality of algorithm servers are configured to transmit results of the defect analysis to the general data layer.   
     
     
         18 . The intelligent defect analysis system of  claim 15 , wherein the analyzer further comprises a cache server connected to the plurality of business servers and the query engine; and
 the cache server is configured to store a portion of results of previously performed defect analysis tasks in a cache.   
     
     
         19 . The intelligent defect analysis system of  claim 18 , wherein the data visualization and interaction interface comprises a defect visualization sub-interface;
 the defect visualization sub-interface is configured to receive a user-defined selection of a defect to be analyzed and generate a call request;   the load balancer is configured to receive the call request and configured to assign the call request to one or more of the plurality of business servers to achieve load balance among the plurality of business servers;   the one or more of the plurality of business servers are configured to transmit the call request to the cache server; and   the cache server is configured to determine whether information on the defect to be analyzed is stored in the cache.   
     
     
         20 . An intelligent defect analysis method performed by a distributed computing system comprising one or more networked computers configured to execute in parallel to perform at least one common task;
 the method comprising:   executing a data management platform configured to intelligently extract, transform, or load raw data from a plurality of data sources into a managed data, wherein the raw data and the managed data comprise defect information, and the managed data is stored in a distributed manner;   executing an analyzer configured to perform defect analysis upon receiving a task request, the analyzer comprising a plurality of algorithm servers configured to obtain the managed data from the data management platform and perform algorithm analysis on the managed data to derive a result data on underlying reasons for defects; and   executing a data visualization and interaction interface configured to generate the task requests and display the result data.