Analyzing large-scale data processing jobs

Abstract

Methods, systems, and apparatus for data analysis in a distributed computing system by accessing data stored at a first processing zone associated with a distributed data processing job, detecting information identifying a particular child job associated with the distributed data processing job, comparing the identifying information to data stored at a second processing zone, and identifying an additional child job as associated with the distributed data processing job based on a result of the comparison. The methods, systems and apparatus are further for correlating particular output data associated with the particular child job and additional output data associated with the additional child job for the distributed data processing job, determining performance data for the distributed data processing job based on the output data associated with each of the particular child job and the additional child job, and providing for display the performance data for the distributed data processing job.

What is claimed is: 
     
         1 . A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
 obtaining job data associated with a distributed data processing job that has been executed at a distributed data processing environment comprising a first processing zone and a second processing zone;   identifying, based on the job data, a first child job and a second child job each created by the distributed data processing job, the first child job associated with first output data and the first processing zone, the second child job associated with second output data and the second processing zone;   determining, based on the first output data and the second output data, performance data of the distributed data processing job;   determining that the performance data satisfies a performance threshold; and   based on determining that the performance data satisfies the performance threshold, performing an action.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the operations further comprise:
 receiving, from a user of the distributed data processing environment, a user selection selecting the performance data of the distributed data processing job; and   in response to receiving the user selection selecting the performance data of the distributed data processing job, displaying the performance data at a display in communication with the data processing hardware.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein displaying the performance data comprises displaying the performance data in an interactive hierarchical structure. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the operations further comprise correlating the first output data and the second output data based on the job data. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein performing the action comprises generating a notification. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the job data comprises first job data stored at a first storage device at the first processing zone. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the job data comprises second job data stored at a second storage device at the second processing zone. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the performance data comprises a counter associated with the distributed data processing job. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the performance data comprises a change of processing environment of the distributed data processing environment. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein the first processing zone and the second processing zone each comprise a respective grouping of storage nodes. 
     
     
         11 . A system comprising:
 data processing hardware of a distributed computing system; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 obtaining job data associated with a distributed data processing job that has been executed at a distributed data processing environment comprising a first processing zone and a second processing zone; 
 identifying, based on the job data, a first child job and a second child job each created by the distributed data processing job, the first child job associated with first output data and the first processing zone, the second child job associated with second output data and the second processing zone; 
 determining, based on the first output data and the second output data, performance data of the distributed data processing job; 
 determining that the performance data satisfies a performance threshold; and 
 based on determining that the performance data satisfies the performance threshold, performing an action. 
   
     
     
         12 . The system of  claim 11 , wherein the operations further comprise:
 receiving, from a user of the distributed data processing environment, a user selection selecting the performance data of the distributed data processing job; and   in response to receiving the user selection selecting the performance data of the distributed data processing job, displaying the performance data at a display in communication with the data processing hardware.   
     
     
         13 . The system of  claim 12 , wherein displaying the performance data comprises displaying the performance data in an interactive hierarchical structure. 
     
     
         14 . The system of  claim 11 , wherein the operations further comprise correlating the first output data and the second output data based on the job data. 
     
     
         15 . The system of  claim 11 , wherein performing the action comprises generating a notification. 
     
     
         16 . The system of  claim 11 , wherein the job data comprises first job data stored at a first storage device at the first processing zone. 
     
     
         17 . The system of  claim 11 , wherein the job data comprises second job data stored at a second storage device at the second processing zone. 
     
     
         18 . The system of  claim 11 , wherein the performance data comprises a counter associated with the distributed data processing job. 
     
     
         19 . The system of  claim 11 , wherein the performance data comprises a change of processing environment of the distributed data processing environment. 
     
     
         20 . The system of  claim 11 , wherein the first processing zone and the second processing zone each comprise a respective grouping of storage nodes.