Data Processing Apparatus and Method for Processing Serial Tasks

Abstract

Embodiments of the present disclosure disclose a data processing apparatus and method for processing serial tasks, which can reduce, in a data processing process, a quantity of times of reading an output result from disks of node devices in a distributed system, thereby reducing time required and network resources occupied for processing a large amount of data. The method in the present disclosure includes: acquiring at least two MapReduce tasks, where the at least two MapReduce tasks are serially arranged according to an execution sequence, and when two MapReduce tasks are serially arranged, an output value obtained after the former MapReduce task is executed is an input value of the latter MapReduce task; combining the at least two MapReduce tasks to obtain a target MapReduce task; and executing the target MapReduce task, and obtaining an output result of the target MapReduce task.

What is claimed is: 
     
         1 . A data processing apparatus for processing serial tasks comprising:
 a memory; and   a processor coupled to the memory and configured to:
 acquire MapReduce tasks comprising a first MapReduce task and a second MapReduce task, wherein the MapReduce tasks are serially arranged according to an execution sequence so that an output value obtained after the first MapReduce task is executed is an input value of the second MapReduce task; 
 combine the MapReduce tasks to obtain a target MapReduce task; 
 execute the target MapReduce task; and 
 obtain an output result of the target MapReduce task. 
   
     
     
         2 . The data processing apparatus of  claim 1 , wherein the processor is further configured to extract Map tasks from the MapReduce tasks, wherein the Map tasks comprise a first Map task and a second Map task. 
     
     
         3 . The data processing apparatus of  claim 2 , wherein the processor is further configured to combine the Map tasks to form a combined Map task, wherein a Map task output value the first Map task corresponding to the first MapReduce task is a Map task input value of the second Map task corresponding to the second MapReduce task. 
     
     
         4 . The data processing apparatus of  claim 3 , wherein the processor is further configured to obtain the target MapReduce task using the combined Map task. 
     
     
         5 . The data processing apparatus of  claim 4 , wherein the processor is further configured to: execute the combined Map task. 
     
     
         6 . The data processing apparatus of  claim 5 , wherein the processor is further configured to send an execution result of the combined Map task to a device used to execute a Reduce task. 
     
     
         7 . The data processing apparatus of  claim 6 , wherein the processor is further configured to receive from the device an output result obtained based on the execution result. 
     
     
         8 . The data processing apparatus of to  claim 7 , wherein the processor is further configured to acquire environment setup operation codes, logical operation codes, and environment cleanup operation codes from the Map tasks. 
     
     
         9 . The data processing apparatus of  claim 8 , wherein the processor is further configured to:
 compile the environment setup operation codes, the logical operation codes, and the environment cleanup operation codes; and   obtain code of the combined Map task after the compiling.   
     
     
         10 . The data processing apparatus of  claim 4 , wherein the processor is further configured to acquire environment setup operation codes, logical operation codes, and environment cleanup operation codes from the Map tasks. 
     
     
         11 . The data processing apparatus of  claim 10 , wherein the processor is further configured to:
 compile the environment setup operation codes, the logical operation codes, and the environment cleanup operation codes; and   obtain code of the combined Map task after the compiling.   
     
     
         12 . A data processing method for processing serial tasks, the method comprising:
 acquiring MapReduce tasks comprising a first MapReduce task and a second MapReduce task, wherein the MapReduce tasks are serially arranged according to an execution sequence so that an output value obtained after the first MapReduce task is executed is an input value of the second MapReduce task;   combining the MapReduce tasks to obtain a target MapReduce task executing the target MapReduce task; and   obtaining an output result of the target MapReduce task.   
     
     
         13 . The method of  claim 12 , wherein combining the MapReduce tasks comprises extracting Map tasks from the MapReduce tasks, wherein the Map tasks comprise a first Map task and a second Map task. 
     
     
         14 . The method of  claim 13 , further comprising combining the Map tasks to form a combined Map task, wherein a Map task output value the first Map task corresponding to the first MapReduce task is a Map task input value of the second Map task corresponding to the second MapReduce task. 
     
     
         15 . The method of  claim 14 , further comprising obtaining the target MapReduce task using the combined Map task. 
     
     
         16 . The method of  claim 15 , wherein executing the target MapReduce task and obtaining the output result comprise:
 executing the combined Map task;   sending an execution result of the combined Map task to a device used to execute a Reduce task; and   receiving from the device an output result obtained based on the execution result.   
     
     
         17 . The method of to  claim 16 , wherein combining the Map tasks comprises acquiring environment setup operation codes, logical operation codes, and environment cleanup operation codes from the Map tasks. 
     
     
         18 . The method of  claim 17 , wherein combining the Map tasks further comprises:
 compiling the environment setup operation codes, the logical operation codes, and the environment cleanup operation codes; and   obtaining code of the combined Map task after the compiling.   
     
     
         19 . The method of  claim 15 , wherein combining the Map tasks comprises acquiring environment setup operation codes, logical operation codes, and environment cleanup operation codes from the Map tasks. 
     
     
         20 . The method of  claim 19 , wherein combining the Map tasks further comprises:
 compiling the environment setup operation codes, the logical operation codes, and the environment cleanup operation codes; and   obtaining code of the combined Map task after the compiling.