Parallel processing of large data files on distributed file systems with dynamic workload balancing

Abstract

A computer hardware system and computer program product for performing parallel processing of files is disclosed. A file is partitioned into a set of blocks, and assignments of individual ones of the set of blocks to individual ones in a set of processing engines are recorded in a control table. In response to a request from an individual one in the set of processing engines, access is provided to the control table to identify a block assigned to the individual one in the set of processing engines. Based upon a determination that all blocks currently assigned to the individual one in the set of processing engines are exhausted an available block assigned to another one in the set of processing engines is dynamically reassigning to the individual one in the set of processing engines. The set of processing engines are configured to perform parallel processing of the set of blocks.

1 - 20 . (canceled) 
     
     
         21 . A computer hardware system configured to perform parallel processing of files, comprising:
 a hardware processor configured to initiate the following executable operations:
 partitioning a file into a set of blocks; 
 recording, in a control table, assignments of individual ones of the set of blocks to individual ones in a set of processing engines; 
 providing, in response to a request from an individual one in the set of processing engines, access to the control table to identify a block assigned to the individual one in the set of processing engines; and 
 dynamically reassigning, based upon a determination that all blocks currently assigned to the individual one in the set of processing engines are exhausted, an available block assigned to another one in the set of processing engines to the individual one in the set of processing engines, wherein 
   the set of processing engines are configured to perform parallel processing of the set of blocks.   
     
     
         22 . The system of  claim 21 , wherein
 a comparison is performed between a number unprocessed blocks in the set of blocks to a number of processing engines in the set of processing engines; and   a re-fragmentation is performed on the unprocessed blocks in the set of blocks based upon the comparison.   
     
     
         23 . The system of  claim 22 , wherein the re-fragmentation is based upon the number of unprocessed blocks being greater than the number of processing engines. 
     
     
         24 . The system of  claim 22 , wherein
 the comparison is performed upon a determination that a breathing point flag has been identified in one of the blocks in the set of blocks being processed.   
     
     
         25 . The system of  claim 22 , wherein
 the re-fragmentation creates new breathing point flags within the unprocessed blocks in the set of blocks.   
     
     
         26 . The system of  claim 22 , wherein
 the re-fragmentation of an unprocessed block is performed by a processing engine to which the unprocessed block is assigned.   
     
     
         27 . The system of  claim 22 , wherein
 the re-fragmentation causes a re-assigning of the unprocessed blocks in the set of blocks.   
     
     
         28 . The system of  claim 22 , wherein
 the re-fragmentation is performed on unprocessed portions of a block being processed.   
     
     
         29 . The system of  claim 22 , wherein
 the re-fragmentation is performed only on unprocessed whole blocks in the set of blocks.   
     
     
         30 . The system of  claim 22 , wherein
 the comparison determines whether all blocks being processed having reached a predetermined minimal block size.   
     
     
         31 . A computer program product, comprising:
 a hardware storage device having stored therein computer-executable program code for performing parallel processing of files,   the computer-executable program code, which when executed by a computer hardware system, causes the computer hardware system to perform:
 partitioning a file into a set of blocks; 
 recording, in a control table, assignments of individual ones of the set of blocks to individual ones in a set of processing engines; 
 providing, in response to a request from an individual one in the set of processing engines, access to the control table to identify a block assigned to the individual one in the set of processing engines; and 
 dynamically reassigning, based upon a determination that all blocks currently assigned to the individual one in the set of processing engines are exhausted, an available block assigned to another one in the set of processing engines to the individual one in the set of processing engines, wherein 
   the set of processing engines are configured to perform parallel processing of the set of blocks.   
     
     
         32 . The computer program product of  claim 31 , wherein
 a comparison is performed between a number unprocessed blocks in the set of blocks to a number of processing engines in the set of processing engines; and   a re-fragmentation is performed on the unprocessed blocks in the set of blocks based upon the comparison.   
     
     
         33 . The computer program product of  claim 32 , wherein the re-fragmentation is based upon the number of unprocessed blocks being greater than the number of processing engines. 
     
     
         34 . The computer program product of  claim 32 , wherein
 the comparison is performed upon a determination that a breathing point flag has been identified in one of the blocks in the set of blocks being processed.   
     
     
         35 . The computer program product of  claim 32 , wherein
 the re-fragmentation creates new breathing point flags within the unprocessed blocks in the set of blocks.   
     
     
         36 . The computer program product of  claim 32 , wherein
 the re-fragmentation of an unprocessed block is performed by a processing engine to which the unprocessed block is assigned.   
     
     
         37 . The computer program product of  claim 32 , wherein
 the re-fragmentation causes a re-assigning of the unprocessed blocks in the set of blocks.   
     
     
         38 . The computer program product of  claim 32 , wherein
 the re-fragmentation is performed on unprocessed portions of a block being processed.   
     
     
         39 . The computer program product of  claim 32 , wherein
 the re-fragmentation is performed only on unprocessed whole blocks in the set of blocks.   
     
     
         40 . The computer program product of  claim 32 , wherein
 the comparison determines whether all blocks being processed having reached a predetermined minimal block size.