Message passing interface (MPI) collectives using multi-allgather

Abstract

Systems, methods, and devices for performing computing operations are provided. In one example, a system is described to include an endpoint belonging to a collective that is organized as a hierarchical tree. The collective includes one or more application groups that are connected to leaf nodes of the hierarchical tree, where each application-level process in the one or more application groups initiate processing of data based on an order of arrival known for at least one other application-level process joining the one or more application groups.

What is claimed is: 
     
       1. A device, comprising:
 one or more processing circuits that receive and process data as part of a collective that is organized as a hierarchical tree; 
 one or more trigger circuits that initiate the one or more processing circuits to process the data based on an order of arrival known for at least one other application-level process joining an application group; and 
 one or more sending circuits that send an output of the one or more processing circuits to an endpoint designated as a leaf node of the collective, wherein the output of the one or more processing circuits includes a result computed by the one or more processing circuits based on processing the data. 
 
     
     
       2. The device of  claim 1 , wherein the application group comprises a number, N, of application-level processes assigned thereto, wherein N is greater than one, and wherein each of the N application-level processes are assigned to the application group based on their order of arrival to the collective. 
     
     
       3. The device of  claim 2 , wherein each of the N application-level processes begin processing respective data after N application-level processes have joined the application group and wherein each of the N application-level processes do not wait for other application-level processes in other application groups to begin processing their respective data. 
     
     
       4. The device of  claim 1 , wherein the hierarchical tree comprises a root node, a plurality of leaf nodes, and a plurality of vertex nodes provided between the root node and the plurality of leaf nodes. 
     
     
       5. The device of  claim 4 , wherein the leaf node that receives the output from the one or more sending circuits also receives a second output from another application-level process belonging to the application group. 
     
     
       6. The device of  claim 4 , wherein contributions of each application-level process belonging to the application group are independent of one another and do not require aggregation prior to being transmitted to the leaf node. 
     
     
       7. The device of  claim 1 , wherein the output is provided in a size that is specified based on an operation being performed by the collective. 
     
     
       8. A device assigned to operate as a leaf node in a collective that is organized as a hierarchical tree, the device comprising:
 one or more receive circuits that receive outputs from one or more application-level processes assigned to an application group, wherein the application group is also part of the collective; 
 one or more aggregation circuits that aggregate the outputs from the application group until one of: (i) a predetermined amount of data has been aggregated; (ii) outputs from all application-level processes assigned to the application group have been received; and (iii) until a timer expires; and 
 one or more sending circuits that transmit data aggregated by the one or more aggregation circuits to a parent node in the hierarchical tree after one of: (i) the predetermined amount of data has been aggregated; (ii) the outputs from all application-level processes assigned to the application group have been received; and (iii) until the timer expires. 
 
     
     
       9. The device of  claim 8 , wherein the parent node comprises a vertex node and wherein the vertex node concatenates the data received from the one or more sending circuits with data from at least one other leaf node in the hierarchical tree. 
     
     
       10. The device of  claim 9 , wherein the hierarchical tree comprises a plurality of vertex nodes and a root node, wherein the root node receives data from the plurality of vertex nodes and transmits the data received from the plurality of vertex nodes to one or more destination endpoints. 
     
     
       11. The device of  claim 8 , wherein the timer is adjustable on a per-operation basis. 
     
     
       12. The device of  claim 8 , wherein the collective is formed to complete a collective operation. 
     
     
       13. The device of  claim 8 , wherein the one or more aggregation circuits concatenate the outputs from the application group. 
     
     
       14. A system, comprising:
 an endpoint belonging to a collective that is organized as a hierarchical tree, wherein the collective further comprises one or more application groups that are connected to leaf nodes of the hierarchical tree, wherein each application-level process in the one or more application groups initiate processing of data based on an order of arrival known for at least one other application-level process joining the one or more application groups. 
 
     
     
       15. The system of  claim 14 , wherein each endpoint in the collective follows a data exchange pattern. 
     
     
       16. The system of  claim 14 , wherein the one or more application groups comprise a first application group and a second application group, wherein a first leaf node of the hierarchical tree receives outputs from the first application group, wherein a second leaf node of the hierarchical tree receives outputs from the second application group, and wherein application-level processes of the first application group begin processing data before application-level processes of the second application group begin processing data. 
     
     
       17. The system of  claim 16 , wherein the application-level processes of the first application group join the collective prior to the application-level processes of the second application group. 
     
     
       18. The system of  claim 14 , wherein the hierarchical tree comprises a root node, a plurality of leaf nodes, and a plurality of vertex nodes provided between the root node and the plurality of leaf nodes. 
     
     
       19. The system of  claim 14 , wherein contributions of each application-level process belonging to the one or more application groups are independent of one another and do not require aggregation prior to being transmitted to the leaf nodes. 
     
     
       20. The system of  claim 14 , wherein the leaf nodes aggregate outputs from the one or more application groups until a predetermined amount of data has been aggregated or until a timer expires.