Deep learning via dynamic root solvers

Abstract

The present invention provides a computer implemented method, system, and computer program product of deep learning via dynamic root solvers. In an embodiment, the present invention includes (1) forming an initial set of GPUs into an initial binary tree architecture, where the initial set includes initially idle GPUs and an initial root solver GPU as the root of the initial binary tree architecture, (2) calculating initial gradients and initial adjusted weight data, (3) choosing a first currently idle GPU as a current root solver GPU, (4) forming a current set of GPUs into a current binary tree architecture, where the current set includes the additional currently idle GPUs and the current root solver GPU as the root of the current binary tree architecture, (5) calculating current gradients and current adjusted weight data, and (6) transmitting an initial update to the weight data to the available GPUs.

What is claimed is: 
     
         1 . A computer implemented method comprising:
 identifying, by a host computer processor, graphic processor units (GPUs) that are available (available GPUs);   identifying, by the host computer processor, GPUs that are idle (initially idle GPUs) among the available GPUs for an initial iteration of deep learning,
 wherein the identifying GPUs that are idle among the available GPUs comprises executing, by the host computer processor, a run command from a central processing unit (CPU) of each of the available GPUs to determine a percentage of the each of the available GPUs being utilized; 
   choosing, by the host computer processor, one of the initially idle GPUs as an initial root solver GPU for the initial iteration;   initializing, by the host computer processor, weight data for an initial set of multidimensional data;   transmitting, by the host computer processor, the initial set of multidimensional data to the available GPUs;   forming, by the host computer processor, an initial set of GPUs into an initial binary tree architecture, wherein the initial set comprises the initially idle GPUs and the initial root solver GPU, wherein the initial root solver GPU is the root of the initial binary tree architecture,
 wherein the forming the initial set of GPUs into the initial binary tree architecture comprises logically connecting, by the host computer processor, a first GPU among the initially idle GPUs as a leaf node to a second GPU among the initially idle GPUs as a parent node if a fast communication link exists between the first GPU and the second GPU, 
 wherein the fast communication link comprises a peer-to-peer connection; 
   calculating, by the initial set of GPUs, initial gradients and a set of initial adjusted weight data with respect to the weight data and the initial set of multidimensional data via the initial binary tree architecture;   in response to the calculating the initial gradients and the initial adjusted weight data, identifying, by the host computer processor, a first GPU among the available GPUs to become idle (first currently idle GPU) for a current iteration of deep learning;   choosing, by the host computer processor, the first currently idle GPU as a current root solver GPU for the current iteration;   transmitting, by the host computer processor, a current set of multidimensional data to the current root solver GPU;   in response to the identifying the first currently idle GPU, identifying, by the host computer processor, additional GPUs that are currently idle (additional currently idle GPUs) among the available GPUs;   transmitting, by the host computer processor, the current set of multidimensional data to the additional currently idle GPUs;   forming, by the host computer processor, a current set of GPUs into a current binary tree architecture, wherein the current set comprises the additional currently idle GPUs and the current root solver GPU, wherein the current root solver GPU is the root of the current binary tree architecture;   calculating, by the current set of GPUs, current gradients and a set of current adjusted weight data with respect to at least the weight data and the current set of multidimensional data via the current binary tree architecture;   in response to the initial root solver GPU receiving a set of calculated initial adjusted weight data, transmitting, by the initial root solver GPU, an initial update to the weight data to the available GPUs;   in response to the current root solver GPU receiving a set of current initial adjusted weight data, transmitting, by the current root solver GPU, a current update to the weight data to the available GPUs; and   repeating the identifying, the choosing, the transmitting, the forming, and the calculating with respect to the weight data, updates to the weight data, and subsequent sets of multidimensional data, transmitting an initial update to the weight data to the available GPUs.