Wavelet representation for accelerated deep learning

Abstract

Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements performs flow-based computations on wavelets of data. Each processing element has a compute element with dedicated storage and a routing element. Each router enables communication with nearest neighbors in a 2D mesh. The communication is via wavelets in accordance with a representation comprising an index specifier, a virtual channel specifier, a task specifier, a data element specifier, and an optional control/data specifier. The virtual channel specifier and the task specifier are associated with one or more instructions. The index specifier and the data element are optionally associated with operands of the one or more instructions.

1 . (canceled) 
     
     
         2 . A method comprising:
 via a fabric, sending a wavelet from a sending processing element to a receiving processing element, the wavelet comprising a color field, a control field, and an optional index field, wherein the sending is in accordance with the color field;   in the receiving processing element and responsive to the wavelet being chosen for processing, conditionally executing one or more instructions beginning at an instruction address; and   in the receiving processing element, determining the instruction address based at least in part on the control field, the optional index field, and the color field.   
     
     
         3 . The method of  claim 2 , further comprising:
 in the receiving processing element, conditionally determining the instruction address
 based at least in part on the optional index field responsive to the control field indicating a first state, and 
 based at least in part on the color field responsive to the control field indicating a second state. 
   
     
     
         4 . The method of  claim 2 , wherein the sending is further in accordance with a virtual channel specified by the color field. 
     
     
         5 . The method of  claim 2 , wherein the wavelet further comprises a data field and further comprising in the receiving processing element and responsive to the wavelet being chosen for processing, conditionally using information from the data field as an input operand to an operation performed by the receiving processing element. 
     
     
         6 . The method of  claim 5 , wherein the information is in accordance with one of a floating-point value and an integer value. 
     
     
         7 . The method of  claim 5 , wherein the wavelet is in accordance with one of a plurality of mutually exclusive formats comprising a sparse format and a dense format, the sparse format is in accordance with the wavelet comprising the optional index field and the data field comprising a single data element, the dense format is in accordance with the data field comprising a plurality of data elements; and further comprising, in the receiving processing element, accessing a data-indicator to determine which of the formats the wavelet is in accordance with. 
     
     
         8 . The method of  claim 7 , wherein the data-indicator is enabled to indicate a mutually exclusive one of a single-instruction-single-data state and a single-instruction-multiple-data state. 
     
     
         9 . The method of  claim 7 , further comprising, responsive to the wavelet being in accordance with the dense format, the conditionally using information from the data field comprises using the plurality of data elements as the input operand. 
     
     
         10 . The method of  claim 2 , wherein the conditionally executing is dependent on a wavelet-indicator enabled to indicate a mutually exclusive one of a task-initiation state and a data-processing state, and further comprising, in the receiving processing element, setting the wavelet-indicator to the task-initiation state responsive to executing a terminate instruction and setting the wavelet-indicator to the data-processing state responsive to initiating a task. 
     
     
         11 . The method of  claim 2 , wherein the sending processing element and the receiving processing element are fabricated via wafer-scale integration, the method is performed via a system using a significant portion of a wafer as an element of the system, and the wafer comprises the sending processing element and the receiving processing element. 
     
     
         12 . The method of  claim 2 , wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network. 
     
     
         13 . A system comprising:
 means for sending a wavelet from a sending processing element to a receiving processing element via a fabric, the wavelet comprising a color field, a control field, and an optional index field, wherein the means for sending is operable in accordance with the color field;   in the receiving processing element and responsive to the wavelet being chosen for processing, means for conditionally executing one or more instructions beginning at an instruction address; and   in the receiving processing element, means for determining the instruction address based at least in part on the control field, the optional index field, and the color field.   
     
     
         14 . The system of  claim 13 , further comprising:
 in the receiving processing element, means for conditionally determining the instruction address
 based at least in part on the optional index field responsive to the control field indicating a first state, and 
 based at least in part on the color field responsive to the control field indicating a second state. 
   
     
     
         15 . The system of  claim 13 , wherein the means for sending is further operable in accordance with a virtual channel specified by the color field. 
     
     
         16 . The system of  claim 13 , wherein the wavelet further comprises a data field and further comprising in the receiving processing element and responsive to the wavelet being chosen for processing, means for conditionally using information from the data field as an input operand to an operation performed by the receiving processing element. 
     
     
         17 . The system of  claim 16 , wherein the wavelet is in accordance with one of a plurality of mutually exclusive formats comprising a sparse format and a dense format, the sparse format is in accordance with the wavelet comprising the optional index field and the data field comprising a single data element, the dense format is in accordance with the data field comprising a plurality of data elements; and further comprising, in the receiving processing element, means for accessing a data-indicator to determine which of the formats the wavelet is in accordance with. 
     
     
         18 . The system of  claim 17 , wherein the data-indicator is enabled to indicate a mutually exclusive one of a single-instruction-single-data state and a single-instruction-multiple-data state. 
     
     
         19 . The system of  claim 13 , wherein the means for conditionally executing is responsive to a wavelet-indicator enabled to indicate a mutually exclusive one of a task-initiation state and a data-processing state, and further comprising, in the receiving processing element, means for setting the wavelet-indicator to the task-initiation state responsive to executing a terminate instruction and means for setting the wavelet-indicator to the data-processing state responsive to initiating a task. 
     
     
         20 . The system of  claim 13 , wherein the sending processing element and the receiving processing element are fabricated via wafer-scale integration, the system uses a significant portion of a wafer as an element of the system, and the wafer comprises the sending processing element and the receiving processing element. 
     
     
         21 . The system of  claim 13 , wherein the sending processing element and the receiving processing element implement at least portions of a neuron of a neural network.