Systems and methods for controlling transformers in artificial intelligence

Abstract

A system and a method for loop exit are disclosed. A unit scheduler includes a command queue, a report queue, an interface, and a unit controller. The command queue is configured to store commands for execution in an iterative procedure for an application using a transformer model with a multi-head attention (MHA) mechanism and a decoder. The report queue is configured to store status reports on the execution of the commands. The interface is configured to communicate with a host processor to receive the commands and to transmit the status reports. The unit controller is configured to determine a change of the iterative procedure based on a loop exit condition being met. The unit controller reports the loop exit condition in the report queue.

What is claimed is: 
     
         1 . A device comprising:
 a device command queue configured to store commands for execution in an iterative procedure for an application using a transformer model with a multi-head attention (MHA) mechanism and a decoder;   a device report queue configured to store status reports on the execution of the commands;   an interface configured to communicate with a host processor to receive the commands and to transmit the status reports;   a device controller configured to determine a change of the iterative procedure based on a loop exit condition being met,   wherein the device controller reports the loop exit condition in the device report queue.   
     
     
         2 . The device of  claim 1 , wherein the loop exit condition is from a comparison between a similarity measure and a threshold. 
     
     
         3 . The device of  claim 2 , wherein the similarity measure is determined by a cosine similarity. 
     
     
         4 . The device of  claim 2 , wherein the similarity measure is obtained from at least one result of the decoder in the iterative procedure. 
     
     
         5 . The device of  claim 1 , wherein the threshold is based at least in part on the application. 
     
     
         6 . The device of  claim 1 , wherein the threshold is set adaptively. 
     
     
         7 . The device of  claim 1  further comprising:
 a computational element configured to perform calculations in the iterative procedure and the loop exit condition. 
 
     
     
         8 . The device of  claim 1 , wherein the host processor includes a host scheduler having a host command queue to store the commands and a host report queue configured to store the status reports on the execution of the commands. 
     
     
         9 . The device of  claim 8 , wherein the host scheduler issues the commands from the host command queue. 
     
     
         10 . The device of  claim 8 , wherein the host scheduler reads the status reports and determines the change of the iterative procedure. 
     
     
         11 . A method comprising:
 communicating with a host processor to receive commands for execution in an iterative procedure for an application using a transformer model with a multi-head attention (MHA) mechanism and a decoder;   storing the commands in a device command queue;   receiving a control word from the host processor, the control word having a loop exit (LE) enable flag;   performing operations in the iterative procedure;   determining a change of the iterative procedure based on an LE condition of the operations being met and the LE enable flag; and   reporting status of the LE condition to the host processor.   
     
     
         12 . The method of  claim 11 , wherein the loop exit condition is from a comparison between a similarity measure and a threshold. 
     
     
         13 . The method of  claim 12 , wherein the similarity measure is determined by a cosine similarity. 
     
     
         14 . The method of  claim 12 , wherein the similarity measure is obtained from at least one result of the decoder in the iterative procedure. 
     
     
         15 . The method of  claim 11 , wherein the threshold is based at least in part on the application. 
     
     
         16 . The method of  claim 11 , wherein the threshold is set adaptively. 
     
     
         17 . The method of  claim 11  further comprising:
 performing calculations in the iterative procedure and the loop exit condition. 
 
     
     
         18 . The method of  claim 11 , wherein the host processor includes a host scheduler having a host command queue to store the commands and a host report queue configured to store the status reports on the execution of the commands. 
     
     
         19 . The method of  claim 18 , wherein the host scheduler issues the commands from the host command queue. 
     
     
         20 . A system comprising:
 a host processor having a host scheduler, the host scheduler comprising:
 a host command queue configured to store commands for execution in an iterative procedure for an application using a transformer model with a multi-head attention (MHA) mechanism and a decoder; 
 a host report queue configured to store status reports on the execution of the commands; 
 a host controller configured to issue the commands from the command queue to at least one processing unit via the interface and to read the status reports from the report queue, 
 wherein the host controller determines a change of the iterative procedure based on a loop exit condition reported in the status reports; and 
   at least one processing unit, the at least one processing unit having a unit scheduler configured to determine the change of the iterative procedure based on the loop exit condition being met.