Training network with batches of input instances

Abstract

Some embodiments provide a method for training a machine-trained (MT) network that processes input data using network parameters. The method maps a set of input instances to a set of output values by propagating the set of input instances through the MT network. The set of input instances include input instances for each of multiple categories. The method selects multiple input instances as anchor instances. For each anchor instance, the method computes a loss function as a comparison between the output value for the anchor instance and each output value for an input instance in a different category than the anchor. The method computes a total loss function for the MT network as a sum of the loss function computed for each anchor instance. The method trains the network parameters using the computed total loss function.

What is claimed is: 
     
       1. A method for training a machine-trained (MT) network that processes input data using a plurality of network parameters, the method comprising:
 for each input instance of a set of input instances used for a training batch, propagating the input instance through the MT network to generate a corresponding output value for the input instance, wherein the set of input instances comprises, for each respective category of a set of three or more different categories, a respective plurality of input instances; 
 for each anchor instance of a plurality of the input instances selected as anchor instances, wherein the plurality of input instances selected as anchor instances comprises instances from each of the three or more different categories:
 identifying a plurality of input instances that are in different categories than the anchor instance; and 
 computing a loss function term for the anchor instance based on (i) comparisons between the output value generated for the anchor instance and the output values generated for each of the identified input instances in different categories than the anchor instance and (ii) a calculated mean of distances between the output value generated for the anchor instance and the output values generated for each input instance in a same category as the anchor instance, 
 wherein, for each specific input instance, the output value generated for the specific input instance is used to compute (i) the loss function term for the specific input instance when the specific input instance is selected as the anchor instance and (ii) the loss function terms for each of a group of input instances in different categories than the specific input instance when the group of input instances are selected as anchor instances, without propagating the specific input instance through the MT network multiple times during the training batch; 
 
 computing a total loss function for the training batch for the MT network as a sum of the loss function terms computed for each selected anchor instance; and 
 training the network parameters using the computed total loss function. 
 
     
     
       2. The method of  claim 1 , wherein each output value is an output point in multiple dimensions, wherein the comparison between the output point generated for a particular anchor instance and the output point generated for a particular other input instance is a distance between said two output points computed by summing squares of distances between the two output points in each of the multiple dimensions. 
     
     
       3. The method of  claim 2 , wherein, for each anchor instance, the calculated mean of distances between the output value generated for the anchor instance and the output values generated for each input instance in the same category as the anchor instance is based on a distribution of distances between the output point generated for the anchor instance and the output points generated for each input instance in the same category as the anchor instance. 
     
     
       4. The method of  claim 3 , wherein the distances between the output point generated for the anchor instance and the output points generated for the input instances in the same category as the anchor instance are normally distributed. 
     
     
       5. The method of  claim 1 , wherein, for each anchor instance, computing the loss function term for the anchor instance further comprises calculating a standard deviation for the anchor instance as a function of the mean computed for the anchor instance. 
     
     
       6. The method of  claim 5 , wherein the standard deviation is further a function of a lower bound for the mean. 
     
     
       7. The method of  claim 1 , wherein the MT network is a neural network comprising input nodes, output nodes, and interior nodes between the input nodes and output nodes, wherein each node produces a node output value and each interior node and output node receives as input values a set of node output values of other nodes. 
     
     
       8. The method of  claim 7 , wherein each node of the MT network applies a set of the network parameters to each of its received input values to generate its node output value, wherein training the network parameters comprises:
 backpropagating the computed total loss function for the training batch through the MT network to determine, for each network parameter of a set of the network parameters, a rate of change in the total loss function relative to a rate of change in the network parameter; and 
 modifying each network parameter in the set according to the determined rate of change for the network parameter. 
 
     
     
       9. The method of  claim 1 , wherein each input instance, of the set of input instances that comprises input instances from each of the three or more different categories, is selected as one of the anchor instances. 
     
     
       10. The method of  claim 9 , wherein, for each respective anchor instance, the respective identified plurality of input instances comprises all of the input instances in the set of input instances that are not in the same category as the respective anchor instance. 
     
     
       11. The method of  claim 1 , wherein:
 the set of input instances comprises a set of input images; 
 the three or more different categories comprise different types of objects represented in the input images; and 
 for each input image, the corresponding output value generated by the MT network correlates to a particular category of object identified in the image by the MT network. 
 
     
     
       12. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a machine-trained (MT) network that processes input data using a plurality of network parameters, the program comprising sets of instructions for:
 for each input instance of a set of input instances used for a training batch, propagating the input instance through the MT network to generate a corresponding output value for the input instance, wherein the set of input instances comprises, for each respective category of a set of three or more different categories, a respective plurality of input instances; 
 for each anchor instance of a plurality of the input instances selected as anchor instances, wherein the plurality of input instances selected as anchor instances comprises instances from each of the three or more different categories:
 identifying a plurality of input instances that are in different categories than the anchor instance; and 
 computing a loss function term for the anchor instance based on (i) comparisons between the output value generated for the anchor instance and the output values generated for each of the identified input instances in different categories than the anchor instance and (ii) a calculated mean of distances between the output value generated for the anchor instance and the output values generated for each input instance in a same category as the anchor instance, 
 wherein, for each specific input instance, the output value generated for the specific input instance is used to compute (i) the loss function term for the specific input instance when the specific input instance is selected as the anchor instance and (ii) the loss function terms for each of a group of input instances in different categories than the specific input instance when the group of input instances are selected as anchor instances, without propagating the specific input instance through the MT network multiple times during the training batch; 
 
 computing a total loss function for the training batch for the MT network as a sum of the loss function terms computed for each selected anchor instance; and 
 training the network parameters using the computed total loss function. 
 
     
     
       13. The non-transitory machine-readable medium of  claim 12 , wherein each output value is an output point in multiple dimensions, wherein the comparison between the output point generated for a particular anchor instance and the output point generated for a particular other input instance is a distance between said two output points computed by summing squares of distances between the two output points in each of the multiple dimensions. 
     
     
       14. The non-transitory machine-readable medium of  claim 13 , wherein, for each anchor instance, the calculated mean of distances between the output value generated for the anchor instance and the output values generated for each input instance in the same category as the anchor instance is based on a distribution of distances between the output point generated for the anchor instance and the output points generated for each input instance in the same category as the anchor instance. 
     
     
       15. The non-transitory machine-readable medium of  claim 11 , wherein the set of instructions for computing, for each anchor instance, the loss function term for the anchor instance comprises a set of instructions for calculating a standard deviation for the anchor instance as a function of the mean computed for the anchor instance. 
     
     
       16. The non-transitory machine-readable medium of  claim 15 , wherein the standard deviation is further a function of a lower bound for the mean. 
     
     
       17. The non-transitory machine-readable medium of  claim 12 , wherein the MT network is a neural network comprising input nodes, output nodes, and interior nodes between the input nodes and output nodes, wherein each node produces a node output value and each interior node and output node receives as input values a set of node output values of other nodes. 
     
     
       18. The non-transitory machine-readable medium of  claim 17 , wherein each node of the MT network applies a set of the network parameters to each of its received input values to generate its node output value, wherein the set of instructions for training the network parameters comprises sets of instructions for:
 backpropagating the computed total loss function for the training batch through the MT network to determine, for each network parameter of a set of the network parameters, a rate of change in the total loss function relative to a rate of change in the network parameter; and 
 modifying each network parameter in the set according to the determined rate of change for the network parameter.