System and method for batch-normalized recurrent highway networks

Abstract

Embodiments of the present disclosure relate to a recurrent framework based on Recurrent Highway Networks (RHNs) for sequence modeling using batch normalization. In certain embodiments, constraints within the RHNs are relaxed to reduce or avoid gradient vanishing or exploding by normalizing the current transition units in highway layers.

1 . A computer-implemented method, comprising:
 loading a pre-trained model into a computing network;   receiving a digital image input, the digital image input containing one or more objects;   batch normalizing the digital image input;   applying one or more non-linear transforms to the digital input image to generate one or more transform outputs from the computing network; and   combining the one or more transform outputs to generate a final output from the computing network.   
     
     
         2 . The method of  claim 1  wherein the pre-trained model contains parameters for the computing network nodes. 
     
     
         3 . The method of  claim 2  wherein the computing network is a multi-layer neural network. 
     
     
         4 . The method of  claim 3  wherein the multi-layer neural network is a Recurrent Highway Network. 
     
     
         5 . The method of  claim 4  wherein the multi-layer neural network is a batch-normalized Recurrent Highway Network. 
     
     
         6 . The method of  claim 5  wherein the constraints within the Recurrent Highway Network are relaxed to reduce or avoid gradient vanishing or exploding by normalizing the current transition units in highway layers. 
     
     
         7 . The method of  claim 1  wherein the final output contains a caption describing the content of the digital image input. 
     
     
         8 . A computer-implemented method, comprising:
 receiving a digital image input, the digital image input containing one or more objects;   batch normalizing the digital image input;   applying one or more non-linear transforms to the digital input image to generate one or more transform outputs; and   combining the one or more transform outputs to generate a final output.   
     
     
         9 . The computer-implemented method of  claim 8 , wherein batch normalizing occurs at one or more highway layers of a recurrent highway network. 
     
     
         10 . The computer-implemented method of  claim 9 , wherein the one or more highway layers comprise a transform gate and a carry gate. 
     
     
         11 . The computer-implemented method of  claim 10 , wherein the transform gate is coupled to the carry gate. 
     
     
         12 . The computer-implemented method of  claim 9 , wherein the digital image input is an output of a prior processing loop in the recurrent highway network. 
     
     
         13 . The computer-implemented method of  claim 8 , wherein batch normalizing the digital image input comprises extracting a mean and a variance across each channel and spatial location. 
     
     
         14 . The computer-implemented method of  claim 13 , wherein batch normalizing the digital image input comprises subtracting the mean of the digital image input and dividing by the variance of the digital image input.