Forward propagation apparatus, learning apparatus, information processing system, processing method, and non-transitory computer readable medium storing program

Abstract

A forward propagation apparatus is a forward propagation apparatus for a neural network, including: a mask generation unit that generates a binary mask; and a layer execution unit that performs an operation for a sparse convolutional layer according to a value at each coordinate of the binary mask, in which the mask generation unit: generates heat maps by performing an operation for a convolutional layer on an input feature map; generates a composite heat map obtained by combining the heat maps, into one heat map by summing up values of heat maps on a coordinate-by-coordinate basis; and generates the binary mask by binarizing a value at each coordinate of the composite heat map by using a predetermined threshold.

What is claimed is: 
     
         1 . A forward propagation apparatus for a neural network, comprising:
 at least one memory storing instructions; and   at least one processor configured to execute the instructions to:   generate heat maps by performing an operation for a convolutional layer on an input feature map, the number of the heat maps being equal to the number of types of objects to be detected by the neural network;   generate a composite heat map obtained by combining the heat maps the number of which is equal to the number of types of objects to be detected, into one heat map by summing up values of heat maps on a coordinate-by-coordinate basis;   generate a binary mask by binarizing a value at each coordinate of the composite heat map by using a predetermined threshold; and   perform an operation for a sparse convolutional layer according to a value at each coordinate of the binary mask.   
     
     
         2 . The forward propagation apparatus according to  claim 1 , wherein a value of a weight of the convolutional layer for generating heat maps the number of which is equal to the number of types of objects to be detected, is a value that is machine-learned by using a heat map generated based on a correct answer label. 
     
     
         3 . The forward propagation apparatus according to  claim 1 , wherein the processor is further configured to execute the instructions to change a resolution of the composite heat map according to a resolution of the sparse convolutional layer. 
     
     
         4 . The forward propagation apparatus according to  claim 1 , wherein the processor is further configured to execute the instructions to generate the binary mask for an operation for a final layer of consecutive sparse convolutional layers, and further generates-generate a binary mask for an operation for a sparse convolutional layer preceding the final layer from the generated binary mask. 
     
     
         5 . The forward propagation apparatus according to  claim 1 , wherein the number of types of objects to be detected is the number of types of joint points of a human being coordinates of which are detected by the neural network. 
     
     
         6 . The forward propagation apparatus according to  claim 1 , wherein the number of types of objects to be detected is the number of classes used in object detection by the neural network. 
     
     
         7 . The forward propagation apparatus according to  claim 1 , wherein the number of types of objects to be detected is the number of classes used in semantic segmentation by the neural network. 
     
     
         8 . A learning apparatus for a forward propagation apparatus for a neural network, comprising:
 at least one first memory storing instructions; and   at least one first processor configured to execute the instructions to:   acquire first heat maps generated by the forward propagation apparatus, the number of the first heat maps being equal to the number of types of objects to be detected by the neural network;   acquire second heat maps generated based on correct answer labels of the objects to be detected, the number of the second heat maps being equal to the number of the types of objects to be detected;   calculate a difference between the first heat maps and the second heat maps; and   update a weighting value of a convolutional layer for generating the first heat maps in the forward propagation apparatus based on the calculated difference, wherein   the forward propagation apparatus comprises:   at least one second memory storing instructions; and   at least one second processor configured to execute the instructions to:   generate the first heat maps by performing an operation for a convolutional layer on an input feature map, the number of the first heat maps being equal to the number of types of objects to be detected by the neural network;   generate a composite heat map obtained by combining the first heat maps, the number of which is equal to the number of types of objects to be detected, into one heat map by summing up values of the first heat maps, the number of which is equal to the number of types of objects to be detected, on a coordinate-by-coordinate basis;   generate a binary mask by binarizing a value at each coordinate of the composite heat map by using a predetermined threshold; and   perform an operation for a sparse convolutional layer according to a value at each coordinate of the binary mask.   
     
     
         9 . The learning apparatus according to  claim 8 , wherein the first processor is further configured to execute the instructions to generate a heat map as the second heat map by arranging a 2D normal distribution so as to correspond to position of coordinate specified by the correct answer labels. 
     
     
         10 . The learning apparatus according to  claim 9 , wherein the first processor is further configured to execute the instructions to generate, as the second heat map, a heat map in which a maximum value of the 2D normal distribution is present at a coordinate of a joint point of a human being indicated by a correct answer label for human pose estimation using the neural network. 
     
     
         11 . The learning apparatus according to  claim 9 , wherein the first processor is further configured to execute the instructions to generate, as the second heat map, a heat map in which a maximum value of the 2D normal distribution is present inside a rectangular area indicated by a correct answer label for object detection using the neural network. 
     
     
         12 . The learning apparatus according to  claim 9 , wherein the first processor is further configured to execute the instructions to generate, as the second heat map, a heat map in which a maximum value of the 2D normal distribution is present in a rectangular area surrounding an area indicated by a correct answer label for semantic segmentation using the neural network, and values outside the area indicated by the correct answer label are set to zero. 
     
     
         13 . (canceled) 
     
     
         14 . (canceled) 
     
     
         15 . A processing method for a forward propagation apparatus for a neural network, comprising:
 generating a binary mask, and   performing an operation for a sparse convolutional layer according to a value at each coordinate of the binary mask, wherein   the generating the binary mask comprises:   generating heat maps by performing an operation for a convolutional layer on an input feature map, the number of the heat maps being equal to the number of types of objects to be detected by the neural network;   generating a composite heat map obtained by combining the heat maps the number of which is equal to the number of types of objects to be detected, into one heat map by summing up values of heat maps on a coordinate-by-coordinate basis; and   generating the binary mask by binarizing a value at each coordinate of the composite heat map by using a predetermined threshold.   
     
     
         16 . (canceled)