Model training apparatus, control method, and non-transitory computer-readable medium

Abstract

The model training apparatus trains an image conversion model to generate, from an input image representing a scene in a first environment, an output image representing the scene in a second environment. The model training apparatus inputs a training image to the image conversion model to obtain a first feature map and an output image, input the output image to the image conversion model to obtain a second feature map, computes a patch-wise loss using the features corresponding to a positive example patch and a negative example patch extracted from the training image and a positive example patch extracted from the output image, and trains the image conversion model based on the patch-wise loss, which is extracted intensively from the region representing an object of a specific type.

What is claimed is: 
     
         1 . A model training apparatus comprising:
 at least one memory that is configured to store instructions; and   at least one processor that is configured to execute the instructions to:
 acquire a training image, which is a first image used for a training of an image conversion model; and 
 perform the training of the image conversion model to generate, from an input image representing a scene in a first environment, an output image representing the scene in a second environment, 
   wherein the image conversion model comprises a feature extraction neural network and an image generation neural network, the feature extraction neural network being configured to generate a feature map by extracting features from each of a plurality of partial regions of the input image, the image generation neural network being configured to generate the output image from the feature map generated by the feature extraction neural network,   wherein the training of the image conversion model further includes:
 acquiring a first feature map and the output image by inputting the training image to the image conversion model, the first feature map being a feature map of the training image; 
 acquiring a second feature map by inputting the output image to the feature extraction neural network, the second feature map being a feature map of the output image; 
 generating one or more patch sets, each of which is a set of a first positive example patch, a first negative example patch, and a second positive example patch, the first positive example patch and the first negative example patch being partial regions of the training image, the second positive example patch being a partial region at a position corresponding to the first positive example patch in the output image; and 
 performing a training of the image conversion model using a patch-wise loss that is computed using the first feature map, the second feature map, and the patch set, 
   wherein the generation of the patch set further includes extracting the first positive example patch intensively from a specific region representing an object of a specific type among regions included in the training image,   wherein the generation of the patch set further includes:
 generating an extraction probability map that represents an extraction probability for each of the plurality of partial regions of the training image, the extraction probability representing a probability that the partial region is extracted as the first positive example patch; and 
 extracting the first positive example patch from the training image according to the extraction probability represented by the extraction probability map, and 
   wherein the generation of the extraction probability map further includes:
 setting, for each partial region in the extraction probability map, a higher extraction probability thereto as degree of overlap between that partial region and the specific region is higher; 
 determining a subtype of the object represented by the specific region included in the training image, the subtype representing a finer classification than the specific type; and 
 setting, for each partial region in the extraction probability map, a higher extraction probability thereto as degree of overlap between that partial region and the specific region is higher and as frequency of appearance of the subtype of the object represented by the specific region overlapping with that partial region is lower. 
   
     
     
         2 . The model training apparatus according to  claim 1 , wherein the generation of the patch set further includes:
 determining whether to extract the first positive example patch from inside or outside the specific region so that a probability of the first positive example patch being extracted from inside the specific region is higher than a probability of the first positive example patch being extracted from outside the specific region; and   extracting the first positive example patch from the training image based on a result of the determination.   
     
     
         3 . The model training apparatus according to  claim 1 , wherein the patch-wise loss is computed so as to be smaller as similarity between features corresponding to the first positive example patch and features corresponding to the second positive example patch is higher, and so as to be greater as similarity between features corresponding to the first negative example patch and the features corresponding to the second positive example patch is lower. 
     
     
         4 . A training method executed by a computer, the training method comprising:
 acquiring a training image, which is a first image used for a training of an image conversion model; and   performing the training of the image conversion model to generate, from an input image representing a scene in a first environment, an output image representing the scene in a second environment,   wherein the image conversion model comprises a feature extraction neural network and an image generation neural network, the feature extraction neural network being configured to generate a feature map by extracting features from each of a plurality of partial regions of the input image, the image generation neural network being configured to generate the output image from the feature map generated by the feature extraction neural network,   wherein the training of the image conversion model further includes:
 acquiring a first feature map and the output image by inputting the training image to the image conversion model, the first feature map being a feature map of the training image; 
 acquiring a second feature map by inputting the output image to the feature extraction neural network, the second feature map being a feature map of the output image; 
 generating one or more patch sets, each of which is a set of a first positive example patch, a first negative example patch, and a second positive example patch, the first positive example patch and the first negative example patch being partial regions of the training image, the second positive example patch being a partial region at a position corresponding to the first positive example patch in the output image; and 
 performing a training of the image conversion model using a patch-wise loss that is computed using the first feature map, the second feature map, and the patch set, 
   wherein the generation of the patch set further includes:
 extracting the first positive example patch intensively from a specific region representing an object of a specific type among regions included in the training image; 
 generating an extraction probability map that represents an extraction probability for each of the plurality of partial regions of the training image, the extraction probability representing a probability that the partial region is extracted as the first positive example patch; and 
 extracting the first positive example patch from the training image according to the extraction probability represented by the extraction probability map, and 
   wherein the generation of the extraction probability map further includes:
 setting, for each partial region in the extraction probability map, a higher extraction probability thereto as degree of overlap between that partial region and the specific region is higher; 
 determining a subtype of the object represented by the specific region included in the training image, the subtype representing a finer classification than the specific type; and 
 setting, for each partial region in the extraction probability map, a higher extraction probability thereto as degree of overlap between that partial region and the specific region is higher and as frequency of appearance of the subtype of the object represented by the specific region overlapping with that partial region is lower. 
   
     
     
         5 . The model training method according to  claim 4 , wherein the generation of the patch set further includes:
 determining whether to extract the first positive example patch from inside or outside the specific region so that a probability of the first positive example patch being extracted from inside the specific region is higher than a probability of the first positive example patch being extracted from outside the specific region; and   extracting the first positive example patch from the training image based on the result of the determination.   
     
     
         6 . The training method according to  claim 4 , wherein the patch-wise loss is computed so as to be smaller as similarity between features corresponding to the first positive example patch and features corresponding to the second positive example patch is higher, and so as to be greater as similarity between features corresponding to the first negative example patch and the features corresponding to the second positive example patch is lower. 
     
     
         7 . A non-transitory computer-readable medium storing a program causing a computer to execute:
 acquiring a training image, which is a first image used for a training of an image conversion model; and   performing the training of the image conversion model to generate, from an input image representing a scene in a first environment, an output image representing the scene in a second environment,   wherein the image conversion model comprises a feature extraction neural network and an image generation neural network, the feature extraction neural network being configured to generate a feature map by extracting features from each of a plurality of partial regions of the input image, the image generation neural network being configured to generate the output image from the feature map generated by the feature extraction neural network,   wherein the training of the image conversion model further includes:
 acquiring a first feature map and the output image by inputting the training image to the image conversion model, the first feature map being a feature map of the training image; 
 acquiring a second feature map by inputting the output image to the feature extraction neural network, the second feature map being a feature map of the output image; 
 generating one or more patch sets, each of which is a set of a first positive example patch, a first negative example patch, and a second positive example patch, the first positive example patch and the first negative example patch being partial regions of the training image, the second positive example patch being a partial region at a position corresponding to the first positive example patch in the output image; and 
 performing a training of the image conversion model using a patch-wise loss that is computed using the first feature map, the second feature map, and the patch set, and 
   wherein the generation of the patch set further includes:
 extracting the first positive example patch intensively from a specific region representing an object of a specific type among regions included in the training image; 
 generating an extraction probability map that represents an extraction probability for each of the plurality of partial regions of the training image, the extraction probability representing a probability that the partial region is extracted as the first positive example patch; and 
 extracting the first positive example patch from the training image according to the extraction probability represented by the extraction probability map, and 
   wherein the generation of the extraction probability map further includes:
 setting, for each partial region in the extraction probability map, a higher extraction probability thereto as degree of overlap between that partial region and the specific region is higher; 
 determining a subtype of the object represented by the specific region included in the training image, the subtype representing a finer classification than the specific type; and 
 setting, for each partial region in the extraction probability map, a higher extraction probability thereto as degree of overlap between that partial region and the specific region and as frequency of appearance of the subtype of the object represented by the specific region overlapping with that partial region is lower. 
   
     
     
         8 . The computer-readable medium according to  claim 7 , wherein the generation of the patch set further includes:
 determining whether to extract the first positive example patch from inside or outside the specific region so that a probability of the first positive example patch being extracted from inside the specific region is higher than a probability of the first positive example patch being extracted from outside the specific region; and   extracting the first positive example patch from the training image based on the result of the determination.   
     
     
         9 . The computer-readable medium according to  claim 7 , wherein the patch-wise loss is computed so as to be smaller as similarity between features corresponding to the first positive example patch and features corresponding to the second positive example patch is higher, and so as to be higher as similarity between features corresponding to the first negative example patch and the features corresponding to the second positive example patch is lower.