Image encoding device, image encoding method, image encoding program, image decoding device, image decoding method, image decoding program, image processing device, learning device, learning method, learning program, similar image search device, similar image search method, and similar image search program

Abstract

A processor encodes a target image to derive at least one first feature amount indicating an image feature for an abnormality of a region of interest included in the target image. In addition, the processor encodes the target image to derive at least one second feature amount indicating an image feature for an image in a case in which the region of interest included in the target image is a normal region.

What is claimed is: 
     
         1 . An image encoding device comprising:
 at least one processor,   wherein the processor is configured to encode a target image to derive at least one first feature amount indicating an image feature for an abnormality of a region of interest included in the target image and to encode the target image to derive at least one second feature amount indicating an image feature for an image in a case in which the region of interest included in the target image is a normal region.   
     
     
         2 . The image encoding device according to  claim 1 ,
 wherein a combination of the first feature amount and the second feature amount indicates an image feature for the target image.   
     
     
         3 . The image encoding device according to  claim 1 , further comprising:
 a storage that stores at least one first feature vector indicating a representative image feature for the abnormality of the region of interest and at least one second feature vector indicating a representative image feature for the image in a case in which the region of interest is the normal region,   wherein the processor is configured to derive the first feature amount by substituting a feature vector indicating the image feature for the abnormality of the region of interest with a first feature vector, which minimizes a difference from the image feature for the abnormality of the region of interest, among the first feature vectors to quantize the feature vector and to derive the second feature amount by substituting a feature vector indicating the image feature for the image in a case in which the region of interest is the normal region with a second feature vector, which minimizes a difference from the image feature for the image in a case in which the region of interest is the normal region, among the second feature vectors to quantize the feature vector.   
     
     
         4 . The image encoding device according to  claim 1 ,
 wherein the processor is configured to derive the first feature amount and the second feature amount, using an encoding learning model which has been trained to derive the first feature amount and the second feature amount in a case in which the target image is input.   
     
     
         5 . An image decoding device comprising:
 at least one processor,   wherein the processor is configured to extract a region corresponding to a type of the abnormality of the region of interest in the target image on the basis of the first feature amount derived from the target image by the image encoding device according to  claim 1 .   
     
     
         6 . The image decoding device according to  claim 5 ,
 wherein the processor is configured to derive a first reconstructed image obtained by reconstructing an image feature for an image in a case in which the region of interest in the target image is a normal region on the basis of the second feature amount and to derive a second reconstructed image obtained by reconstructing an image feature for the target image on the basis of the first feature amount and the second feature amount.   
     
     
         7 . The image decoding device according to  claim 6 ,
 wherein the processor is configured to derive a label image corresponding to the type of the abnormality of the region of interest in the target image, the first reconstructed image, and the second reconstructed image, using a decoding learning model which has been trained to derive the label image corresponding to the type of the abnormality of the region of interest in the target image on the basis of the first feature amount, to derive the first reconstructed image obtained by reconstructing the image feature for the image in a case in which the region of interest in the target image is the normal region on the basis of the second feature amount, and to derive the second reconstructed image obtained by reconstructing the image feature of the target image on the basis of the first feature amount and the second feature amount.   
     
     
         8 . An image processing device comprising:
 the image encoding device according to  claim 1 ; and   the image decoding device according to  claim 5 .   
     
     
         9 . A learning device that trains the encoding learning model in the image encoding device according to  claim 4  and the decoding learning model in the image decoding device according to  claim 7 , using training data consisting of a training image including a region of interest and a training label image corresponding to a type of an abnormality of the region of interest in the training image, the learning device comprising:
 at least one processor, 
 wherein the processor is configured to derive a first learning feature amount and a second learning feature amount corresponding to the first feature amount and the second feature amount, respectively, from the training image using the encoding learning model, to derive a learning label image corresponding to the type of the abnormality of the region of interest included in the training image on the basis of the first learning feature amount, to derive a first learning reconstructed image obtained by reconstructing an image feature for an image in a case in which the region of interest in the training image is a normal region on the basis of the second learning feature amount, and to derive a second learning reconstructed image obtained by reconstructing an image feature for the training image on the basis of the first learning feature amount and the second learning feature amount, using the decoding learning model, and to train the encoding learning model and the decoding learning model such that at least one of a first loss which is a difference between the first learning feature amount and a predetermined probability distribution of the first feature amount, a second loss which is a difference between the second learning feature amount and a predetermined probability distribution of the second feature amount, a third loss based on a difference between the training label image included in the training data and the learning label image as semantic segmentation for the training image, a fourth loss based on a difference between the first learning reconstructed image and an image outside the region of interest in the training image, a fifth loss based on a difference between the second learning reconstructed image and the training image, or a sixth loss based on a difference between regions corresponding to an inside and an outside of the region of interest in the first learning reconstructed image and in the second learning reconstructed image satisfies a predetermined condition. 
 
     
     
         10 . A similar image search device comprising:
 at least one processor; and   the image encoding device according to  claim 1 ,   wherein the processor is configured to derive a first feature amount and a second feature amount for a query image using the image encoding device, to derive a similarity between the query image and each of a plurality of reference images on the basis of at least one of the first feature amount or the second feature amount derived from the query image with reference to an image database in which a first feature amount and a second feature amount for each of the plurality of reference images are registered in association with each of the plurality of reference images, and to extract a reference image that is similar to the query image as a similar image from the image database on the basis of the similarity.   
     
     
         11 . An image encoding method comprising:
 encoding a target image to derive at least one first feature amount indicating an image feature for an abnormality of a region of interest included in the target image; and   encoding the target image to derive at least one second feature amount indicating an image feature for an image in a case in which the region of interest included in the target image is a normal region.   
     
     
         12 . An image decoding method comprising:
 extracting a region corresponding to a type of an abnormality of the region of interest in the target image on the basis of the first feature amount derived from the target image by the image encoding device according to  claim 1 .   
     
     
         13 . A learning method for training the encoding learning model in the image encoding device according to  claim 4  and the decoding learning model in the image decoding device according to  claim 7 , using training data consisting of a training image including a region of interest and a training label image corresponding to a type of an abnormality of the region of interest in the training image, the learning method comprising:
 deriving a first learning feature amount and a second learning feature amount corresponding to the first feature amount and the second feature amount, respectively, from the training image using the encoding learning model; 
 deriving a learning label image corresponding to the type of the abnormality of the region of interest included in the training image on the basis of the first learning feature amount, deriving a first learning reconstructed image obtained by reconstructing an image feature for an image in a case in which the region of interest in the training image is a normal region on the basis of the second learning feature amount, and deriving a second learning reconstructed image obtained by reconstructing an image feature for the training image on the basis of the first learning feature amount and the second learning feature amount, using the decoding learning model; and 
 training the encoding learning model and the decoding learning model such that at least one of a first loss which is a difference between the first learning feature amount and a predetermined probability distribution of the first feature amount, a second loss which is a difference between the second learning feature amount and a predetermined probability distribution of the second feature amount, a third loss based on a difference between the training label image included in the training data and the learning label image as semantic segmentation for the training image, a fourth loss based on a difference between the first learning reconstructed image and an image outside the region of interest in the training image, a fifth loss based on a difference between the second learning reconstructed image and the training image, or a sixth loss based on a difference between regions corresponding to an inside and an outside of the region of interest in the first learning reconstructed image and in the second learning reconstructed image satisfies a predetermined condition. 
 
     
     
         14 . A similar image search method comprising:
 deriving a first feature amount and a second feature amount for a query image using the image encoding device according to  claim 1 ;   deriving a similarity between the query image and each of a plurality of reference images on the basis of at least one of the first feature amount or the second feature amount derived from the query image with reference to an image database in which a first feature amount and a second feature amount for each of the plurality of reference images are registered in association with each of the plurality of reference images; and   extracting a reference image that is similar to the query image as a similar image from the image database on the basis of the similarity.   
     
     
         15 . A non-transitory computer-readable storage medium that stores an image encoding program that causes a computer to execute:
 a procedure of encoding a target image to derive at least one first feature amount indicating an image feature for an abnormality of a region of interest included in the target image; and   a procedure of encoding the target image to derive at least one second feature amount indicating an image feature for an image in a case in which the region of interest included in the target image is a normal region.   
     
     
         16 . A non-transitory computer-readable storage medium that stores an image decoding program that causes a computer to execute:
 a procedure of extracting a region corresponding to a type of an abnormality of the region of interest in the target image on the basis of the first feature amount derived from the target image by the image encoding device according to  claim 1 .   
     
     
         17 . A non-transitory computer-readable storage medium that stores a learning program that causes a computer to execute a procedure of training the encoding learning model in the image encoding device according to  claim 4  and the decoding learning model in the image decoding device according to  claim 7 , using training data consisting of a training image including a region of interest and a training label image corresponding to a type of an abnormality of the region of interest in the training image, the learning program causing the computer to execute:
 a procedure of deriving a first learning feature amount and a second learning feature amount corresponding to the first feature amount and the second feature amount, respectively, from the training image using the encoding learning model; 
 a procedure of deriving a learning label image corresponding to the type of the abnormality of the region of interest included in the training image on the basis of the first learning feature amount, deriving a first learning reconstructed image obtained by reconstructing an image feature for an image in a case in which the region of interest in the training image is a normal region on the basis of the second learning feature amount, and deriving a second learning reconstructed image obtained by reconstructing an image feature for the training image on the basis of the first learning feature amount and the second learning feature amount, using the decoding learning model; and 
 a procedure of training the encoding learning model and the decoding learning model such that at least one of a first loss which is a difference between the first learning feature amount and a predetermined probability distribution of the first feature amount, a second loss which is a difference between the second learning feature amount and a predetermined probability distribution of the second feature amount, a third loss based on a difference between the training label image included in the training data and the learning label image as semantic segmentation for the training image, a fourth loss based on a difference between the first learning reconstructed image and an image outside the region of interest in the training image, a fifth loss based on a difference between the second learning reconstructed image and the training image, or a sixth loss based on a difference between regions corresponding to an inside and an outside of the region of interest in the first learning reconstructed image and in the second learning reconstructed image satisfies a predetermined condition. 
 
     
     
         18 . A non-transitory computer-readable storage medium that stores a similar image search program that causes a computer to execute:
 a procedure of deriving a first feature amount and a second feature amount for a query image using the image encoding device according to  claim 1 ;   a procedure of deriving a similarity between the query image and each of a plurality of reference images on the basis of at least one of the first feature amount or the second feature amount derived from the query image with reference to an image database in which a first feature amount and a second feature amount for each of the plurality of reference images are registered in association with each of the plurality of reference images; and   a procedure of extracting a reference image that is similar to the query image as a similar image from the image database on the basis of the similarity.