Image processing apparatus, image processing method, and non-transitory computer-readable medium

Abstract

An image processing apparatus is provided. The apparatus acquires input data including a captured image and/or information relating to the captured image. The apparatus acquires a feature of the input data by performing processing on the input data using a neural network. The apparatus generates an integrated feature by integrating the feature and at least some of the input data. The apparatus generates an estimation result of at least one of a defocus range and a depth range for a subject within the captured image, by performing processing on the integrated feature.

What is claimed is: 
     
         1 . An image processing apparatus comprising one or more memories storing instructions and one or more processors that execute the instructions to:
 acquire input data including a captured image and/or information relating to the captured image;   acquire a feature of the input data by performing processing on the input data using a neural network;   generate an integrated feature by integrating the feature and at least some of the input data; and   generate an estimation result of at least one of a defocus range and a depth range for a subject within the captured image, by performing processing on the integrated feature.   
     
     
         2 . The image processing apparatus according to  claim 1 ,
 wherein the one or more processors execute the instructions to integrate the feature and the at least some of the input data by calculating an element-wise product.   
     
     
         3 . The image processing apparatus according to  claim 1 ,
 wherein the one or more processors execute the instructions to perform processing for matching a spatial resolution between the feature and the at least some of the input data, and integrate the feature and the at least some of the input data after the processing for matching.   
     
     
         4 . The image processing apparatus according to  claim 1 ,
 wherein the one or more processors execute the instructions to perform processing for matching a number of channels between the feature and the at least some of the input data, and integrate the feature and the at least some of the input data after the processing for matching.   
     
     
         5 . The image processing apparatus according to  claim 1 ,
 wherein the one or more processors execute the instructions to generate the integrated feature, by performing processing for combining the feature and the at least some of the input data in a channel direction, and perform a product-sum operation on data obtained by the processing for combining.   
     
     
         6 . The image processing apparatus according to  claim 1 ,
 wherein the one or more processors execute the instructions to perform noise reduction processing on the at least some of the input data, and generate the integrated feature by integrating the feature and the at least some of the input data after the noise reduction processing.   
     
     
         7 . The image processing apparatus according to  claim 1 ,
 wherein the one or more processors execute the instructions to perform high-resolution processing on the at least some of the input data, and generate the integrated feature by integrating the feature and the at least some of the input data obtained after the high-resolution processing.   
     
     
         8 . The image processing apparatus according to  claim 1 ,
 wherein the processing performed on the integrated feature is a linear operation.   
     
     
         9 . The image processing apparatus according to  claim 1 ,
 wherein the processing performed on the integrated feature includes processing in a fully-connected layer.   
     
     
         10 . The image processing apparatus according to  claim 1 ,
 wherein the processing performed on the integrated feature does not include a nonlinear operation.   
     
     
         11 . The image processing apparatus according to  claim 1 ,
 wherein the processing performed on the input data includes a nonlinear operation.   
     
     
         12 . The image processing apparatus according to  claim 1 ,
 wherein the processing performed on the input data is processing in a neural network including an activation layer.   
     
     
         13 . The image processing apparatus according to  claim 1 ,
 wherein a calculation amount of the processing performed on the integrated feature is less than a calculation amount of the processing performed on the input data.   
     
     
         14 . The image processing apparatus according to  claim 1 ,
 wherein the input data further includes data designating a region in which the subject is located within the captured image.   
     
     
         15 . The image processing apparatus according to  claim 1 ,
 wherein the input data includes the captured image, and   the at least some of the input data is the captured image.   
     
     
         16 . The image processing apparatus according to  claim 1 ,
 wherein the input data includes the captured image and at least one of a defocus map showing a defocus amount for each position in the captured image and a depth map showing a depth value for each position in the captured image, and   the at least some of the input data is the at least one of the defocus map and the depth map.   
     
     
         17 . An image processing method comprising:
 acquiring input data including a captured image and/or information relating to the captured image;   acquiring a feature of the input data by performing processing on the input data using a neural network;   generating an integrated feature by integrating the feature and at least some of the input data; and   generating an estimation result of at least one of a defocus range and a depth range for a subject within the captured image, by performing processing on the integrated feature.   
     
     
         18 . A non-transitory computer-readable medium storing a program executable by a computer to perform a method comprising:
 acquiring input data including a captured image and/or information relating to the captured image;   acquiring a feature of the input data by performing processing on the input data using a neural network;   generating an integrated feature by integrating the feature and at least some of the input data; and   generating an estimation result of at least one of a defocus range and a depth range for a subject within the captured image, by performing processing on the integrated feature.