Image processing apparatus, image processing method, and image processing program

Abstract

One embodiment discloses an image processing apparatus which generates an interpolation frame from consecutive first and second frames, including: a motion estimation section which assigns, to the interpolation frame, motion vectors from the first frame to the second frame; a first degree-of-difference calculation section which extracts calculates a first degree of difference in terms of pixel values; a second degree-of-difference calculation section which calculates a second degree of difference in terms of vectors; and an interpolation frame generation section which generates the interpolation frame by determining a motion vector that should be assigned to a pixel of attention on the basis of combined weights obtained by combining the first degrees of difference and the second degrees of difference, respectively.

The invention claimed is: 
     
       1. An image processing apparatus, comprising processing circuitry configured to at least:
 assign motion vectors from a first frame to a second frame to pixels of an interpolation frame, the second frame to be displayed at a different time than the first frame, the interpolation frame to be interpolated between the first frame and the second frame; 
 extract, as a pixel of attention, a pixel in the interpolation frame to which no motion vector is assigned 
 extract, as candidate motion vectors for the pixel of attention, motion vectors assigned to respective pixels in a first region of the interpolation frame around the pixel of attention; 
 calculate first degrees of difference based on a difference, for each respective candidate motion vector, between a pixel value of a pixel in the second frame to which the respective candidate motion vector points and a pixel value of a pixel in the second frame to which the respective candidate motion vector points when the respective candidate motion vector is assigned to the pixel of attention; 
 calculate second degrees of difference based on each of the candidate motion vectors and a motion vector calculated from motion vectors assigned to pixels in a second region of the interpolation frame that is wider than the first region; and 
 output a motion vector to be assigned to the pixel of attention based on combined weights obtained by combining the first degrees of difference and the second degrees of difference, respectively. 
 
     
     
       2. The image processing apparatus of  claim 1 , wherein the processing circuitry is configured to generate the interpolation frame based on the output motion vector or pixel values for the pixel of attention. 
     
     
       3. The image processing apparatus of  claim 2 , wherein the processing circuitry is configured to calculate the combined weights by combining the first degrees of difference and the second degrees of difference, respectively; and output vectors by performing weighted addition on the candidate motion vectors using the combined weights, and generate the interpolation frame based on the assigned motion vectors, the output vectors, and pixel values of the first frame and the second frame. 
     
     
       4. The image processing apparatus of  claim 3 , wherein the processing circuitry is configured to calculate, from the first degrees of difference, a first degree of reliability that increases as first degree of difference decreases, calculate, from the second degrees of difference, a second degree of reliability that increases as second degree of difference decreases, and calculate the combined weights based on a product of the first degree of reliability and the second degree of reliability. 
     
     
       5. The image processing apparatus of  claim 3 , wherein the processing circuitry is configured to calculate the combined weights based on weighted addition involving the first degrees of difference and the second degrees of difference. 
     
     
       6. The image processing apparatus of  claim 3 , wherein the processing circuitry is configured to select a candidate motion vector that produces a largest combined weight. 
     
     
       7. The image processing apparatus of  claim 3 , wherein the processing circuitry is configured to select a candidate motion vector that produces a smallest combined weight. 
     
     
       8. The image processing apparatus of  claim 1 , wherein the processing circuitry is configured to calculate the second degrees of difference based on each of the candidate motion vectors and a major background vector representing a motion of a major background region. 
     
     
       9. The image processing apparatus of  claim 8 , wherein the major background vector is a moving vector of a subject associated with largest depth values. 
     
     
       10. The image processing apparatus of  claim 1 , wherein the processing circuitry comprises a central processing unit (CPU) configured to execute an image processing program. 
     
     
       11. The image processing apparatus of  claim 1 , wherein the processing circuitry comprises one or more dedicated hardware circuits. 
     
     
       12. The image processing apparatus of  claim 1 , wherein the processing circuitry comprises one or more processors. 
     
     
       13. An image processing method, comprising:
 assigning motion vectors from a first frame to a second frame to pixels of an interpolation frame, the second frame to be displayed at a different time than the first frame, the interpolation frame to be interpolated between the first frame and the second frame; 
 extracting, as a pixel of attention, a pixel in the interpolation frame to which no motion vector is assigned; 
 extracting, as candidate motion vectors for the pixel of attention, motion vectors assigned to respective pixels in a first region of the interpolation frame around the pixel of attention; 
 calculating first degrees of difference based on a difference, for each respective candidate motion vector, between a pixel value of a pixel in the second frame to which the respective candidate motion vector points and a pixel value of a pixel in the second frame to which the respective candidate motion vector points when the respective candidate motion vector is assigned to the pixel of attention; 
 calculating second degrees of difference based on each of the candidate motion vectors and a vector calculated from motion vectors assigned to pixels in a second region of the interpolation frame that is wider than the first region; and 
 outputting a motion vector to be assigned to the pixel of attention based on combined weights obtained by combining the first degrees of difference and the second degrees of difference, respectively. 
 
     
     
       14. A computer readable storage medium storing an image processing program that, when executed by processing circuitry of an image processing apparatus, causes the image processing apparatus to perform at least:
 assigning motion vectors from a first frame to a second frame to pixels of an interpolation frame, the second frame to be displayed at a different time than the first frame, the interpolation frame to be interpolated between the first frame and the second frame; 
 extracting, as a pixel of attention, a pixel in the interpolation frame to which no motion vector is assigned; 
 extracting, as candidate motion vectors for the pixel of attention, motion vectors assigned to respective pixels in a first region of the interpolation frame around the pixel of attention; 
 calculating first degrees of difference based on a difference, for each respective candidate motion vector, between a pixel value of a pixel in the second frame to which the respective candidate motion vector points and a pixel value of a pixel in the second frame to which the respective candidate motion vector points when the respective candidate motion vector is assigned to the pixel of attention; 
 calculating second degrees of difference based on each of the candidate motion vectors and a vector calculated from motion vectors assigned to pixels existing in a second region of the interpolation frame that is wider than the first region; and 
 outputting a motion vector to be assigned to the pixel of attention based on combined weights obtained by combining the first degrees of difference and the second degrees of difference, respectively.