Rendering a dental model in an image

Abstract

Method for realistic visualization of a virtual dental model in a first image of a face comprising at least an inner mouth region of the face, the inner mouth region comprising teeth (31) and a mouth cavity background, the method comprising using an inpainting algorithm to compute, based on background pixels corresponding to the mouth cavity background (35), plausible background pixels for at least a part of tooth pixels of the first image, the tooth pixels corresponding to the teeth, and to replace at least a part of the tooth pixels by plausible background pixels, at least partially removing the teeth (31) from the first image; and visualizing the first image with the at least partially removed teeth (31′), wherein the first image is overlaid with a representation (21) of the dental model.

1 . A method for realistic visualization of a virtual dental model ( 1 ) in a first image ( 20 ) of a face comprising at least an inner mouth region of the face, the inner mouth region comprising teeth ( 31 ) and a mouth cavity background ( 35 ), the method comprising:
 using an inpainting algorithm
 to compute, based on background pixels corresponding to the mouth cavity background ( 35 ), plausible background pixels for at least a part of tooth pixels of the first image ( 20 ), the tooth pixels corresponding to the teeth ( 31 ), and 
 to replace at least a part of the tooth pixels by plausible background pixels, at least partially removing the teeth ( 31 ) from the first image ( 20 ); and 
   visualizing the first image ( 20 ) with the at least partially removed teeth ( 31 ′), wherein the first image ( 20 ) is overlaid with a representation ( 21 ) of the dental model ( 1 ).   
     
     
         2 . The method according to  claim 1 ,
 characterized by   using a recognition algorithm to determine in the first image ( 20 ) background pixels corresponding to the mouth cavity background ( 35 ) and tooth pixels corresponding to the teeth ( 31 ), particularly wherein the mouth cavity background ( 35 ) comprises at least a tongue and/or a palate.   
     
     
         3 . The method according to  claim 1 ,
 characterized by   using a recognition algorithm to determine in a previously captured image of the same face background pixels corresponding to the mouth cavity background ( 35 ), in particular wherein more of the mouth cavity background ( 35 ) is visible in the previously captured image than in the first image ( 20 ), particularly wherein the mouth cavity background ( 35 ) comprises at least a tongue and/or a palate.   
     
     
         4 . The method according to  claim 3 ,
 characterized by   using the recognition algorithm to determine in the first image ( 20 ) tooth pixels corresponding to the teeth ( 31 ).   
     
     
         5 . The method according to  claim 1 ,
 characterized in that   computing the plausible background pixels comprises a stretching of a texture of the mouth cavity background ( 35 ) along a vertical direction of the face in image space, particularly wherein the vertical direction is a direction of nose to chin or a direction of chin to nose.   
     
     
         6 . The method according to  claim 5 ,
 characterized in that
 a three-dimensional proxy geometry is used for the stretching of the texture of the mouth cavity background ( 35 ), wherein the proxy geometry at least roughly follows an arch form slightly extending beyond the teeth ( 31 ); and/or 
 a fragment shader, particularly using a texel lookup downwards with a weighted vector length, is used for the stretching of the texture of the mouth cavity background ( 35 ). 
   
     
     
         7 . The method according to  claim 5 ,
 characterized in that   a distance in the image ( 20 ) between teeth ( 31 ) of the lower jaw and the upper jaw is determined, wherein the stretching of the texture of the mouth cavity background ( 35 ) is performed only if the distance lies above a predetermined threshold.   
     
     
         8 . The method according to  claim 5 ,
 characterized in that
 a necessary stretching degree is determined, the necessary stretching degree being that intensity of stretching of the texture of the mouth cavity background ( 35 ) that is necessary to replace all tooth pixels that would be visible when the image ( 20 ) is overlaid with the dental model ( 1 ); and 
 the texture of the mouth cavity background ( 35 ) is stretched at least with the determined necessary stretching degree, in particular exactly with the determined necessary stretching degree. 
   
     
     
         9 . The method according to  claim 1 ,
 characterized in that   the virtual dental model ( 1 )
 is an upper jaw dental model, wherein the method is performed only for the upper jaw, only tooth pixels corresponding to the teeth ( 31 ) of the upper jaw being determined and replaced; or 
 is a lower jaw dental model, wherein the method is performed only for the lower jaw, only tooth pixels corresponding to the teeth ( 31 ) of the lower jaw being determined and replaced; or 
 comprises an upper jaw dental model and a lower jaw dental model, wherein the determining and replacing tooth pixels corresponding to the teeth ( 31 ) of the upper jaw and determining and replacing tooth pixels corresponding to the teeth ( 31 ) of the lower jaw are performed subsequently. 
   
     
     
         10 . The method according to  claim 1 ,
 characterized by   determining, before using the inpainting algorithm, whether tooth pixels are visible when the image ( 20 ) is overlaid with the dental model ( 1 ), particularly wherein the visibility is determined only for those tooth pixels that correspond to teeth ( 31 ) of a jaw at which teeth of the virtual dental model ( 1 ) are positioned.   
     
     
         11 . The method according to  claim 1 ,
 characterized in that   the image ( 20 ) is part of an image stream, and the method is performed in real time for at least a multitude of images ( 20 ) of the image stream.   
     
     
         12 . Method according to  claim 11 ,
 characterized in that   the method further comprises capturing the image stream by means of a camera ( 4 ) and visualizing the dental model ( 1 ) and the two-dimensional image ( 20 ) on a displaying device ( 12 ) to a user, particularly wherein the face is the face of the user,   in particular wherein the camera ( 4 ) and the displaying device ( 12 ) are part of the same mobile device ( 10 ), wherein the method is performed by means of one or more algorithms installed on a computing unit of the mobile device ( 10 ).   
     
     
         13 . The method according to  claim 1 ,
 characterized in that   the virtual dental model ( 1 ) is in a polygon mesh format and comprises a plurality of vertices.   
     
     
         14 . A mobile device ( 10 ) comprising a camera ( 4 ) and a display ( 12 ) that arranged so that images ( 20 ) of a user's face are capturable by the camera ( 4 ) while the user watches the display ( 12 ),
 characterized in that   the device comprises a computing unit with at least one algorithm that is adapted to perform the method of  claim 1 .   
     
     
         15 . A computer programme product comprising programme code which is stored on a non-transitory machine-readable medium, the machine-readable medium comprising computer instructions executable by a processor, which computer instructions cause the processor to perform the method according to  claim 1 . 
     
     
         16 . The computer programme product according to  claim 15  executed on a mobile device comprising a camera ( 4 ) and a display ( 12 ) that is arranged so that images ( 20 ) of a user's face are capturable by the camera ( 4 ) while the user watches the display ( 12 ).