Closed form method and system for matting a foreground object in an image having a background

Abstract

In a method and system for matting a foreground object F having an opacity α constrained by associating a characteristic with selected pixels in an image having a background B, weights are determined for all edges of neighboring pixels for the image and used to build a Laplacian matrix L. The equation α is solved where α=arg min α T Lα s.t.α i =s i , ∀i ∈ S, S is the group of selected pixels, and s i is the value indicated by the associated characteristic. The equation I i =α i F i +(1−α i )B i is solved for F and B with additional smoothness assumptions on F and B; after which the foreground object F may be composited on a selected background B′ that may be the original background B or may be a different background, thus allowing foreground features to be extracted from the original image and copied to a different background.

1 . A method for matting a foreground object F having an opacity α in an image having a background B, the respective opacity of selected pixels in the foreground object F and the background B being constrained by associating a characteristic with said pixels, the method comprising: 
 (a) determining weights for all edges of neighboring pixels for the image;    (b) build a Laplacian matrix L with the weights;    (c) solve the equation α where α=arg min α T  Lα s.t.α i =s i , ∀ i  ∈ S, S is the group of selected pixels, and s i  is the value indicated by said characteristic;    (d) solve for F and B in the equation I i =α i F i +(1−α i )B i  with additional smoothness assumptions on F and B; and    (e) compositing the foreground object F on a selected background B′.    
   
   
       2 . The method according to  claim 1 , wherein the characteristic with said pixels is a respective distinctive color.  
   
   
       3 . The method according to  claim 2 , wherein areas of F and B are each identified by a scribble of a respective distinctive color.  
   
   
       4 . The method according to  claim 2 , wherein only two distinctive colors are used to constrain the opacity α to be 0 or 1.  
   
   
       5 . The method according to  claim 2 , wherein additional colors are used to identify F and B respectively so as to constrain the value of α.  
   
   
       6 . The method according to  claim 3 , wherein F and B are constrained to be constant but unknown within a scribble.  
   
   
       7 . The method according to  claim 1 , wherein the selected background B′ is different from the background B.  
   
   
       8 . The method according to  claim 1 , wherein the input is a video sequence and the foreground F, background B and opacity α are also video sequences.  
   
   
       9 . A system for matting a foreground object F having an opacity α in an image having a background B, the respective opacity of selected pixels in the foreground object F and the background B being constrained by associating a characteristic with said pixels, comprising: a graphics processing unit; and software loadable on the graphics processing unit, the software being operable to: 
 (a) determining weights for all edges of neighboring pixels for the image;    (b) build a Laplacian matrix L with the weights;    (c) solve the equation α where α=arg min α T  Lα s.t.α i =s i , ∀ i  ∈ S, S is the group of selected pixels and s i  is the value indicated by said characteristic;    (d) solve for F and B in the equation I i =α I F i +(1−α i )B i  with additional smoothness assumptions on F and B; and    (e) compositing the foreground object F on a selected background B′.    
   
   
       10 . The system according to  claim 9 , wherein the characteristic with said pixels is a respective distinctive color.  
   
   
       11 . The system according to  claim 10 , wherein areas of F and B are each identified by a scribble of a respective distinctive color.  
   
   
       12 . The system according to  claim 10 , wherein only two distinctive colors are used to constrain the opacity α to be 0 or 1.  
   
   
       13 . The system according to  claim 10 , wherein additional colors are used to identify F and B respectively so as to constrain the value of α.  
   
   
       14 . The system according to  claim 11 , wherein F and B are constrained to be constant but unknown within a scribble.  
   
   
       15 . The system according to  claim 9 , wherein the selected background B′ is different from the background B.  
   
   
       16 . The system according to  claim 9 , wherein the input is a video sequence and the foreground F, background B and opacity α are also video sequences.  
   
   
       17 . A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform a method for matting a foreground object F having an opacity α in an image having a background B, the respective opacity of selected pixels in the foreground object F and the background B being constrained by associating a characteristic with said pixels, the method comprising: 
 (a) determining weights for all edges of neighboring pixels for the image;    (b) build a Laplacian matrix L with the weights;    (c) solve the equation α where α=arg min α T  Lα s.t.α i =s i , ∀ i  ∈ S, S is the group of selected pixels, and s i  is the value indicated by said characteristic;    (d) solve for F and B in the equation I i =α i F i +(1−α i )B i  with additional smoothness assumptions on F and B; and    (e) compositing the foreground object F on a selected background B′.