Multimodal Foreground Background Segmentation

Abstract

The subject disclosure is directed towards a framework that is configured to allow different background-foreground segmentation modalities to contribute towards segmentation. In one aspect, pixels are processed based upon RGB background separation, chroma keying, IR background separation, current depth versus background depth and current depth versus threshold background depth modalities. Each modality may contribute as a factor that the framework combines to determine a probability as to whether a pixel is foreground or background. The probabilities are fed into a global segmentation framework to obtain a segmented image.

What is claimed is: 
     
         1 . A system comprising, a foreground background segmentation framework, including a multimodal segmentation algorithm configured to accept contribution factors from different segmentation modalities and process the contribution factors to determine foreground versus background data for each element of an image that is useable to determine whether that element is a foreground or background element. 
     
     
         2 . The system of  claim 1  wherein at least one element comprises a pixel. 
     
     
         3 . The system of  claim 1  wherein the foreground versus background data comprises a probability score. 
     
     
         4 . The system of  claim 1  wherein the different segmentation modalities correspond to any of: a red, green blue (RGB) background subtraction, chroma keying, infrared (IR) background subtraction, a current computed depth versus previously computed background depth evaluation, or a current depth versus threshold depth evaluation. 
     
     
         5 . The system of  claim 1  wherein the foreground background segmentation framework is further configured to output the foreground versus background data for each element to a global binary segmentation algorithm. 
     
     
         6 . The system of  claim 1  wherein the framework is configured to apply a weight for each contribution factor. 
     
     
         7 . The system of  claim 6  wherein the framework is configured to select a weight set from among a plurality of weight sets to apply the weight for each contribution factor. 
     
     
         8 . The system of  claim 6  wherein the framework is coupled to a multiple camera set environment, and wherein the framework is configured to apply a weight set to one camera set that is different from a weight set applied to another camera set. 
     
     
         9 . The system of  claim 1  wherein the framework is coupled to a multiple camera set environment, and wherein the framework is configured to determine the foreground versus background data based on zero or more contribution factors in conjunction with information that corresponds to other camera foreground versus background data. 
     
     
         10 . The system of  claim 1  wherein the framework is configured to determine the foreground versus background data based on zero or more contribution factors and detection information processed from an image. 
     
     
         11 . A method, comprising, processing a frame of image data and processing depth data computed from a corresponding depth-related image, including performing background subtraction on an element of the image data to obtain a background subtraction contribution factor for that element, determining one or more other depth-based contribution factors based upon the depth data associated with that element, computing a combined data term based at least in part upon a contribution from the background contribution factor and a contribution from each of the one or more other depth-based contribution factors, and using the data term in conjunction with other data terms as input to a global binary segmentation mechanism to obtain a segmented image. 
     
     
         12 . The method of  claim 11  further comprising processing a frame of image data using chroma keying to obtain a chroma keying contribution factor, for the element and wherein computing the combined data term further comprises using a contribution from the chroma keying contribution factor. 
     
     
         13 . The method of  claim 11  wherein performing the background subtraction on an element of the image data comprises performing infrared background subtraction using captured infrared image data for a current element and previously captured background infrared image data. 
     
     
         14 . The method of  claim 11  wherein determining the one or more other depth-based contribution factors comprises evaluating a difference between currently captured depth data corresponding to the element and previously captured background depth data corresponding to the element. 
     
     
         15 . The method of  claim 11  wherein determining the one or more other depth-based contribution factors comprises evaluating currently captured depth data corresponding to the element and threshold depth data 
     
     
         16 . The method of  claim 11  further comprising, using information corresponding to background versus foreground information corresponding to at least one other cameras as in computing the combined data term. 
     
     
         17 . One or more machine-readable storage media or logic having executable instructions, which when executed perform steps, comprising:
 (a) selecting a pixel as a selected pixel;   (b) processing pixel data, including:
 processing red, green and blue (RGB) pixel data of one or more images to determine one or more RGB contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in a current image; 
 processing infrared (IR) pixel data of one or more infrared images to determine one or more IR contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in the current image; 
 processing pixel depth data to determine one or more depth-based contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in the current image; 
   (c) combining the contributing factors into a data term for the selected pixel;   (d) maintaining the data term for the selected pixel independent of other data terms for any other pixels;   (e) selecting a different pixel as the selected pixel; and   (f) returning to step (b) for a plurality of pixels to obtain a plurality of data terms.   
     
     
         18 . The one or more machine-readable storage media or logic of  claim 17  wherein processing the RGB pixel data of the one or more images to determine the one or more RGB contributing factors comprises performing at least one of: background subtraction based on a previous RGB background image and a current RGB image, or performing chroma keying based on known background data and a current RGB image. 
     
     
         19 . The one or more machine-readable storage media or logic of  claim 17  wherein processing the IR pixel data of the one or more images to determine the one or more IR contributing factors comprises performing background subtraction based on a previous IR background image and a current IR image. 
     
     
         20 . The one or more machine-readable storage media or logic of  claim 17  wherein processing the pixel depth data to determine the one or more depth-based contributing factors comprises performing at least one of: evaluating current pixel depth data against previous background pixel data, or evaluating current pixel depth data against threshold depth data.