Fine-grain object segmentation in video with deep features and multi-level graphical models

Abstract

Techniques related to automatically segmenting a video frame into fine grain object of interest and background regions using a ground truth segmentation of an object in a previous frame are discussed. Such techniques apply multiple levels of segmentation tracking and prediction based on color, shape, and motion of the segmentation to determine per-pixel object probabilities, and solve an energy summation model to generate a final segmentation for the video frame using the object probabilities.

What is claimed is: 
     
         1 . A system for providing segmentation in video comprising:
 a memory to store a current video frame; and   one or more processors coupled to the memory, the one or more processors to:
 train a color mixture model using a region within a first bounding box of a first video frame, the first bounding box surrounding a ground truth segmentation of an object from a background within the bounding box; 
 determine, based on an optical flow between at least the ground truth segmentation and a second bounding box of a second video frame, a first shape estimation of the object in the second video frame; 
 apply an affine transformation to the first shape estimation to generate a second shape estimation of the object in the second video frame, the affine transformation generated based on object landmark tracking between the first and second video frames; and 
 determine a final segmentation of the object in the second video frame based at least on the second shape estimation and application of the color mixture model to the second bounding box.