Deep learning for dense semantic segmentation in video with automated interactivity and improved temporal coherence

Abstract

Techniques related to automatically segmenting video frames into per pixel dense object of interest and background regions are discussed. Such techniques include applying a segmentation convolutional neural network (CNN) to a CNN input including a current video frame, a previous video frame, an object of interest indicator frame, a motion frame, and multiple feature frames each including features compressed from feature layers of an object classification convolutional neural network as applied to the current video frame to generate candidate segmentations and selecting one of the candidate segmentations as a final segmentation of the current video frame.

What is claimed is: 
     
         1 . A system for providing segmentation in video comprising:
 a memory to store a current video frame; and   one or more processors coupled to the memory, the one or more processors to:
 generate a convolutional neural network input comprising the current video frame, a temporally previous video frame, an object of interest indicator frame comprising one or more indicators of an object of interest in the current video frame, a motion frame comprising motion indicators indicative of motion from the previous video frame to the current video frame, and a plurality of feature frames each comprising features compressed from feature layers of an object classification convolutional neural network as applied to the current video frame; 
 apply a segmentation convolutional neural network to the convolutional neural network input to generate a plurality of candidate segmentations of the current video frame; and 
 select one of the candidate segmentations as a final segmentation corresponding to the current video frame.