Object annotation using sparse active learning and core set selection

Abstract

The technology disclosed presents a system that comprises a memory, a data partitioning logic, and an annotation logic. The memory stores a sequence of frames of a video. The data partitioning logic is configured to partition the sequence of frames into an oracle set and an unannotated set. Frames in the oracle set are annotated by a user. Frames in the unannotated set are candidates for user annotation conditional upon being members of a core set, and for machine annotation conditional upon being non-members of the core set. The annotation logic is configured to generate annotations for the frames in the unannotated set. The annotations include user annotations based on membership in the core set, and machine annotations based on non-membership in the core set.

1 . A system, comprising:
 memory storing a sequence of frames of a video;   data partitioning logic configured to partition the sequence of frames into an oracle set and an unannotated set, wherein frames in the oracle set are annotated by a user, and wherein frames in the unannotated set are candidates for user annotation by the user and thereby for inclusion in the oracle set conditional upon being members of a core set, and for machine annotation by an annotation network trained on the oracle set conditional upon being non-members of the core set; and   annotation logic configured to generate annotations for the frames in the unannotated set, the annotations including:
 user annotations, by the user, for a first subset of frames in the unannotated set, wherein the first subset of frames includes those frames that are recommended to the user for being the members of the core set and are selected by the user for annotation, and 
 machine annotations, by the trained annotation network, for a second subset of frames in the unannotated set, wherein the second subset of frames includes those frames that are not recommended to the user for being the non-members of the core set, and those frames that are recommended to the user for being the members of the core set but are not selected by the user for annotation. 
   
     
     
         2 . The system of  claim 1 , wherein the annotation logic is further configured to generate the annotations for the frames in the unannotated set over a plurality of iterations. 
     
     
         3 . The system of  claim 2 , wherein a forward pass of a training category of iterations in the plurality of iterations includes:
 prior annotation logic configured to process a subject frame in the unannotated set through a prior training configuration of the trained annotation network, and to generate a prior annotation for the subject frame;   posterior annotation logic configured to process the subject frame through a posterior training configuration of the trained annotation network, and to generate a posterior annotation for the subject frame;   a gradient flow encoder configured to generate a gradient flow measurement for the subject frame, and to embed the gradient flow measurement in a latent frame space constructed from previous gradient flow measurements;   core set sampling logic configured to select a plurality of gradient flow measurements from the latent frame space;   template network configured to generate rankings for gradient flow measurements in the plurality of gradient flow measurements based on comparing contributions of associated frames to ablation-based accuracy metrics of the trained annotation network;   alignment network configured to generate rerankings for the ranked gradient flow measurements based on past user responses to prior frame recommendations;   frame recommendation logic configured to reorder the associated frames according to the rerankings, and to present the reordered associated frames to the user for annotation via an interface; and   the annotation logic further configured to include in the first subset of frames at least one of the reordered associated frames for which an annotation is received from the user, and to store the annotation as a user annotation for the at least one of the reordered associated frames.   
     
     
         4 . The system of  claim 3 , wherein the subject frame is one of the reordered associated frames. 
     
     
         5 . The system of  claim 4 , wherein the subject frame is the at least one of the reordered associated frames for which the annotation is received from the user. 
     
     
         6 . The system of  claim 5 , wherein the subject frame is not the at least one of the reordered associated frames for which the annotation is received from the user. 
     
     
         7 . The system of  claim 3 , wherein the subject frame is not one of the reordered associated frames. 
     
     
         8 . The system of  claim 3 , wherein the prior annotation is a first attention map that identifies a first configuration of regions-of-interest in the subject frame. 
     
     
         9 . The system of  claim 8 , wherein the posterior annotation is a second attention map that identifies a second configuration of the regions-of-interest in the subject frame. 
     
     
         10 . The system of  claim 3 , wherein the gradient flow encoder is further configured to generate the gradient flow measurement as an intermediate, compressed representation in response to processing the prior annotation as input and replicating the posterior annotation as output. 
     
     
         11 . The system of  claim 10 , wherein the gradient flow measurement encodes progression of gradient updates to coefficients of the trained annotation network from the prior training configuration to the posterior training configuration. 
     
     
         12 . The system of  claim 3 , wherein the latent frame space clusters gradient flow measurements and thereby the frames in the unannotated set into discrete latent frame subspaces. 
     
     
         13 . The system of  claim 12 , wherein the core set sampling logic is implemented by a rule-based selector configured to select at least one gradient flow measurement from each of the latent frame subspaces. 
     
     
         14 . The system of  claim 3 , wherein the core set sampling logic is implemented by a tree-based selector configured to select the plurality of gradient flow measurements based on importance. 
     
     
         15 . The system of  claim 3 , wherein the core set sampling logic is implemented by a neural network configured to select the plurality of gradient flow measurements based on supervisory signals learned during training 
     
     
         16 . The system of  claim 3 , wherein the core set sampling logic is further configured to randomly select the plurality of gradient flow measurements from the latent frame space. 
     
     
         17 . The system of  claim 3 , wherein a contribution of an associated frame to the ablation-based accuracy metrics of the trained annotation network is determined by:
 generating a dependent training configuration of the annotation network based on training the annotation network on a first training set that includes the associated frame;   generating an independent training configuration of the annotation network based on training the annotation network on a second training set that excludes the associated frame;   calculating a dependent annotation accuracy by applying the dependent training configuration of the annotation network on a validation set;   calculating an independent annotation accuracy by applying the independent training configuration of the annotation network on the validation set; and   measuring a difference between the dependent annotation accuracy and the independent annotation accuracy.   
     
     
         18 . The system of  claim 3 , wherein the template network is implemented by a neural network configured to generate the rankings of the gradient flow measurements based on the contributions of the associated frames to the ablation-based accuracy metrics of the trained annotation network. 
     
     
         19 . A system, comprising:
 memory storing a sequence of frames of a video;   data partitioning logic configured to partition the sequence of frames into an oracle set and an unannotated set, wherein frames in the oracle set are annotated by a user, and wherein frames in the unannotated set are candidates for user annotation by the user and thereby for inclusion in the oracle set conditional upon being members of a core set, and for machine annotation by an annotation network trained on the oracle set conditional upon being non-members of the core set; and   annotation logic configured to generate annotations for the frames in the unannotated set, the annotations including:   user annotations, by the user, for a first subset of frames in the unannotated set, wherein the first subset of frames includes those frames that are recommended to the user for being the members of the core set, and machine annotations, by the trained annotation network, for a second subset of frames in the unannotated set, wherein the second subset of frames includes those frames that are not recommended to the user for being the non-members of the core set.   
     
     
         20 . A system, comprising:
 memory storing a sequence of frames of a video;   data partitioning logic configured to partition the sequence of frames into an oracle set and an unannotated set, wherein frames in the oracle set are annotated by a user, and wherein frames in the unannotated set are candidates for user annotation conditional upon being members of a core set, and for machine annotation conditional upon being non-members of the core set; and   annotation logic configured to generate annotations for the frames in the unannotated set, the annotations including user annotations based on membership in the core set, and machine annotations based on non-membership in the core set.