Artificial inelligence system with intuitive interactive interfaces for guided labeling of training data for machine learning models

Abstract

At an artificial intelligence system, during a labeling feedback session, a visualization data set is presented via a programmatic interface. The visualization data set comprises a representation of data items for which labeling feedback is requested for generating a training set of a classifier. At least one of the data items is selected based on an estimated rank with respect to a metric associated with including the data item in a training set. During the session, respective labels for the data items and a filter criterion to be used to select additional data items are obtained. A classifier trained using the labels is stored.

1 .- 22 . (canceled) 
     
     
         23 . A computer-implemented method, comprising:
 obtaining, at a cloud computing environment, a request to initiate an interactive labeling session for at least a portion of machine learning data set;   presenting, from the cloud computing environment, via one or more programmatic interfaces subsequent to receiving the request, (a) an image representing a first item of the machine learning data set and (b) a first label generated for the first item at the cloud computing environment; and   storing, at the cloud computing environment, a second label for the first item, wherein the second label is received via the one or more programmatic interfaces in response to said presenting, and wherein the second label differs from the first label.   
     
     
         24 . The computer-implemented method as recited in  claim 23 , further comprising:
 assigning respective ranks to a plurality of items of the machine learning data set, including the first item, based at least in part on estimated learning contributions of individual ones of the plurality of items to a training iteration of a machine learning model; and   determining, based at least in part on a rank assigned to the first item, to present an indication of the first item via the one or more programmatic interfaces, wherein the indication comprises the image.   
     
     
         25 . The computer-implemented method as recited in  claim 23 , wherein the first label corresponds to a first class of a plurality of classes into which items of the machine learning data set are to be classified, wherein the second label corresponds to a second class of the plurality of classes, the computer-implemented method further comprising:
 presenting, from the cloud computing environment via the one or more programmatic interfaces, prior to receiving the second label, an indication of a correlation between an attribute of the first item and membership of the first item in a particular class of the plurality of classes.   
     
     
         26 . The computer-implemented method as recited in  claim 23 , further comprising:
 receiving, at the cloud computing environment via the one or more programmatic interfaces, an indication of a justification for the second label.   
     
     
         27 . The computer-implemented method as recited in  claim 26 , further comprising:
 presenting, from the cloud computing environment via the one or more programmatic interfaces, the indication of the justification.   
     
     
         28 . The computer-implemented method as recited in  claim 23 , wherein the second label is received at the cloud computing environment from a first label provider, the computer-implemented method further comprising:
 obtaining, at the cloud computing environment from the first label provider via the one or more programmatic interfaces, a filter criterion to be used to select one or more additional items of the portion of the machine learning data set for which labels are to be obtained; and   presenting, by the cloud computing environment to the first label provider via the one or more programmatic interfaces, a representation of a second item, wherein the second item is selected from the portion of the machine learning data set using the filter criterion.   
     
     
         29 . The computer-implemented method as recited in  claim 23 , further comprising:
 training, at the cloud computing environment, in one or more training iterations, a machine learning model using labeled versions of items of the portion of the machine learning data set, wherein a labeled version of the first data item which includes the second label is used during a particular training iteration of the one or more training iterations, and wherein the second label is received at the cloud computing environment asynchronously with respect to the particular training iteration.   
     
     
         30 . A system, comprising:
 one or more computing devices;   wherein the one or more computing devices include instructions that upon execution on or across one or more processors cause the one or more processors to:
 obtain, at a cloud computing environment, a request to initiate an interactive labeling session for at least a portion of machine learning data set; 
 present, from the cloud computing environment, via one or more programmatic interfaces subsequent to receiving the request, (a) an image representing a first item of the machine learning data set and (b) a first label generated for the first item at the cloud computing environment; and 
 store, at the cloud computing environment, a second label for the first item, wherein the second label is received via the one or more programmatic interfaces in response to presentation of the image and the first label, and wherein the second label differs from the first label. 
   
     
     
         31 . The system as recited in  claim 30 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more processors further cause the one or more processors to:
 assign respective ranks to a plurality of items of the machine learning data set, including the first item, based at least in part on estimated learning contributions of individual ones of the plurality of items to a training iteration of a machine learning model; and   determine, based at least in part on a rank assigned to the first item, to present an indication of the first item via the one or more programmatic interfaces, wherein the indication comprises the image.   
     
     
         32 . The system as recited in  claim 30 , wherein the first label corresponds to a first class of a plurality of classes into which items of the machine learning data set are to be classified, wherein the second label corresponds to a second class of the plurality of classes, and wherein the one or more computing devices include further instructions that upon execution on or across the one or more processors further cause the one or more processors to:
 present, from the cloud computing environment via the one or more programmatic interfaces, prior to receiving the second label, an indication of a correlation between an attribute of the first item and membership of the first item in a particular class of the plurality of classes.   
     
     
         33 . The system as recited in  claim 30 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more processors further cause the one or more processors to:
 receive, at the cloud computing environment via the one or more programmatic interfaces, an indication of a justification for the second label.   
     
     
         34 . The system as recited in  claim 33 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more processors further cause the one or more processors to:
 present, from the cloud computing environment via the one or more programmatic interfaces, the indication of the justification.   
     
     
         35 . The system as recited in  claim 30 , wherein the second label is received at the cloud computing environment from a first label provider, and wherein the one or more computing devices include further instructions that upon execution on or across the one or more processors further cause the one or more processors to:
 obtain, at the cloud computing environment from the first label provider via the one or more programmatic interfaces, a filter criterion to be used to select one or more additional items of the portion of the machine learning data set for which labels are to be obtained; and   present, by the cloud computing environment to the first label provider via the one or more programmatic interfaces, a representation of a second item, wherein the second item is selected from the portion of the machine learning data set using the filter criterion.   
     
     
         36 . The system as recited in  claim 30 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more processors further cause the one or more processors to:
 train, at the cloud computing environment, in one or more training iterations, a machine learning model using labeled versions of items of the portion of the machine learning data set, wherein a labeled version of the first data item which includes the second label is used during a particular training iteration of the one or more training iterations, and wherein the second label is received at the cloud computing environment asynchronously with respect to the particular training iteration.   
     
     
         37 . One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors:
 obtain, at a cloud computing environment, a request to initiate an interactive labeling session for at least a portion of machine learning data set;   present, from the cloud computing environment, via one or more programmatic interfaces subsequent to receiving the request, (a) an image representing a first item of the machine learning data set and (b) a first label generated for the first item at the cloud computing environment; and   store, at the cloud computing environment, a second label for the first item, wherein the second label is received via the one or more programmatic interfaces in response to presentation of the image and the first label, and wherein the second label differs from the first label.   
     
     
         38 . The one or more non-transitory computer-accessible storage media as recited in  claim 37 , storing further program instructions that when executed on or across the one or more processors:
 assign respective ranks to a plurality of items of the machine learning data set, including the first item, based at least in part on estimated learning contributions of individual ones of the plurality of items to a training iteration of a machine learning model; and   determine, based at least in part on a rank assigned to the first item, to present an indication of the first item via the one or more programmatic interfaces, wherein the indication comprises the image.   
     
     
         39 . The one or more non-transitory computer-accessible storage media as recited in  claim 37 , wherein the first label corresponds to a first class of a plurality of classes into which items of the machine learning data set are to be classified, wherein the second label corresponds to a second class of the plurality of classes, and wherein the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:
 present, from the cloud computing environment via the one or more programmatic interfaces, prior to receiving the second label, an indication of a correlation between an attribute of the first item and membership of the first item in a particular class of the plurality of classes.   
     
     
         40 . The one or more non-transitory computer-accessible storage media as recited in  claim 37 , storing further program instructions that when executed on or across the one or more processors:
 receive, at the cloud computing environment via the one or more programmatic interfaces, an indication of a justification for the second label.   
     
     
         41 . The one or more non-transitory computer-accessible storage media as recited in  claim 40 , storing further program instructions that when executed on or across the one or more processors:
 present, from the cloud computing environment via the one or more programmatic interfaces, the indication of the justification.   
     
     
         42 . The one or more non-transitory computer-accessible storage media as recited in  claim 37 , wherein the second label is received at the cloud computing environment from a first label provider, and wherein the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:
 obtain, at the cloud computing environment from the first label provider via the one or more programmatic interfaces, a filter criterion to be used to select one or more additional items of the portion of the machine learning data set for which labels are to be obtained; and   present, by the cloud computing environment to the first label provider via the one or more programmatic interfaces, a representation of a second item, wherein the second item is selected from the portion of the machine learning data set using the filter criterion.