Asymmetric adversarial learning framework for multi-turn dialogue response generation

Abstract

In a variety of embodiments, machine classifiers may model multi-turn dialogue as a one-to-many prediction task. The machine classifier may be trained using adversarial bootstrapping between a generator and a discriminator with multi-turn capabilities. The machine classifiers may be trained in both auto-regressive and traditional teacher-forcing modes, with the generator including a hierarchical recurrent encoder-decoder network and the discriminator including a bi-directional recurrent neural network. The discriminator input may include a mixture of ground truth labels, the teacher-forcing outputs of the generator, and/or noise data. This mixture of input data may allow for richer feedback on the autoregressive outputs of the generator. The outputs can be ranked based on the discriminator feedback and a response selected from the ranked outputs.

What is claimed is: 
     
         1 . A computer-implemented method for training a generative adversarial network comprising a generator and a discriminator, comprising:
 obtaining training data comprising a set of conversation data;   obtaining a set of teacher forcing samples;   generating, by the generator and based on the training data, initial autoregression data;   generating, by the generator and based on the set of teacher forcing samples, initial teacher forcing data;   calculating, by the discriminator and based on the initial autoregression data and the initial teacher forcing data, a discriminator accuracy;   training the discriminator when the discriminator accuracy is below a discriminator threshold value;   retraining the generator using a teacher forcing loss function of the generator when the discriminator accuracy is below a generator threshold value;   retraining the generator using the teacher forcing loss function and an autoregressive loss function when the discriminator accuracy is above the generator threshold value; and   storing the trained generative adversarial network.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the discriminator comprises a convolutional neural network. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein the discriminator comprises a recurrent neural network. 
     
     
         4 . The computer-implemented method of  claim 1 , further comprising:
 sampling a subset of the initial autoregression data;   sampling a subset of the initial teacher forcing data corresponding to the subset of the initial autoregression data; and   calculating the discriminator accuracy based on the subset of the initial autoregression data and the subset of the initial teacher forcing data.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein retraining the generator comprises using a maximum likelihood estimation function. 
     
     
         6 . The computer-implemented method of  claim 1 , further comprising:
 sampling a random sample from the training data; and   calculating the discriminator accuracy further based on the random sample.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein the training data further comprises automatically generated noise data inserted into the training data. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the generator comprises hierarchical recurrent encoder-decoder network. 
     
     
         9 . A device for training a generative adversarial network, comprising:
 a processor; and   a memory in communication with the processor and storing instructions that, when executed by the processor, cause the device to:
 obtain training data comprising a set of teacher forcing samples; 
 train, using the set of teacher forcing samples and based on a maximum likelihood estimation criterion, a generator of the generative adversarial network; 
 train, using the set of teacher forcing samples and based on a maximum likelihood estimation criterion, a diversity discriminator of the generative adversarial network; 
 generate, by the generator, initial response data based on the training data; 
 evaluate, by the diversity discriminator, the initial response data based on a comparison to the training data; 
 calculate, by the diversity discriminator and based on a comparison of the initial response data to the training data, a discriminator accuracy; 
 adjust one or more parameters of the diversity discriminator when the discriminator accuracy is below a discriminator threshold value; 
 adjust one or more parameters of the generator using a teacher forcing loss function of the generator when the discriminator accuracy is below a generator threshold value; 
 adjust one or more parameters of the generator using the teacher forcing loss function and an autoregressive loss function when the discriminator accuracy is above the generator threshold value; and 
 store the adjusted one or more parameters of the generator and the adjusted one or more parameters of the diversity discriminator. 
   
     
     
         10 . The device of  claim 9 , wherein the instructions, when read by the processor, further cause the device to:
 generate, by the generator, adjusted response data based on the training data;   train, using the adjusted response data, an exposure bias discriminator of the generative adversarial network;   evaluate, by the exposure bias discriminator, the adjusted response data based on a comparison of the adjusted response data to the training data;   generate, by the exposure bias discriminator and based on the evaluation of the adjusted response data, ranked adjusted response data; and   retrain, using the ranked adjusted response data, the generator.   
     
     
         11 . The device of  claim 10 , wherein the instructions, when read by the processor, further cause the device to remove the diversity discriminator from the generative adversarial network when the generator has been retrained. 
     
     
         12 . The device of  claim 10 , wherein the instructions, when read by the processor, further cause the device to remove the exposure bias discriminator from the generative adversarial network when the generator has been retrained. 
     
     
         13 . The device of  claim 9 , wherein the instructions, when read by the processor, further cause the device to:
 sample a random sample from the training data; and   calculate the discriminator accuracy further based on the random sample.   
     
     
         14 . The device of  claim 9 , wherein the training data further comprises automatically generated noise data inserted into the training data. 
     
     
         15 . The device of  claim 9 , wherein the diversity discriminator comprises a convolutional neural network. 
     
     
         16 . The device of  claim 9 , wherein the diversity discriminator comprises a recurrent neural network. 
     
     
         17 . A computer-implemented method for training a generative adversarial network comprising a generator and a discriminator, comprising:
 obtaining training data comprising a set of conversation data;   obtaining a set of teacher forcing samples;   generating, by the generator and based on the training data, initial autoregression data;   generating, by the generator and based on the set of teacher forcing samples, initial teacher forcing data;   sampling a subset of the initial autoregression data;   sampling a subset of the initial teacher forcing data corresponding to the subset of the initial autoregression data.   calculating, by the discriminator and based on the subset of the initial autoregression data and the subset of the initial teacher forcing data, a discriminator accuracy;   training the discriminator when the discriminator accuracy is below a discriminator threshold value;   retraining the generator using a teacher forcing loss function of the generator when the discriminator accuracy is below a generator threshold value;   retraining the generator using the teacher forcing loss function and an autoregressive loss function when the discriminator accuracy is above the generator threshold value; and   storing the trained generative adversarial network.   
     
     
         18 . The computer-implemented method of  claim 17 , wherein retraining the generator comprises using a maximum likelihood estimation function. 
     
     
         19 . The computer-implemented method of  claim 17 , further comprising:
 sampling a random sample from the training data; and   calculating the discriminator accuracy further based on the random sample.   
     
     
         20 . The computer-implemented method of  claim 19 , wherein the training data further comprises automatically generated noise data inserted into the training data.