Auditing user feedback data

Abstract

Methods and systems are presented for auditing user feedback data corresponding to user communications received via at least one interface of a service provider. The user feedback data includes a first set of feedback categories associated with a first classification of the user communications. A first feature representation of the communications is generated from the user feedback data. The first feature representation includes a first set of textual data features extracted from the communications. A second feature representation is generated from the first feature representation using a first machine learning model. The second feature representation includes a second set of textual data features including semantic equivalents of the first set of features. A second machine learning model is trained with the second feature representation. A second classification of the user communications according to a second set of feedback categories is generated using the trained second machine learning model.

What is claimed is: 
     
       1. A system comprising:
 one or more hardware processors; and 
 a non-transitory memory having stored therein instructions that are executable by the one or more hardware processors to cause the system to perform operations comprising: 
 accessing user feedback data corresponding to user communications received via at least one customer service interface of a service provider, the user feedback data including a first set of feedback categories associated with a first classification of the user communications; 
 generating a first feature representation of the user communications from the user feedback data, the first feature representation including a first set of textual data features extracted from the user communications; 
 generating, using a first machine learning model, a second feature representation of the user communications from the first feature representation, the second feature representation including a second set of textual data features extracted from the user communications, and the second set of textual data features including semantic equivalents of the first set of textual data features; 
 training a second machine learning model with the second feature representation generated from the first machine learning model; and 
 generating, using the trained second machine learning model, a second classification of the user communications according to a second set of feedback categories. 
 
     
     
       2. The system of  claim 1 , wherein the operations further comprise:
 identifying, from the first machine learning model using a deep learning framework, the semantic equivalents of the first set of textual data features within the user communications; and 
 extracting, from the user communications, the second set of textual data features including the identified semantic equivalents. 
 
     
     
       3. The system of  claim 1 , wherein the operations further comprise:
 clustering the user communications into a plurality of clusters using one or more clustering algorithms, each of the plurality of clusters corresponding to a different one of the feedback categories in the second set based on a unique pattern of the second set of textual features within the user communications. 
 
     
     
       4. The system of  claim 1 , wherein the second machine learning model is selected from among a plurality of machine learning models, and wherein the operations further comprise:
 training each machine learning model in the plurality of machine learning models to reclassify the user communications based on the second feature representation; 
 comparing confidence scores for respective classifications of the user communications generated by the plurality of trained machine learning models; and 
 selecting, from among the plurality of trained machine learning models, at least one trained machine learning model as the second machine learning model based on the comparison, wherein the at least one trained machine learning model is used to generate the second classification of the user communications, and the second classification has a highest confidence score from among the respective classifications generated by the plurality of trained machine learning models. 
 
     
     
       5. The system of  claim 1 , wherein the at least one customer service interface of the service provider is a customer support hotline for receiving phone calls from one or more users of the service provider, and wherein the user feedback data further includes audio files corresponding to voice communications between the one or more users and customer service agents of the service provider via the customer support hotline. 
     
     
       6. The system of  claim 5 , wherein the operations further comprise:
 preprocessing the voice communications for use by at least one of the first machine learning model or the second machine learning model. 
 
     
     
       7. The system of  claim 6 , wherein the preprocessing comprises:
 converting each of the audio files into a text file including a transcription of the corresponding voice communications. 
 
     
     
       8. The system of  claim 7 , wherein the preprocessing further comprises:
 vectorizing the text file using a word embedding algorithm; and 
 extracting the first set of textual data features from the vectorized text file. 
 
     
     
       9. A method comprising:
 accessing, by a computing device from a database, user feedback data corresponding to user communications received via at least one communications interface of a service provider, the user feedback data including a first set of feedback categories associated with a first classification of the user communications; 
 generating, by the computing device, an input file including the user communications in a data format associated with a first machine learning model; 
 extracting, using the first machine learning model, a plurality of semantic features of the user communications from the generated input file; 
 determining a second set of feedback categories for the user communications using a second machine learning model trained with the extracted plurality of semantic features; and 
 reclassifying the user communications according to the second set of feedback categories. 
 
     
     
       10. The method of  claim 9 , wherein the data format of the user communications in the input file is a text format, and wherein the input file includes a first feature representation of the user communications that includes a first set of textual data features extracted from the user communications. 
     
     
       11. The method of  claim 10 , wherein extracting the plurality of semantic features comprises:
 generating a second feature representation of the user communications using the first machine learning model with the first feature representation, the second feature representation including a second set of textual data features extracted from the user communications, and the second set of textual data features including semantic equivalents of the first set of textual data features. 
 
     
     
       12. The method of  claim 9 , wherein the user communications include audio files corresponding to voice communications between one or more users and customer service agents associated with the service provider. 
     
     
       13. The method of  claim 12 , wherein generating the input file further comprises:
 converting each of the audio files into a text file including a transcription of the corresponding voice communications; and 
 preprocessing the text file for use by the first machine learning model. 
 
     
     
       14. The method of  claim 13 , wherein the preprocessing comprises:
 removing noise or unwanted information from the text file. 
 
     
     
       15. The method of  claim 14 , further comprising:
 vectorizing the text file using a word embedding algorithm; and 
 extracting the first set of textual data features from the vectorized text file. 
 
     
     
       16. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
 accessing user feedback data corresponding to user communications received via at least one communications interface of a service provider, the user feedback data including a first set of feedback categories associated with a first classification of the user communications; 
 generating a first feature representation of the user communications from the user feedback data, the first feature representation including a first set of textual data features extracted from the user communications; 
 generating a second feature representation of the user communications from a first machine learning model based on the first feature representation, the second feature representation including a second set of textual data features extracted from the user communications, and the second set of textual data features including semantic equivalents of the first set of textual data features; 
 determining a second set of feedback categories for the user communications using a second machine learning model trained with the second feature representation; and 
 generating, using the trained second machine learning model, a second classification of the user communications according to the second set of feedback categories. 
 
     
     
       17. The non-transitory machine-readable medium of  claim 16 , wherein the user feedback data further includes audio files corresponding to voice communications between users of the service provider and customer service agents associated with the service provider, and wherein the operations further comprise:
 converting each of the audio files into a text file including a transcription of the corresponding voice communications; 
 vectorizing the text file using a word embedding algorithm; and 
 extracting the first set of textual data features from the vectorized text file. 
 
     
     
       18. The non-transitory machine-readable medium of  claim 16 , wherein the operations further comprise:
 identifying, from the first machine learning model using a deep learning framework, the semantic equivalents of the first set of textual data features within the user communications; and 
 extracting, from the user communications, the second set of textual data features including the identified semantic equivalents. 
 
     
     
       19. The non-transitory machine-readable medium of  claim 16 , wherein the operations further comprise:
 clustering the user communications into a plurality of clusters using one or more clustering algorithms, each of the plurality of clusters corresponding to a different one of the feedback categories in the second set based on a unique pattern of the second set of textual features within the user communications. 
 
     
     
       20. The non-transitory machine-readable medium of  claim 16 , wherein the second machine learning model is selected from among a plurality of machine learning models, and wherein the operations further comprise:
 training each machine learning model in the plurality of machine learning models to reclassify the user communications based on the second feature representation; 
 comparing confidence scores for respective classifications of the user communications generated by the plurality of trained machine learning models; and 
 selecting, from among the plurality of trained machine learning models, at least one trained machine learning model as the second machine learning model based on the comparison, wherein the at least one trained machine learning model is used to generate the second classification of the user communications, and the second classification has a highest confidence score from among the respective classifications generated by the plurality of trained machine learning models.