Model stacking for support ticket categorization

Abstract

Example methods and systems are directed to categorizing support tickets for more efficient handling by support staff. Support staff may be divided into multiple support groups. An incoming support ticket is converted to a machine representation and provided as input to one or more trained machine learning models. Based on the output from the one or more trained machine learning models, the support ticket is routed to one of the support groups. As a result, some tickets will be directly routed to higher-level support groups instead of having all tickets first be evaluated by L1 support personnel. Accordingly, support staff resources are conserved. A model stacking technique may be used in which models of varying complexities, ranging from very simple to highly complex, are stacked in sequence one after another.

What is claimed is: 
     
         1 . A system comprising:
 a memory that stores instructions; and   one or more processors coupled to the memory and configured to execute the instructions to perform operations comprising:
 providing a support ticket for a software application to a trained machine learning model as input; 
 receiving, from the trained machine learning model, a probability that the support ticket is addressed by modification of source code of the software application; 
 based on the probability and a predetermined threshold, selecting a support group to send the support ticket to; and 
 sending the support ticket to the selected support group. 
   
     
     
         2 . The system of  claim 1 , wherein the operations further comprise:
 generating the trained machine learning model by providing a training set comprising a set of historical support tickets, each of the historical support tickets labeled with a class of a set of classes comprising a minority class and a majority class, more of the historical support tickets having majority class labels than minority class labels.   
     
     
         3 . The system of  claim 2 , wherein the class that labels each historical support ticket identifies a support group that resolved the historical support ticket. 
     
     
         4 . The system of  claim 2 , wherein the operations further comprise:
 validating the trained machine learning model by determining a proportion of correct classifications of the historical support tickets classified by the trained machine learning model as the minority class out of a total number of classifications of the historical support tickets classified by the trained machine learning model as the minority class.   
     
     
         5 . The system of  claim 2 , wherein the operations further comprise:
 validating the trained machine learning model by determining a proportion of correct classifications of the historical support tickets classified by the trained machine learning model as the minority class out of a total number of the historical support tickets labeled as the minority class.   
     
     
         6 . The system of  claim 5 , wherein the minority class comprises no more than 10% of the support tickets and the majority class comprises at least 70% of the support tickets. 
     
     
         7 . The system of  claim 6 , wherein the generating of the trained machine learning model comprises applying a balanced log loss function that penalizes misclassifications of the minority class more than misclassifications of the majority class. 
     
     
         8 . The system of  claim 1 , wherein:
 the trained machine learning model is a second machine learning model;   the probability that the support ticket is addressed by modification of source code of the software application is a second probability;   the predetermined threshold is a second predetermined threshold;   the operations further comprise:
 providing the support ticket for a software application to a first trained machine learning model as input; and 
 receiving, from the first trained machine learning model, a first probability that the support ticket is addressed by modification of source code of the software application; and 
   the providing of the support ticket for the software application to the second machine learning model is based on the first probability and a first predetermined threshold.   
     
     
         9 . The system of  claim 8 , wherein the first machine learning model is a simpler model than the second machine learning model. 
     
     
         10 . A non-transitory computer-readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
 providing a support ticket for a software application to a trained machine learning model as input;   receiving, from the trained machine learning model, a probability that the support ticket is addressed by modification of source code of the software application;   based on the probability and a predetermined threshold, selecting a support group to send the support ticket to; and   sending the support ticket to the selected support group.   
     
     
         11 . The non-transitory computer-readable medium of  claim 10 , wherein the operations further comprise:
 generating the trained machine learning model by providing a training set comprising a set of historical support tickets, each of the historical support tickets labeled with a class of a set of classes comprising a minority class and a majority class, more of the historical support tickets having majority class labels than minority class labels.   
     
     
         12 . The non-transitory computer-readable medium of  claim 11 , wherein the class that labels each historical support ticket identifies a support group that resolved the historical support ticket. 
     
     
         13 . The non-transitory computer-readable medium of  claim 11 , wherein the operations further comprise:
 validating the trained machine learning model by determining a proportion of correct classifications of the historical support tickets classified by the trained machine learning model as the minority class out of a total number of classifications of the historical support tickets classified by the trained machine learning model as the minority class.   
     
     
         14 . The non-transitory computer-readable medium of  claim 11 , wherein the operations further comprise:
 validating the trained machine learning model by determining a proportion of correct classifications of the historical support tickets classified by the trained machine learning model as the minority class out of a total number of the historical support tickets labeled as the minority class.   
     
     
         15 . The non-transitory computer-readable medium of  claim 14 , wherein the minority class comprises no more than 10% of the support tickets and the majority class comprises at least 70% of the support tickets. 
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein the generating of the trained machine learning model comprises applying a balanced log loss function that penalizes misclassifications of the minority class more than misclassifications of the majority class. 
     
     
         17 . A method comprising:
 providing, by one or more processors, a support ticket for a software application to a trained machine learning model as input;   receiving, from the trained machine learning model, a probability that the support ticket is addressed by modification of source code of the software application;   based on the probability and a predetermined threshold, selecting a support group to send the support ticket to; and   sending, by the one or more processors, the support ticket to the selected support group.   
     
     
         18 . The method of  claim 17 , further comprising:
 generating the trained machine learning model by providing a training set comprising a set of historical support tickets, each of the historical support tickets labeled with a class of a set of classes comprising a minority class and a majority class, more of the historical support tickets having majority class labels than minority class labels.   
     
     
         19 . The method of  claim 18 , wherein the class that labels each historical support ticket identifies a support group that resolved the historical support ticket. 
     
     
         20 . The method of  claim 18 , further comprising:
 validating the trained machine learning model by determining a proportion of correct classifications of the historical support tickets classified by the trained machine learning model as the minority class out of a total number of classifications of the historical support tickets classified by the trained machine learning model as the minority class.