Updating natural language interfaces by processing usage data

Abstract

A third-party company may assist companies in providing natural language interfaces for their customers. To implement a natural language interface for a company, a configuration may be received that includes information, such as a list intents, seed messages for the intents, and hierarchical information of the intents. An intent classifier may be trained using the configuration, and the natural language interface may be deployed for use with customers. Usage data of the natural language classifier may be collected and used to improve the natural language interface. Messages corresponding to an intent may be clustered into clusters of similar messages, and a prototype message may be obtained for each cluster to provide a human understandable description of the cluster. The information about the clusters may be used to improve the natural language interface, such as by creating a new intent with a cluster or moving a cluster to a different intent.

What is claimed is: 
     
         1 . A system for implementing a natural language interface, the system comprising:
 at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to:   deploy the natural language interface, wherein the natural language interface is configured to recognize a list of intents;   collect usage data for the natural language interface, wherein the usage data comprises messages received from users of the natural language interface and intents selected by the natural language interface as corresponding to the messages;   obtain a first set of messages from the usage data, wherein the first set of messages corresponds to a first intent of the list of intents;   cluster the first set of messages into a plurality of clusters;   obtain a first prototype message for a first cluster of the plurality of clusters, wherein the first prototype message represents messages of the first cluster;   present the first prototype message;   receive an input to (i) create a new intent using the first cluster or (ii) move the first cluster to a second intent of the list of intents;   update the natural language interface using the received input;   deploy the updated natural language interface.   
     
     
         2 . The system of  claim 1 , wherein the system is implemented by a third-party company providing services to a company and the at least one server computer is configured to:
 provide a user interface to the company to allow the company to configure the natural language interface; and   receive a configuration for the natural language interface from the company.   
     
     
         3 . The system of  claim 2 , wherein the at least one server computer is configured to:
 receive a second configuration for a second natural language interface for a second company; and   deploy the second natural language interface, wherein the second natural language interface is used by customers of the second company.   
     
     
         4 . The system of  claim 1 , wherein the usage data includes an indication of success for one or more of the messages. 
     
     
         5 . The system of  claim 4 , wherein the at least one server computer is configured to compute a success rate for the first cluster. 
     
     
         6 . The system of  claim 4 , wherein the at least one server computer is configured to compute a success rate for the first intent. 
     
     
         7 . The system of  claim 4 , wherein a first indication of success for a first message of a first user is determined by one or more of:
 asking the first user if a first interaction corresponding to the first message was a success;   asking a customer service representative assisting the first user if the first interaction was a success; or   determining that the first user completed an action.   
     
     
         8 . The system of  claim 1 , wherein the at least one server computer is configured to:
 determine to present a suggestion for improving the natural language interface;   present a suggestion (i) to create the new intent using the first cluster or (ii) to move the first cluster to the second intent.   
     
     
         9 . The system of  claim 1 , wherein the natural language interface processes a message received from a user, determines an intent of the message, selects an action using the intent, and causes the action to be implemented with the user. 
     
     
         10 . A computer-implemented method for implementing a natural language interface, the method comprising:
 deploying the natural language interface, wherein the natural language interface is configured to recognize a list of intents;   collecting usage data for the natural language interface, wherein the usage data comprises wherein the usage data comprises messages received from users of the natural language interface and intents selected by the natural language interface as corresponding to the messages;   obtaining a first set of messages from the usage data, wherein the first set of messages corresponds to a first intent of the list of intents,   clustering the first set of messages into a plurality of clusters, and   obtaining a first prototype message for a first cluster of the plurality of clusters, wherein the first prototype message represents messages of the first cluster;   presenting the first prototype message;   receiving an input to (i) create a new intent using a first cluster or (ii) move the first cluster to a second intent of the list of intents;   updating the natural language interface using the received input; and   deploying the updated natural language interface.   
     
     
         11 . The method of  claim 10 , wherein the natural language interface comprises a recurrent neural network. 
     
     
         12 . The method of  claim 10 , wherein the natural language interface comprises a specification of an action to be performed with a user for each intent of the list of intents. 
     
     
         13 . The method of  claim 10 , wherein clustering the first set of messages comprises:
 computing message embeddings for the first set of messages; and   clustering the message embeddings.   
     
     
         14 . The method of  claim 10 , wherein obtaining the first prototype message comprises selecting a message with a largest count or selecting a message closest to a center of the first cluster. 
     
     
         15 . The method of  claim 10 , wherein obtaining the first prototype message comprises generating the first prototype message with a neural network. 
     
     
         16 . The method of  claim 10 , comprising computing a success rate for the first intent. 
     
     
         17 . One or more non-transitory computer-readable media comprising computer executable instructions that, when executed, cause at least one processor to perform actions comprising:
 deploying a natural language interface, wherein the natural language interface is configured to recognize a list of intents;   collecting usage data for the natural language interface, wherein the usage data comprises wherein the usage data comprises messages received from users of the natural language interface and intents selected by the natural language interface as corresponding to the messages;   obtaining a first set of messages from the usage data, wherein the first set of messages corresponds to a first intent of the list of intents,   clustering the first set of messages into a plurality of clusters, and   obtaining a first prototype message for a first cluster of the plurality of clusters, wherein the first prototype message represents messages of the first cluster;   presenting the first prototype message;   receiving an input to (i) create a new intent using a first cluster or (ii) move the first cluster to a second intent of the list of intents;   updating the natural language interface using the received input; and   deploying the updated natural language interface.   
     
     
         18 . The one or more non-transitory computer-readable media of  claim 17 , wherein the natural language interface was trained using a plurality of seed messages for each intent of the list of intents. 
     
     
         19 . The one or more non-transitory computer-readable media of  claim 17 , wherein the actions comprise:
 training an intent classifier using a plurality of seed messages for each intent of the list of intents; and   wherein the natural language interface comprises the intent classifier.   
     
     
         20 . The one or more non-transitory computer-readable media of  claim 17 , wherein the actions comprise presenting a suggestion (i) to create the new intent using the first cluster or (ii) to move the first cluster to the second intent.