Evaluation of artificial intelligence models usable for a conversational artificial intelligence system

Abstract

Methods and systems are presented for providing an artificial intelligence (AI)-based conversation system for facilitating a conversation with users and processing transactions for the users. The AI-based conversation system includes an AI model coupled with different backend modules. Based on an utterance submitted by a user during a chat session, the AI model is configured to generate instructions for a backend module to perform a transaction for the user based on a prompt template. The AI model also communicates the instructions to the backend module using a protocol specified in the prompt template. Upon receiving an output from the backend module, the AI model is configured to generate content for the chat session based on the output, and provide the content to the user.

What is claimed is: 
     
         1 . A system comprising:
 a non-transitory memory; and   one or more hardware processors coupled with the non-transitory memory and configured to execute instructions from the non-transitory memory to cause the system to:
 monitor a conversation between an artificial intelligence (AI) model and a user conducted via a chat interface during a chat session; 
 derive a set of quality metrics for responses to the conversation based on attributes associated with the conversation; 
 determine a performance level of the AI model based on the set of quality metrics; and 
 modify the AI model based on the performance level. 
   
     
     
         2 . The system of  claim 1 , wherein executing the instructions further causes the system to:
 obtain a first response provided by the user via the chat interface during the chat session, the first response being responsive to a first question provided by the AI model during the chat session; and   evaluate a quality of the first response, wherein the set of quality metrics is derived based on the quality of the first response.   
     
     
         3 . The system of  claim 2 , wherein executing the instructions further causes the system to:
 determine that the quality of the first response falls below a threshold; and   in response to determining that the quality of the first response falls below the threshold, cause the AI model to generate a second question based on modifying the first question.   
     
     
         4 . The system of  claim 3 , wherein the first question was generated by the AI model to prompt the user for data corresponding to a particular data type or particular content using a first syntax, and wherein the second question is generated to prompt the user for the data corresponding to the particular data type or the particular content using a second content different from the first syntax. 
     
     
         5 . The system of  claim 1 , wherein executing the instructions further causes the system to:
 obtain data generated by the AI model and provided to the user via the chat interface during the chat session based on monitoring the conversation;   evaluate a quality of the data generated by the AI model, wherein the set of quality metrics is derived based on the quality of the data.   
     
     
         6 . The system of  claim 5 , wherein the quality of the data comprises a semantic metric and a syntactic metric. 
     
     
         7 . The system of  claim 5 , wherein evaluating the quality of the data comprises comparing the data against benchmark data. 
     
     
         8 . A method, comprising:
 monitoring interactions between a computer system and a user during an online chat session, wherein the computer system comprises an artificial intelligence (AI) model, and wherein the interactions comprise at least one dialogue between the AI model and the user conducted via a chat interface;   deriving a set of quality metrics for the interactions based on attributes associated with the interactions;   determining a performance level of the computer system based on the set of quality metrics; and   in response to determining that the performance level is below a threshold, modifying one or more components of the computer system.   
     
     
         9 . The method of  claim 1 , further comprising:
 determining a request for performing a transaction for the user based on the interactions, wherein the deriving the set of quality metrics for the interactions comprises determining a number of dialogue turns between the AI mode and the user before the transaction is completed.   
     
     
         10 . The method of  claim 8 , wherein the modifying the one or more components of the computer system comprises adjusting one or more parameters of the AI model. 
     
     
         11 . The method of  claim 8 , further comprising:
 selecting, from a plurality of prompt templates, a particular prompt template usable by the AI model to conduct a set of dialogues with the user, wherein the AI model is configured to generate a set of questions based on the particular prompt template, and wherein the modifying the one or more components of the computer system comprises modifying the particular prompt template.   
     
     
         12 . The method of  claim 8 , further comprising:
 determining a request for performing a transaction for the user based on the interactions;   selecting, from a plurality of prompt templates, a particular prompt template usable by the AI model to instruct a software module for performing the transaction; and   causing the AI model to generate instructions for the software module based on the particular prompt template, wherein the deriving the set of quality metrics comprises evaluating a quality of the instructions generated by the AI model based on a response from the user for the AI model to perform the transaction.   
     
     
         13 . The method of  claim 12 , wherein the modifying the one or more components of the computer system comprises modifying the particular prompt template in response to determining that the quality of the instructions is below a quality threshold. 
     
     
         14 . The method of  claim 8 , wherein the at least one dialogue comprises a request for information related to a topic provided by the user, and wherein the method further comprises:
 causing the AI model to generate instructions for a software module to obtain data related to the topic from a plurality of data sources; and   generating, by the AI model, content for the user based on the data, wherein the deriving the set of quality metrics comprises evaluating a quality of the content generated by the AI model.   
     
     
         15 . A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
 accessing a conversation conducted between an artificial intelligence (AI) model and a user via a chat interface during a chat session;   deriving a set of quality metrics for the conversation based on attributes associated with the conversation;   determining a performance level of the AI model based on the set of quality metrics; and   modifying the AI model based on the performance level.   
     
     
         16 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise:
 obtaining a first response provided by the user via the chat interface during the chat session based on the accessing the conversation, the first response being responsive to a first question provided by the AI model during the chat session; and   evaluating a quality of the first response, wherein the set of quality metrics is derived based on the quality of the first response.   
     
     
         17 . The non-transitory machine-readable medium of  claim 16 , wherein the first question prompts the user for first data corresponding to a first data type, and wherein the evaluating the quality of the first response comprises determining whether the first response comprises the first data corresponding to the first data type. 
     
     
         18 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise:
 obtaining a first question generated by the AI model and provided to the user via the chat interface based on the accessing the conversation, wherein the first question prompts the user for data corresponding to a particular data type using a first syntax;   determining that a first answer provided by the user does not correspond to the particular data type; and   causing the AI model to generate a second question to prompt the user for the data corresponding to the particular data type using a second syntax different from the first syntax.   
     
     
         19 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise:
 obtaining content generated by the AI model and provided to the user via the chat interface during the chat session based on the accessing the conversation;   evaluating a quality of the content, wherein the set of quality metrics is derived based on the quality of the content.   
     
     
         20 . The non-transitory machine-readable medium of  claim 19 , wherein the quality of the content comprises a semantic quality and a syntactic quality.