Insight generation to facilitate interpretation of inference model outputs

Abstract

Methods and systems for interpreting outputs from inference models are disclosed. To interpret the outputs, questions usable to establish a level of confidence in the outputs may be obtained using a first large language model (LLM) and a second LLM. The questions and contextual data usable to contextualize the questions may be ingested by a third LLM trained to generate insights. The insights may be intended to provide a response to the questions. The insights may be evaluated to determine whether the insights are acceptable. If the insights are acceptable, the insights may be provided to a downstream consumer for use in providing computer-implemented services. If the insights are not considered acceptable, the questions may be iteratively modified until insights based on the modified questions are considered acceptable.

What is claimed is: 
     
         1 . A method of interpreting an output generated by an inference model, the method comprising:
 obtaining a question, the question being usable to establish a level of confidence in the output and the question being generated by a second large language model (LLM);   obtaining, using at least the question and contextual data, insights intended to provide a response to the question, the insights being generated by a third LLM;   making a first determination, based on a level of success of the insights, regarding whether the insights are acceptable; and   in a first instance of the first determination in which the insights are acceptable:
 providing the insights to a downstream consumer for use in interpreting the output. 
   
     
     
         2 . The method of  claim 1 , further comprising:
 in a second instance of the first determination in which the insights are not acceptable:
 obtaining updated insights; 
 making a second determination regarding whether the updated insights are acceptable; and 
 in a first instance of the second determination in which the updated insights are not acceptable:
 continuing to iteratively modify the updated insights until the modified updated insights are acceptable. 
 
   
     
     
         3 . The method of  claim 2 , wherein obtaining the updated insights comprises:
 modifying the question to obtain an updated question; and   using the updated question as input for the third LLM to generate the updated insights.   
     
     
         4 . The method of  claim 2 , wherein obtaining the updated insights comprises:
 providing instructions to the third LLM, the instructions indicating that a portion of the contextual data is not to be used to generate the updated insights.   
     
     
         5 . The method of  claim 1 , wherein obtaining the question comprises:
 obtaining, based on at least the output, analytic data generated by a first large language model (LLM), the analytic data comprising:
 leading indicators from ingest data used by the inference model to generate the output; and 
 emerging trends from the ingest data used by the inference model to identify the leading indicators; and 
   obtaining, using at least the analytic data and a set of question generation templates, the question generated by a second LLM.   
     
     
         6 . The method of  claim 5 , wherein obtaining the analytic data comprises:
 feeding first ingest data into the first LLM, the first ingest data comprising:
 inference model ingest data used by the inference model to generate the output; 
 the output; and 
 a set of queries comprising questions to be answered by the first LLM, the questions being based on the inference model ingest data and the output; and 
   obtaining, as output from the first LLM, the analytic data.   
     
     
         7 . The method of  claim 6 , wherein the question is usable to identify facts to establish a causal relationship between at least a portion of the output and at least a portion of the analytic data. 
     
     
         8 . The method of  claim 1 , wherein making the first determination comprises:
 obtaining manual insights, the manual insights being generated by a subject matter expert (SME) and being based on the question and the contextual data;   comparing, using a fourth LLM, the manual insights to the insights to obtain a similarity score, the similarity score indicating a degree of similarity between the insights and the manual insights;   making a third determination, based on the similarity score and success criteria, regarding whether the insights are acceptable, the insights being considered acceptable when the similarity score meets the success criteria.   
     
     
         9 . The method of  claim 8 , wherein the level of success indicates an extent to which the similarity score meets the success criteria. 
     
     
         10 . The method of  claim 9 , further comprising:
 in the first instance of the first determination in which the insights are acceptable:
 applying reinforced learning to the second LLM using at least the question to increase a likelihood of questions generated by the second LLM at future points in time being usable to obtain insights that meet the success criteria. 
   
     
     
         11 . The method of  claim 1 , wherein the contextual data comprises economic report data. 
     
     
         12 . The method of  claim 1 , wherein the output comprises a prediction for a condition impacting a business at a future point in time. 
     
     
         13 . The method of  claim 12 , wherein the condition impacting the business at the future point in time is a change in availability of supply of a product from a supplier. 
     
     
         14 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for interpreting an output generated by an inference model, the operations comprising:
 obtaining a question, the question being usable to establish a level of confidence in the output and the question being generated by a second large language model (LLM);   obtaining, using at least the question and contextual data, insights intended to provide a response to the question, the insights being generated by a third LLM;   making a first determination, based on a level of success of the insights, regarding whether the insights are acceptable; and   in a first instance of the first determination in which the insights are acceptable:
 providing the insights to a downstream consumer for use in interpreting the output. 
   
     
     
         15 . The non-transitory machine-readable medium of  claim 14 , further comprising:
 in a second instance of the first determination in which the insights are not acceptable:
 obtaining updated insights; 
 making a second determination regarding whether the updated insights are acceptable; and 
 in a first instance of the second determination in which the updated insights are not acceptable:
 continuing to iteratively modify the updated insights until the modified updated insights are acceptable. 
 
   
     
     
         16 . The non-transitory machine-readable medium of  claim 15 , wherein obtaining the updated insights comprises:
 modifying the question to obtain an updated question; and   using the updated question as input for the third LLM to generate the updated insights.   
     
     
         17 . The non-transitory machine-readable medium of  claim 15 , wherein obtaining the updated insights comprises:
 providing instructions to the third LLM, the instructions indicating that a portion of the contextual data is not to be used to generate the updated insights.   
     
     
         18 . A data processing system, comprising:
 a processor; and   a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for interpreting an output generated by an inference model, the operations comprising:
 obtaining a question, the question being usable to establish a level of confidence in the output and the question being generated by a second large language model (LLM); 
 obtaining, using at least the question and contextual data, insights intended to provide a response to the question, the insights being generated by a third LLM; 
 making a first determination, based on a level of success of the insights, regarding whether the insights are acceptable; and 
 in a first instance of the first determination in which the insights are acceptable:
 providing the insights to a downstream consumer for use in interpreting the output. 
 
   
     
     
         19 . The data processing system of  claim 18 , further comprising:
 in a second instance of the first determination in which the insights are not acceptable:
 obtaining updated insights; 
 making a second determination regarding whether the updated insights are acceptable; and 
 in a first instance of the second determination in which the updated insights are not acceptable:
 continuing to iteratively modify the updated insights until the modified updated insights are acceptable. 
 
   
     
     
         20 . The data processing system of  claim 19 , wherein obtaining the updated insights comprises:
 modifying the question to obtain an updated question; and   using the updated question as input for the third LLM to generate the updated insights.