Automatically determining workstreams from common actions in online applications

Abstract

An embodiment uses a computer-implemented approach to identify and automate workstreams from common actions in online applications. A browser extension collects and processes user event metadata, which is used to train machine learning models for workflow automation. The extension reads relevant fields from web services and delivers prompts with high-level actions (HLAs) based on user or organizational details. These HLAs are appended to prompts and sent to a large language model (LLM) via an API call. The LLM responds with a solution and a sequence of actions, which the browser extension executes automatically. Users retain control to pause, accelerate, or modify these actions. This method ensures secure, end-to-end workflow automation across various computing devices, enabling efficient and autonomous resolution of tasks while maintaining user oversight. The browser extension processes metadata, not enterprise data, ensuring security and privacy.

What is claimed is: 
     
         1 . One or more non-transitory computer-readable storage media storing a workflow automation application comprising one or more sequences of instructions which, when executed using one or more processors, the one or more processors being communicatively coupled to one or more network interfaces that are communicatively coupled to one or more internetworks and capable of network communication with a program hosted on an agent computer that also hosts a browser, cause the one or more processors to execute:
 receiving browser tab event data indicating a change to a new or different browser tab associated with navigation of the browser to a web service;   in response to receiving the browser tab event data indicating the change, extracting web service data from a web page served to the browser from the web service;   retrieving, from digital data storage, a first prompt text for a first prompt to a large language model (LLM) and a first list of actions from a vocabulary of common user actions;   programmatically calling the LLM and providing a first engineered prompt comprising the first prompt text, the first list of actions, and the web service data;   receiving a response to the first engineered prompt from the LLM, the response comprising a second list of actions; and   posting a plurality of requests to the web service, each request among the plurality of requests specifying to execute a different successive action in the second list of actions.   
     
     
         2 . The one or more non-transitory computer-readable storage media of  claim 1 , wherein the vocabulary of common user actions comprises actions selected by a trained machine learning model. 
     
     
         3 . The one or more non-transitory computer-readable storage media of  claim 2 , wherein the actions selected by the trained machine learning model are validated based on a determination that the actions are similar to previously detected actions. 
     
     
         4 . The one or more non-transitory computer-readable storage media of  claim 3 , wherein the actions selected by the trained machine learning model are summarized by the LLM. 
     
     
         5 . The one or more non-transitory computer-readable storage media of  claim 4 , wherein the trained machine learning model is trained with a training data set comprising an additional set of actions summarized by the LLM. 
     
     
         6 . The one or more non-transitory computer-readable storage media of  claim 1 , wherein the vocabulary of common user actions comprises second actions, and wherein the one or more sequences of instructions, when executed using the one or more processors, cause the one or more processors to execute:
 retrieving, from digital data storage, a second prompt text and structured event data representing discrete user events of an event stream;   programmatically calling the LLM and providing a second engineered prompt comprising the second prompt text and the structured event data;   receiving a response to the second engineered prompt from the LLM, the response comprising the vocabulary of common user actions and a summary of the vocabulary; and   storing the vocabulary and summary in digital data storage.   
     
     
         7 . The one or more non-transitory computer-readable storage media of  claim 6 , wherein the program comprises a browser extension. 
     
     
         8 . The one or more non-transitory computer-readable storage media of  claim 7 , wherein the browser extension is configured to monitor and record an execution of the second actions on the web service and store the second actions in digital data storage. 
     
     
         9 . The one or more non-transitory computer-readable storage media of  claim 1 , wherein the one or more sequences of instructions, when executed using the one or more processors, cause the one or more processors to execute:
 inspecting the vocabulary of common user actions;   identifying frequently performed groups of actions in the vocabulary of common user actions; and   storing a selected group of actions from the frequently performed groups of actions as a workstream.   
     
     
         10 . The one or more non-transitory computer-readable storage media of  claim 9 , wherein storing a selected group of actions further comprises:
 prompting a user to confirm the workstream; and   storing the workstream in response to receiving user confirmation.   
     
     
         11 . The one or more non-transitory computer-readable storage media of  claim 9 , wherein identifying frequently performed groups of actions comprises comparing a number of times an action has been performed to a threshold value. 
     
     
         12 . The one or more non-transitory computer-readable storage media of  claim 9 , further comprising:
 detecting, by the program, an additional navigation to the web service; and   displaying, by the program, a visual list of one or more workstreams corresponding to the web service.   
     
     
         13 . The one or more non-transitory computer-readable storage media of  claim 12 , further comprising:
 executing a displayed workstream in the visual list of one or more workstreams by executing the selected group of actions grouped in the workstream.   
     
     
         14 . A computer-implemented method, comprising:
 detecting, by a program hosted on an agent computer, a change to a new or different browser tab associated with a navigation to a web service;   in response to detecting the change, extracting, by a workflow automation computer communicatively coupled to one or more internetworks and capable of network communication with the program hosted on an agent computer, web service data from the web service;   retrieving, by the workflow automation computer, prompt text for a large language model (LLM) and a first list of actions from a vocabulary of common user actions;   calling, by the workflow automation computer, the LLM with an engineered prompt comprising prompt text, the first list of actions, and the web service data;   receiving by the workflow automation computer, a response to the engineered prompt from the LLM, the response comprising a second list of actions; and   executing, by the program, instructions to the web service to execute each action in the second list of actions.   
     
     
         15 . The computer-implemented method of  claim 14 , wherein the vocabulary of common user actions comprises actions selected by a trained machine learning model. 
     
     
         16 . The computer-implemented method of  claim 15 , wherein the actions selected by the trained machine learning model are validated based on a determination that the actions are similar to previously detected actions. 
     
     
         17 . The computer-implemented method of  claim 16 , wherein the actions selected by the trained machine learning model are summarized by the LLM. 
     
     
         18 . The computer-implemented method of  claim 17 , wherein the trained machine learning model is trained with a training data set comprising an additional set of actions summarized by the LLM. 
     
     
         19 . The computer-implemented method of  claim 14 , further comprising:
 retrieving, from digital data storage, a second prompt text for a second prompt to the LLM and structured event data representing discrete user events of an event stream;   programmatically calling the LLM and providing a second engineered prompt comprising the prompt text and the structured event data;   receiving a response to the engineered prompt from the LLM, the response comprising the vocabulary of common user actions and a summary of the vocabulary; and   storing the vocabulary and summary in digital data storage.   
     
     
         20 . The computer-implemented method of  claim 14 , further comprising:
 inspecting, by the workflow automation computer, the vocabulary of common user actions;   identifying, by the workflow automation computer, frequently performed groups of actions in the vocabulary of common user actions; and   storing, by the workflow automation computer, a selected group of actions from the frequently performed groups of actions as a workstream.   
     
     
         21 . The computer-implemented method of  claim 20 , wherein storing a selected group of actions further comprises:
 prompting a user to confirm the workstream; and   storing the workstream in response to receiving user confirmation.   
     
     
         22 . The computer-implemented method of  claim 20 , wherein identifying frequently performed groups of actions comprises comparing a number of times an action has been performed to a threshold value. 
     
     
         23 . The computer-implemented method of  claim 20 , further comprising:
 detecting, by the program, an additional navigation to the web service; and   displaying, by the program, a visual list of one or more workstreams corresponding to the web service.   
     
     
         24 . The computer-implemented method of  claim 23 , further comprising:
 executing a displayed workstream in the visual list of one or more workstreams by executing the selected group of actions grouped in the workstream.