Graphical element detection using a combination of user interface descriptor attributes from two or more graphical element detection techniques

Abstract

Graphical element detection using a combination of user interface (UI) descriptor attributes from two or more graphical element detection techniques is disclosed. UI descriptors may be used to compare attributes for a given UI descriptor with attributes of UI elements found at runtime in the UI. At runtime, the attributes for the UI elements found in the UI can be searched for matches with attributes for a respective RPA workflow activity, and if an exact match or a match within a matching threshold is found, the UI element may be identified and interacted with accordingly.

1 . A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to:
 compare user interface (UI) element attributes to UI descriptor attributes from a plurality of graphical element detection techniques for an activity of a robotic process automation (RPA) workflow; and   when a UI element matching the attributes of the plurality of graphical element detection techniques is found via an exact match or a threshold match:   take an action associated with the activity involving the UI element.   
     
     
         2 . The non-transitory computer-readable medium of  claim 1 , wherein the UI attributes comprise images, text, relationships between graphical elements in the UI, a hierarchical representation of the graphical elements in the UI, or a combination thereof. 
     
     
         3 . The non-transitory computer-readable medium of  claim 1 , wherein the process of  claim 1  is repeated for at least one additional activity. 
     
     
         4 . The non-transitory computer-readable medium of  claim 1 , wherein types of the UI descriptors comprise two or more of a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor. 
     
     
         5 . The non-transitory computer-readable medium of  claim 1 , wherein the computer program is configured to automatically utilize the plurality of graphical element detection techniques. 
     
     
         6 . The non-transitory computer-readable medium of  claim 5 , wherein the computer program is configured to automatically utilize UI descriptor attributes from the plurality of graphical element detection techniques. 
     
     
         7 . The non-transitory computer-readable medium of  claim 1 , wherein the computer program is or comprises an RPA robot. 
     
     
         8 . The non-transitory computer-readable medium of  claim 1 , wherein the computer program is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques. 
     
     
         9 . A computer-implemented method, comprising:
 analyzing a user interface (UI) at runtime to identify UI element attributes, by a computing system;   comparing the identified UI element attributes from the analysis to UI descriptor attributes from a plurality of graphical element detection techniques for an activity of a robotic process automation (RPA) workflow, by the computing system; and   when a UI element matching the attributes of the plurality of graphical element detection techniques is found via an exact match:   taking an action associated with the activity involving the UI element, by the computing system.   
     
     
         10 . The computer-implemented method of  claim 9 , wherein the UI attributes comprise images, text, relationships between graphical elements in the UI, a hierarchical representation of the graphical elements in the UI, or a combination thereof. 
     
     
         11 . The computer-implemented method of  claim 9 , wherein the method of  claim 9  is repeated for at least one additional activity. 
     
     
         12 . The computer-implemented method of  claim 9 , wherein types of the UI descriptors comprise two or more of a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor. 
     
     
         13 . The computer-implemented method of  claim 9 , wherein the computing system is configured to automatically utilize the plurality of graphical element detection techniques. 
     
     
         14 . The computer-implemented method of  claim 13 , wherein the computing system is configured to automatically utilize UI descriptor attributes from the plurality of graphical element detection techniques. 
     
     
         15 . The computer-implemented method of  claim 9 , wherein the computing system is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques. 
     
     
         16 . A computing system, comprising:
 memory storing computer program instructions; and   at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:   analyze a user interface (UI) at runtime to identify UI element attributes,   compare the identified UI element attributes from the analysis to UI descriptor attributes from a plurality of graphical element detection techniques for an activity of a robotic process automation (RPA) workflow, and   when a UI element matching the attributes of the plurality of graphical element detection techniques is found via a threshold match:   take an action associated with the activity involving the UI element.   
     
     
         17 . The computing system of  claim 16 , wherein the UI attributes comprise images, text, relationships between graphical elements in the UI, a hierarchical representation of the graphical elements in the UI, or a combination thereof. 
     
     
         18 . The computing system of  claim 16 , wherein the process of  claim 16  is repeated for at least one additional activity. 
     
     
         19 . The computing system of  claim 16 , wherein types of the UI descriptors comprise two or more of a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor. 
     
     
         20 . The computing system of  claim 16 , wherein the computer program instructions are configured to automatically utilize the plurality of graphical element detection techniques. 
     
     
         21 . The computing system of  claim 20 , wherein the computer program instructions are configured to automatically utilize UI descriptor attributes from the plurality of graphical element detection techniques. 
     
     
         22 . The computing system of  claim 16 , wherein the computer program instructions are or comprise an RPA robot. 
     
     
         23 . The computing system of  claim 16 , wherein the computer program instructions are configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques.