Graphical element detection using a combination of user interface descriptor attributes from two or more graphical element detection techniques

Abstract

Graphical element detection using a combination of user interface (UI) descriptor attributes from two or more graphical element detection techniques is disclosed. UI descriptors may be used to compare attributes for a given UI descriptor with attributes of UI elements found at runtime in the UI. At runtime, the attributes for the UI elements found in the UI can be searched for matches with attributes for a respective RPA workflow activity, and if an exact match or a match within a matching threshold is found, the UI element may be identified and interacted with accordingly.

1 . A computer-implemented method for detecting graphical elements in a user interface (UI), comprising:
 providing, by a designer application, a selection interface that facilitates selection of graphical element detection techniques of a plurality of graphical element detection techniques for individual activities of a robotic process automation (RPA) workflow on a per-activity basis;   receiving, by the designer application, a selection from the selection interface of an activity to be configured in the RPA workflow;   receiving, by the designer application, selections of and/or modifications to one or more graphical element detection techniques of the plurality of graphical element detection techniques, one or more UI descriptor attributes of the one or more graphical element detection techniques, or a combination thereof, for the selected activity; and   configuring the selected activity, by the designer application, based on the received selections and/or modifications.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 automatically configuring one or more of the plurality of graphical element detection techniques, one or more of the UI descriptor attributes, or both, by the designer application.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the automatic configuration is performed by the designer application based on an action implemented by the activity, a type of the target graphical element, a presence of one or more other graphical elements in the UI, or a combination thereof. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the process of  claim 1  is repeated for at least one additional activity in the RPA workflow. 
     
     
         5 . The computer-implemented method of  claim 1 , further comprising:
 generating an RPA robot configured to implement the configured activity, by the designer application.   
     
     
         6 . The computer-implemented method of  claim 5 , wherein the RPA robot is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques. 
     
     
         7 . The computer-implemented method of  claim 5 , further comprising:
 analyzing a UI at runtime, by the RPA robot, to identify UI element attributes;   comparing the UI element attributes to the configured UI descriptor attributes for the activity, by the RPA robot; and   when a UI element matching the attributes of the plurality of graphical element detection techniques is found via an exact match or a threshold match:   taking an action associated with the activity involving the UI element, by the RPA robot.   
     
     
         8 . The computer-implemented method of  claim 7 , wherein the UI attributes comprise images, text, relationships between graphical elements in the UI, a hierarchical representation of the graphical elements in the UI, or a combination thereof. 
     
     
         9 . The computer-implemented method of  claim 7 , wherein the process of  claim 7  is repeated for at least one additional activity. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein types of the UI descriptors comprise two or more of a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor. 
     
     
         11 . A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to:
 provide a selection interface that facilitates selection of graphical element detection techniques of a plurality of graphical element detection techniques for individual activities of a robotic process automation (RPA) workflow on a per-activity basis;   receive a selection from the selection interface of an activity to be configured in the RPA workflow;   receive selections of and/or modifications to one or more graphical element detection techniques of the plurality of graphical element detection techniques, one or more UI descriptor attributes of the plurality of graphical element detection techniques, or a combination thereof; and   configure the selected activity based on the received selections and/or modifications.   
     
     
         12 . The non-transitory computer-readable medium of  claim 11 , wherein the computer program is further configured to cause the at least one processor to:
 automatically configure one or more of the plurality of graphical element detection techniques, one or more of the UI descriptor attributes, or both.   
     
     
         13 . The non-transitory computer-readable medium of  claim 12 , wherein the automatic configuration is performed based on an action implemented by the activity, a type of the target graphical element, a presence of one or more other graphical elements in the UI, or a combination thereof. 
     
     
         14 . The non-transitory computer-readable medium of  claim 11 , wherein the process of  claim 11  is repeated for at least one additional activity in the RPA workflow. 
     
     
         15 . The non-transitory computer-readable medium of  claim 11 , wherein the computer program is further configured to cause the at least one processor to:
 generate an RPA robot configured to implement the configured activity.   
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein the RPA robot is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques. 
     
     
         17 . The non-transitory computer-readable medium of  claim 11 , wherein the types of UI descriptors comprise two or more of a selector, a computer vision (CV) descriptor, an image matching descriptor, and an optical character recognition (OCR) descriptor. 
     
     
         18 . A computing system, comprising:
 memory storing computer program instructions for detecting graphical elements in a user interface (UI); and   at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:   provide a selection interface that facilitates selection of graphical element detection techniques of a plurality of graphical element detection techniques for individual activities of a robotic process automation (RPA) workflow on a per-activity basis;   receive selections of multiple graphical element detection techniques of the plurality of graphical element detection techniques for an activity; and   configure the activity based on the received graphical element detection technique selections.   
     
     
         19 . The computing system of  claim 18 , wherein the computer program instructions are further configured to cause the at least one processor to:
 automatically configure one or more of the plurality of graphical element detection techniques, one or more of the UI descriptor attributes, or both.   
     
     
         20 . The computing system of  claim 18 , wherein the automatic configuration is performed based on an action implemented by the activity, a type of the target graphical element, a presence of one or more other graphical elements in the UI, or a combination thereof. 
     
     
         21 . The computing system of  claim 18 , wherein the process of  claim 18  is repeated for at least one additional activity in the RPA workflow. 
     
     
         22 . The computing system of  claim 18 , wherein the computer program instructions are further configured to cause the at least one processor to:
 generate an RPA robot configured to implement the configured activity.   
     
     
         23 . The computing system of  claim 18 , wherein the RPA robot is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques.