Automating execution of arbitrary graphical interface applications

Abstract

Interaction with an application interface is monitored and an image of the interface prior to a user interaction is compared with an image after the user interaction to identify a region of interaction. The comparison may be carried out by detecting that a threshold change has occurred using a computationally inexpensive algorithm and then drawing boundaries for the region using a more computationally intensive algorithm. A marker in a UI evolution timeline is associated with an image of the region and coordinates of the interaction normalized and relative to the region. An interaction interval and a profile of change are also stored in association with the marker. For playback, a script can be constructed from the timeline to control automated execution of the application by matching a current interval and profile to the interval and profile for a recorded interaction, and automatically applying that interaction to the current user interface.

1 . A method of recording the evolution of a user interface for a software application, comprising:
 monitoring image variations in the user interface as a user interacts with the software application executing on a computer system;   detecting a user interaction with the user interface during said monitoring;   comparing a first image of the user interface immediately prior to the user interaction with a second image of the user interface immediately after the user interaction to identify a region of interaction related to the user interaction;   placing a marker in a user interface evolution timeline to indicate a point in time when the user interaction occurred; and   storing an image of the region of interaction and coordinates of the user interaction in association with the marker.   
     
     
         2 . The method of  claim 1  wherein the coordinates of the user interaction are normalized and relative to the region of interaction. 
     
     
         3 . The method of  claim 1  wherein said comparing includes:
 detecting that a change in the user interface has occurred using a fuzzy pattern matching algorithm which compares the first image to the second image; and 
 responsive to said detecting, drawing boundaries for the region of interaction using image diffs between the first image and the second image. 
 
     
     
         4 . The method of  claim 1 , further comprising:
 determining that the user interaction resulted in a new user interface element which is present in the second image but not present in the first image; and   responsive to said determining, creating the image of the region of interaction using the first image.   
     
     
         5 . The method of  claim 1 , further comprising:
 determining that the user interaction resulted in removal of a user interface element which is present in the first image but not present in the second image; and   responsive to said determining, creating the image of the region of interaction using the first image.   
     
     
         6 . The method of  claim 1 , further comprising storing an interaction interval and a profile of change for the user interaction in association with the marker. 
     
     
         7 . The method of  claim 6 , further comprising:
 constructing a GUI script from the user interface evolution timeline; and   controlling automated execution of the software application using the GUI script, wherein said controlling includes matching a current interaction interval and current profile to the interaction interval and profile of change for the user interaction and, responsive to said matching, locating a target region in an image of the current user interface of the automatically executing software application which corresponds to the region of interaction and applying the user interaction to the current user interface based on the coordinates normalized and relative to the target region.   
     
     
         8 .- 21 . (canceled) 
     
     
         22 . A method of automating execution of a graphical interface application, comprising:
 receiving a metric record of prior usage of the graphical interface application including a plurality of user interactions with a user interface of the graphical interface application, each user interaction having a recorded region of interaction image, a recorded interaction interval, a recorded profile of change, and recorded interaction coordinates;   monitoring image variations in the user interface of the graphical interface application;   matching a current interaction interval and current profile for the user interface of the graphical interface application to a recorded interaction interval and recorded profile of change for a matching one of the user interactions; and   locating a target region in an image of the user interface of the graphical interface application which corresponds to a recorded region of interaction image for the matching user interaction; and   automatically applying the matching user interaction to the user interface of the graphical interface application using the recorded interaction coordinates.   
     
     
         23 . The method of  claim 22  wherein the target region is located using normalized cross-correlation. 
     
     
         24 . The method of  claim 22  wherein the recorded interaction coordinates are applied normalized and relative to the target region.