Scenario generation method in which various data are associated with each other, scenario execution method in which various data are associated with each other, scenario generation device, and scenario execution device

Abstract

A scenario generation device includes a processor that executes a procedure. The procedure includes: detecting data representing a target object of an operation target and data representing a user operation on the target object based on objects displayed on a screen by application software operating on a computer, detecting data representing a peripheral object positioned at the target object periphery from out of the objects displayed on the screen, and detecting a positional relationship on the screen between the target object and the peripheral object; and generating a scenario in which the data representing the user operation, the data representing the target object, the data representing the peripheral object, and data representing the positional relationship are associated with each other.

What is claimed is: 
     
         1 . A scenario execution device comprising:
 a processor, and   a memory storing instructions, which when executed by the processor perform a procedure, the procedure including:   reading-in a scenario including data representing a target object of an operation target displayed as an image on a screen by application software operating on a computer, data representing a user operation on the target object, data representing a peripheral object that is a detected image as an image positioned around the target object, and data representing a positional relationship on the screen of the peripheral object to the target object;   detecting whether the scenario includes data representing a command to operate a mouse;   in a case that the scenario includes data representing the command to operate the mouse, acquiring data representing an image of a region on the screen that is operable by the user operation;   detecting whether the acquired image includes an image representing a scroll target region;   in a case that the acquired image includes the image representing a scroll target region, generating image data representing an image of an entire scroll region by executing an image data generation process;   identifying a position of the target object in the region operable by the user operation based on the data representing the target object, the data representing the peripheral object, and the data representing the positional relationship that are contained in the read-in scenario, and based on the data representing the image of the user operable region; and   executing the operation on the target object at the identified position based on the data representing the user operation on the target object contained in the scenario.   
     
     
         2 . The scenario execution device of  claim 1 , wherein the image data generation process includes
 acquiring, as the data representing an image of a region operable by the user operation, a plurality of data each representing an image of a display region on the screen at the time of execution of the application software; and   generating data representing a stitched image of a user operation region operable by the user operation based on the acquired data representing the plurality of images.   
     
     
         3 . The scenario execution device of  claim 1 , wherein at least a portion of the target object or a portion of the peripheral object is a character string. 
     
     
         4 . The scenario execution device of  claim 1 , wherein the procedure further includes:
 reading-in a scenario in which data representing a priority ranking in accordance with a separation distance between the target object and the peripheral object is associated with each of the data representing the positional relationships between the target object and a plurality of peripheral objects; and   identifying a position of the target object based on the data representing the priority ranking that was read-in.   
     
     
         5 . The scenario execution device of  claim 1 , wherein the procedure further includes:
 determining that the scenario is not suitable for execution on the application software in a case in which a position of the target object cannot be identified.   
     
     
         6 . The scenario execution device of  claim 1 , wherein the procedure further includes:
 identifying a candidate for the target object; and   in a case that the identified candidate does not match the target object, identifying an image representing an icon as a candidate for the target object.   
     
     
         7 . A non-transitory computer-readable recording medium having stored thereon a program for causing a computer to execute a scenario execution process, the process comprising:
 reading-in a scenario including data representing a target object of an operation target displayed as an image on a screen by application software operating on a computer, data representing a user operation on the target object, data representing a peripheral object that is a detected image as an image positioned around the target object, and data representing a positional relationship on the screen of the peripheral object to the target object;   detecting whether the scenario includes data representing a command to operate a mouse;   in a case that the scenario includes data representing the command to operate the mouse, acquiring data representing an image of a region on the screen that is operable by the user operation;   detecting whether the acquired image includes an image representing a scroll target region;   in a case that the acquired image includes the image representing a scroll target region, generating image data representing an image of an entire scroll region by executing an image data generation process;   identifying a position of the target object in the region operable by the user operation based on the data representing the target object, the data representing the peripheral object, and the data representing the positional relationship that are contained in the read-in scenario, and based on the data representing the image of the user operable region; and   executing the operation on the target object at the identified position based on the data representing the user operation on the target object contained in the scenario.   
     
     
         8 . The non-transitory computer-readable recording medium of  claim 7 , wherein the image data generation process includes
 acquiring, as the data representing an image of a region operable by the user operation, a plurality of data each representing an image of a display region on the screen at the time of execution of the application software; and   generating data representing a stitched image of a user operation region operable by the user operation based on the acquired data representing the plurality of images.   
     
     
         9 . The non-transitory computer-readable recording medium of  claim 7 , wherein at least a portion of the target object or a portion of the peripheral object is a character string. 
     
     
         10 . The non-transitory computer-readable recording medium of  claim 7 , wherein the scenario execution process further includes:
 reading-in a scenario in which data representing a priority ranking in accordance with a separation distance between the target object and the peripheral object is associated with each of the data representing the positional relationships between the target object and a plurality of peripheral objects; and   identifying a position of the target object based on the data representing the priority ranking that was read-in.   
     
     
         11 . The non-transitory computer-readable recording medium of  claim 7 , wherein the scenario execution process further includes:
 determining that the scenario is not suitable for execution on the application software in a case in which a position of the target object cannot be identified.   
     
     
         12 . The non-transitory computer-readable recording medium of  claim 7 , wherein the scenario execution process further comprises:
 identifying a candidate for the target object;   in a case that the identified candidate does not match the target object, identifying an image representing an icon as a candidate for the target object.   
     
     
         13 . A scenario execution method comprising:
 by a processor, reading-in a scenario including data representing a target object of an operation target displayed as an image on a screen by application software operating on a computer, data representing a user operation on the target object, data representing a peripheral object that is a detected image as an image positioned around the target object, and data representing a positional relationship on the screen of the peripheral object to the target object;   detecting whether the scenario includes data representing a command to operate a mouse;   in a case that the scenario includes data representing the command to operate the mouse, acquiring data representing an image of a region on the screen operable by the user operation;   detecting whether the acquired image includes an image representing a scroll target region;   in a case that the acquired image includes the image representing a scroll target region, generating image data representing an image of an entire scroll region by executing an image data generation process;   identifying a position of the target object in the region operable by the user operation based on the data representing the target object, the data representing the peripheral object, and the data representing the positional relationship that are contained in the read-in scenario, and based on the data representing the image of the user operable region; and   executing the operation on the target object at the identified position based on the data representing the user operation on the target object contained in the scenario.   
     
     
         14 . The scenario execution method of  claim 13 , wherein the image data generation process includes
 acquiring, as the data representing an image of a region operable by the user operation, a plurality of data each representing an image of a display region on a screen at the time of application software execution; and   generating data representing a stitched image of a user operation region operable by the user operation based on the acquired data representing the plurality of images.   
     
     
         15 . The scenario execution method of  claim 13 , wherein at least a portion of the target object or a portion of the peripheral object is a character string. 
     
     
         16 . The scenario execution method of  claim 13 , further comprising:
 reading-in a scenario in which data representing a priority ranking in accordance with a separation distance between the target object and the peripheral object is associated with each of the data representing the positional relationships between the target object and a plurality of peripheral objects; and   identifying a position of the target object based on the data representing the priority ranking that was read-in.   
     
     
         17 . The scenario execution method of  claim 13 , further comprising:
 determining that the scenario is not suitable for execution on the application software in a case in which a position of the target object cannot be identified.   
     
     
         18 . The scenario execution method of  claim 13 , further comprising:
 identifying a candidate for the target object; and   in a case that the identified candidate does not match the target object, identifying an image representing an icon as a candidate for the target object.