Learning user interface controls via incremental data synthesis

Abstract

A User Interface (UI) interface object detection system employs an initial dataset comprising a set of images, that may include synthesized images, to train a Machine Learning (ML) engine to generate an initial trained model. A data point generator is employed to generate an updated synthesized image set which is used to further train the ML engine. The data point generator may employ images generated by an application program as a reference by which to generate the updated synthesized image set. The images generated by the application program may be tagged in advance. Alternatively, or in addition, the images generated by the application program may be captured dynamically by a user using the application program.

What is claimed is: 
     
         1 . A computer-implemented method for detecting one or more user interface control objects contained in a screen image of a user interface generated by an application program, the method comprising:
 providing an initial dataset comprising a plurality of images, wherein each image represents a screen of information containing one or more user interface control objects, wherein each user interface control object is part of a corresponding user interface object class;   modifying the initial dataset by,
 processing screen images generated by a first application program to detect user interface control objects in the screen images generated by the first application program, 
 creating a set of new application screen images as a function of the user interface control objects detected in the screen images generated by the first application program, and 
 modifying the initial dataset with the set of new application screen images to produce a modified dataset; and 
   training a machine learning model with the modified dataset to generate a trained machine learning model, the machine learning model configured to detect user interface control objects within images representing a screen of information containing one or more user interface control objects.