Machine-learned architecture for structured synthetic data generation

Abstract

Techniques may generate realistic synthetic data by programmatically generating a configuration file object type and relationship data. This configuration file may be used to retrieve source data matching the object type(s) and/or specific records indicated by the configuration file. The techniques may detect and anonymize private/proprietary information and may determine statistical characteristic(s) of the source data. A batch of prompt(s) may be generated using the source data, the statistical characteristic(s), and the configuration file and may be transmitted to one or more instances of a transformer-based machine-learned model. Sets of synthetic data received from the model instance(s) may be de-duplicated, checked for similarity to the source data (e.g., via embedding the synthetic data and the source data), and may be used to generate synthetic object(s) using the relationship(s) and/or other data indicated by the configuration file. These synthetic object(s) may then be deployed in a software environment.

What is claimed is: 
     
         1 . A system comprising:
 one or more processors; and   one or more non-transitory computer-readable media that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
 receiving a configuration file indicating a first object type and a relationship between the first object type and a second object type; 
 receiving, in response to a query sent to a host computing service using the configuration file, source data comprising a first set of objects of the first object type and a second set of objects of the second object type; 
 sending, to a transformer-based machine-learned model, a set of input data determined based at least in part on the source data, wherein first input data of the set of input data comprises a prompt and data determined from a subset of objects of the source data; 
 receiving, from the transformer-based machine-learned model, a first set of synthetic data associated with the first object type and a second set of synthetic data associated with the second object type; 
 determining, based at least in part on the first set of synthetic data, the second set of synthetic data, and the relationship indicated in the configuration file, a set of synthetic objects; and 
 deploying, at the host computing service, the set of synthetic objects according to the configuration file. 
   
     
     
         2 . The system of  claim 1 , wherein the operations further comprise:
 determining, based at least in part on the source data, a statistical characteristic of the source data; and   determining the prompt based at least in part on including in the prompt at least one of:
 a first rate at which a portion of the first set of objects is formatted according to a first format; 
 a second rate at which the portion of the first set of objects is empty; 
 a set or range of entries indicated in the portion of the first set of objects and a third rate at which the portion of the first set of objects indicates a particular entry in the set or range of entries; or 
 a fourth rate at which first objects of the first set of objects are indicated as related to second objects of the second set of objects, 
   wherein at least one of the first rate, the second rate, the set or range, the third rate, or the fourth rate are part of the statistical characteristic.   
     
     
         3 . The system of  claim 2 , wherein determining the set of synthetic objects further comprises at least one of:
 populating the portion of a first synthetic object with part of the first set of synthetic data according to the statistical characteristic; or   creating a relationship between the first synthetic object and a second synthetic object according to the fourth rate and based at least in part on the relationship indicated by the configuration file.   
     
     
         4 . The system of  claim 1 , wherein the operations further comprise:
 determining, by an encoder model and based at least in part on the source data, a first set of embeddings;   determining, by the encoder model and based at least in part on the first set of synthetic data and the second set of synthetic data, a second set of embeddings; and   determining a similarity metric between the second set of embeddings and the first set of embeddings.   
     
     
         5 . The system of  claim 4 , wherein:
 deploying the set of synthetic objects at the host computing service comprises determining that the similarity metric meets or exceeds a threshold similarity metric for a percentage of the second set of embeddings that meets or exceeds a threshold percentage; or   the operations further comprise:
 determining that the similarity metric is less than a threshold similarity metric for a percentage of the second set of embeddings that is less than the threshold percentage; and 
 replacing a subset of synthetic objects, for which first synthetic metrics were determined to be less than the threshold similarity metric, with a subset of new synthetic objects. 
   
     
     
         6 . The system of  claim 1 , wherein:
 determining the set of input data comprises determining multiple subsets of input data based at least in part on a first number of synthetic objects indicated by a user or a second number of objects in the source data and determining a maximum input size of the transformer-based machine-learned model; and   providing the set of input data comprises separately providing the multiple subsets of input data to different ones of multiple instances of the transformer-based machine-learned model or to different transformer-based machine-learned models.   
     
     
         7 . The system of  claim 1 , wherein the operations further comprise re-generating, by the transformer-based machine-learned model, a portion of first synthetic data of the first set of synthetic data or discarding the first synthetic data based at least in part on determining that the first synthetic data comprises a first entry or pair of entries that matches a second entry or pair of entries of second synthetic data of the first set of synthetic data; and
 wherein the portion is identified by the configuration file for de-duplication across the first set of synthetic data associated with the first object type.   
     
     
         8 . The system of  claim 1 , wherein the operations further comprise creating the configuration file based at least in part on receiving a user indication of a number of synthetic objects to generate and one or more types of objects to generate, and wherein creating the configuration file comprises:
 transmitting, to the host computing service, a request indicating the one or more types of objects;   receiving metadata indicating one or more relationships associated with the first object type and entry types associated with the first object type; and   creating the configuration file based at least in part on the one or more relationships and the entry types.   
     
     
         9 . The system of  claim 1 , wherein deploying the set of synthetic objects according to the configuration file comprises:
 determining, based at least in part on the relationship indicated by the configuration file, that the second object type is dependent on the first object type; and   deploying, at the host computing service, the first set of synthetic objects before deploying the second set of synthetic objects.   
     
     
         10 . One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
 receiving a configuration file indicating a first object type and a relationship between the first object type and a second object type;   receiving source data comprising a first set of objects having the first object type;   sending, to a transformer-based machine-learned model, a set of input data determined based at least in part on the source data, wherein first input data of the set of input data comprises a prompt and a subset of the first set of objects;   receiving, from the transformer-based machine-learned model, a set of synthetic data associated with the first object type;   determining, based at least in part on the set of synthetic data and the relationship indicated in the configuration file, a set of synthetic objects; and   deploying the set of synthetic objects to a host computing service according to the configuration file.   
     
     
         11 . The one or more non-transitory computer-readable media of  claim 10 , wherein the operations further comprise:
 determining, based at least in part on the source data, a statistical characteristic of the source data; and   determining the prompt based at least in part on including in the prompt at least one of:
 a first rate at which a portion of the first set of objects is formatted according to a first format; 
 a second rate at which the portion of the first set of objects is empty; 
 a set or range of entries indicated in the portion of the first set of objects and a third rate at which the portion of the first set of objects indicates a particular entry in the set or range of entries; or 
 a fourth rate at which first objects of the first set of objects are indicated as related to second objects, 
   wherein at least one of the first rate, the second rate, the set or range, the third rate, or the fourth rate are part of the statistical characteristic.   
     
     
         12 . The one or more non-transitory computer-readable media of  claim 11 , wherein determining the set of synthetic objects further comprises at least one of:
 populating the portion of a first synthetic object with part of the set of synthetic data according to the statistical characteristic; or   creating a relationship between the first synthetic object and a second synthetic object according to the fourth rate and based at least in part on the relationship indicated by the configuration file.   
     
     
         13 . The one or more non-transitory computer-readable media of  claim 10 , wherein the operations further comprise:
 determining, by an encoder model and based at least in part on the source data, a first set of embeddings;   determining, by the encoder model and based at least in part on the set of synthetic data, a second set of embeddings; and   determining a similarity metric between the second set of embeddings and the first set of embeddings; and   wherein:   deploying the set of synthetic objects at the host computing service comprises determining that the similarity metric meets or exceeds a threshold similarity metric for a first percentage of the second set of embeddings that meets or exceeds a threshold percentage; or   the operations further comprise:
 determining that the similarity metric is less than the threshold similarity metric for a second percentage of the second set of embeddings that is less than the threshold percentage; and 
 replacing a subset of synthetic objects, for which first synthetic metrics were determined to be less than the threshold similarity metric, with a subset of new synthetic objects. 
   
     
     
         14 . The one or more non-transitory computer-readable media of  claim 10 , wherein:
 determining the set of input data comprises determining multiple subsets of input data based at least in part on a first number of synthetic objects indicated by a user or a second number of objects in the source data and determining a maximum input size of the transformer-based machine-learned model; and   providing the set of input data comprises separately providing the multiple subsets of input data to different ones of multiple instances of the transformer-based machine-learned model or to different transformer-based machine-learned models.   
     
     
         15 . The one or more non-transitory computer-readable media of  claim 10 , wherein the operations further comprise creating the configuration file based at least in part on receiving a user indication of a number of synthetic objects to generate and one or more types of objects to generate, and wherein creating the configuration file comprises:
 transmitting, to the host computing service, a request indicating the one or more types of objects;   receiving metadata indicating one or more relationships associated with the first object type and entry types associated with the first object type; and   creating the configuration file based at least in part on the one or more relationships and the entry types.   
     
     
         16 . The one or more non-transitory computer-readable media of  claim 10 , wherein deploying the set of synthetic objects according to the configuration file comprises:
 determining, based at least in part on the relationship indicated by the configuration file, that the second object type is dependent on the first object type; and   deploying, at the host computing service, the set of synthetic objects before deploying a second set of synthetic objects having the second object type.   
     
     
         17 . A method comprising:
 receiving a configuration file indicating a first object type and a relationship between the first object type and a second object type;   receiving source data comprising a first set of objects having the first object type;   sending, to a transformer-based machine-learned model, a set of input data determined based at least in part on the source data, wherein first input data of the set of input data comprises a prompt and a subset of the first set of objects;   receiving, from the transformer-based machine-learned model, a set of synthetic data associated with the first object type;   determining, based at least in part on the set of synthetic data and the relationship indicated in the configuration file, a set of synthetic objects; and   deploying the set of synthetic objects to a host computing service.   
     
     
         18 . The method of  claim 17 , further comprising:
 determining, based at least in part on the source data, a statistical characteristic of the source data; and   determining the prompt based at least in part on including in the prompt the statistical characteristic and a portion of the source data associated with the statistical characteristic; and at least one of:
 populating the portion of a first synthetic object with part of the set of synthetic data according to the statistical characteristic; or 
 creating a relationship between the first synthetic object and a second synthetic object according to the statistical characteristic and based at least in part on the relationship indicated by the configuration file. 
   
     
     
         19 . The method of  claim 17 , further comprising creating the configuration file based at least in part on receiving a user indication of a number of synthetic objects to generate and one or more types of objects to generate, and wherein creating the configuration file comprises:
 transmitting, to the host computing service, a request indicating the one or more types of objects;   receiving metadata indicating one or more relationships associated with the first object type and entry types associated with the first object type; and   creating the configuration file based at least in part on the one or more relationships and the entry types.   
     
     
         20 . The method of  claim 17 , wherein deploying the set of synthetic objects comprises:
 determining, based at least in part on the relationship indicated by the configuration file, that the second object type is dependent on the first object type; and   deploying, at the host computing service, the set of synthetic objects before deploying a second set of synthetic objects having the second object type.