Systems and methods for automatically testing changes to a software application using machine learning

Abstract

Systems and methods for automatically testing changes to a software application using machine learning are disclosed. In some embodiments, a disclosed method includes: obtaining, from a computing device, a request for a proposed change to an application; generating at least one baseline instance running an existing version of the application before the proposed change; generating a candidate instance running a new version of the application based on the proposed change; performing an analysis on the at least one baseline instance and the candidate instance to determine, using at least one machine learning model, whether an anomaly exists in the candidate instance; generating a report for the proposed change based on the analysis; and transmitting the report to the computing device.

What is claimed is: 
     
         1 . A system, comprising:
 a non-transitory memory having instructions stored thereon; and   at least one processor operatively coupled to the non-transitory memory, and configured to read the instructions to:
 obtain, from a computing device, a request for a proposed change to an application, 
 generate at least one baseline instance running an existing version of the application before the proposed change, 
 generate a candidate instance running a new version of the application based on the proposed change, 
 perform an analysis on the at least one baseline instance and the candidate instance to determine, using at least one machine learning model, whether an anomaly exists in the candidate instance, 
 generate a report for the proposed change based on the analysis, and 
 transmit the report to the computing device. 
   
     
     
         2 . The system of  claim 1 , wherein:
 the application is at least one of: a software application associated with an individual or an entity, a software application running on a data service platform, or a software application running on an online platform; and   the request is triggered by at least one of: a pull request initiated by an engineer working on the application, or a detection of the proposed change by monitoring the application.   
     
     
         3 . The system of  claim 1 , wherein the at least one processor is configured to:
 create an isolated environment for the application that is running in a production environment, by identifying and isolating traffic flows that are capable of impacting users of the application,   wherein the analysis is performed by executing the at least one baseline instance and the candidate instance in the isolated environment.   
     
     
         4 . The system of  claim 3 , wherein the analysis is performed based on:
 analyzing production traffic in the production environment;   generating mirrored traffic in the isolated environment based on the production traffic in the production environment;   selecting a subset of traffic from the mirrored traffic;   replaying the subset of traffic in the isolated environment against both the at least one baseline instance and the candidate instance; and   determining whether an anomaly exists in the candidate instance based on the replaying.   
     
     
         5 . The system of  claim 4 , wherein determining whether an anomaly exists in the candidate instance comprises at least one of:
 identifying any logical error;   detecting any performance degradation;   monitoring any deviation in a call pattern of downstream dependencies;   identifying any change in contracts with downstream or upstream services; or   scrutinizing any logging pattern discrepancy between the at least one baseline instance and the candidate instance.   
     
     
         6 . The system of  claim 4 , wherein selecting the subset of traffic comprises:
 determining a plurality of sessions in the mirrored traffic, wherein:
 each session includes a sequence of requests, each of which calls to a functional endpoint of the application, 
 the requests in the plurality of sessions belong to a plurality of request types; and 
   sampling the requests in the plurality of sessions to select the subset of traffic subject to a minimum number of request samples per request type.   
     
     
         7 . The system of  claim 6 , wherein determining whether an anomaly exists comprises:
 sending a same set of sampled requests to the at least one baseline instance and the candidate instance;   receiving responses from the at least one baseline instance and the candidate instance;   comparing the responses from the at least one baseline instance and the candidate instance to determine at least one difference in the responses; and   determining whether the at least one difference represents an anomaly due to the proposed change.   
     
     
         8 . The system of  claim 6 , wherein:
 the at least one baseline instance comprises two baseline instances; and   determining whether an anomaly exists comprises:
 sending a same set of sampled requests to the two baseline instances and the candidate instance; 
 receiving responses from the two baseline instances and the candidate instance; 
 comparing the responses from the two baseline instances to identify noise; 
 filtering out the noise from all responses of the two baseline instances and the candidate instance to generate filtered responses; 
 comparing the filtered responses of the two baseline instances and the candidate instance to determine at least one difference in the filtered responses; and 
 determining whether the at least one difference represents an anomaly due to the proposed change. 
   
     
     
         9 . The system of  claim 4 , wherein the at least one processor is configured to:
 in accordance with a determination that an anomaly exists in the candidate instance, determine whether the anomaly is temporary or persistent based on one or more retries of replaying the subset of traffic in the isolated environment against both the at least one baseline instance and the candidate instance.   
     
     
         10 . The system of  claim 9 , wherein the at least one processor is further configured to:
 in accordance with a determination that the anomaly is persistent, determine insight data indicating one or more factors causing the anomaly, wherein the report includes: the anomaly, the insight data, and a summary of the analysis.   
     
     
         11 . The system of  claim 3 , wherein:
 the at least one machine learning model includes a reinforcement learning (RL) model running in the isolated environment;   a system state of the RL model represents a snapshot of the production environment, incorporating data from multiple sources;   based on a current system state of the RL model, an agent of the RL model takes one or more of the following actions:
 modifying a speed of processing data streams to stabilize system throughput, 
 implementing or adjusting a retry logic for failed operations, 
 sampling and replaying requests to exploit service behavioral anomalies in the application; 
   a reward for the agent is computed based on a difference in anomaly severity from a last system state to the current system state following an action taken by the agent;   the agent of the RL model is continuously trained based on updated data to adjust its strategies within the isolated environment that mirrors the production environment.   
     
     
         12 . The system of  claim 1 , wherein the at least one processor is configured to:
 collect raw data from a plurality of sources associated with the application;   process the raw data based on timestamp synchronization, null value handling and text normalization, to generate normalized raw data;   identify and construct features from the normalized raw data based on dimensionality reduction, cluster analysis, time series analysis, and anomaly detection, using a first machine learning model;   rank the features using a second machine learning model based on their predictive importance scores regarding application health;   select a subset of features having highest predictive importance scores;   construct health signals each corresponding to a service in the application using a third machine learning model based on the subset of features, wherein the third machine learning model is a supervised learning model trained based on historical data with application states labeled as healthy or unhealthy; and   monitor a health of the application based on the health signals.   
     
     
         13 . The system of  claim 1 , wherein the at least one processor is configured to:
 in accordance with a determination that the proposed change is approved to be deployed into the application, re-perform the analysis, using the at least one machine learning model, on the at least one baseline instance and the candidate instance during a deployment flow involving the proposed change and at least one additional approved change, before deploying the proposed change and the at least one additional approved change into the application.   
     
     
         14 . A computer-implemented method, comprising:
 obtaining, from a computing device, a request for a proposed change to an application;   generating at least one baseline instance running an existing version of the application before the proposed change;   generating a candidate instance running a new version of the application based on the proposed change;   performing an analysis on the at least one baseline instance and the candidate instance to determine, using at least one machine learning model, whether an anomaly exists in the candidate instance;   generating a report for the proposed change based on the analysis; and   transmitting the report to the computing device.   
     
     
         15 . The computer-implemented method of  claim 14 , further comprising:
 creating an isolated environment for the application that is running in a production environment, by identifying and isolating traffic flows that are capable of impacting users of the application,   wherein the analysis is performed by executing the at least one baseline instance and the candidate instance in the isolated environment.   
     
     
         16 . The computer-implemented method of  claim 15 , wherein performing the analysis comprises:
 analyzing production traffic in the production environment;   generating mirrored traffic in the isolated environment based on the production traffic in the production environment;   selecting a subset of traffic from the mirrored traffic;   replaying the subset of traffic in the isolated environment against both the at least one baseline instance and the candidate instance; and   determining whether an anomaly exists in the candidate instance based on the replaying.   
     
     
         17 . The computer-implemented method of  claim 16 , wherein selecting the subset of traffic comprises:
 determining a plurality of sessions in the mirrored traffic, wherein:
 each session includes a sequence of requests, each of which calls to a functional endpoint of the application, 
 the requests in the plurality of sessions belong to a plurality of request types; and 
   sampling the requests in the plurality of sessions to select the subset of traffic subject to a minimum number of request samples per request type.   
     
     
         18 . The computer-implemented method of  claim 17 , wherein determining whether an anomaly exists comprises:
 sending a same set of sampled requests to the at least one baseline instance and the candidate instance;   receiving responses from the at least one baseline instance and the candidate instance;   comparing the responses from the at least one baseline instance and the candidate instance to determine at least one difference in the responses; and   determining whether the at least one difference represents an anomaly due to the proposed change.   
     
     
         19 . The computer-implemented method of  claim 17 , wherein:
 the at least one baseline instance comprises two baseline instances; and   determining whether an anomaly exists comprises:
 sending a same set of sampled requests to the two baseline instances and the candidate instance, 
 receiving responses from the two baseline instances and the candidate instance, 
 comparing the responses from the two baseline instances to identify noise, 
 filtering out the noise from all responses of the two baseline instances and the candidate instance to generate filtered responses, 
 comparing the filtered responses of the two baseline instances and the candidate instance to determine at least one difference in the filtered responses, and 
 determining whether the at least one difference represents an anomaly due to the proposed change. 
   
     
     
         20 . A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause at least one device to perform operations comprising:
 obtaining, from a computing device, a request for a proposed change to an application;   generating at least one baseline instance running an existing version of the application before the proposed change;   generating a candidate instance running a new version of the application based on the proposed change;   performing an analysis on the at least one baseline instance and the candidate instance to determine, using at least one machine learning model, whether an anomaly exists in the candidate instance;   generating a report for the proposed change based on the analysis; and   transmitting the report to the computing device.