Performance regression framework

Abstract

A performance regression test that monitors a resource based on a metric measured for each of a plurality of builds of a test instance is run. Measurement data of the metric triggered by an operation is logged in an operation log for each of a plurality of builds. The operation is logged in response to an incoming request for processing associated with the performance regression test for monitoring the resource. Based on the operation log and for each of the plurality of builds, the measurement data of the metric triggered by the operation associated with the performance regression test for monitoring the resource is collected. The collected measurement data of the metric for each of the plurality of builds is presented.

What is claimed is: 
     
         1 . A method comprising:
 installing, by one or more processors of a continuous integration (CI) server, a build of a test instance, wherein the build of the test instance is representative of a version of a portion of a source code updated by a plurality of commits;   receiving, by the one or more processors, a file for a performance regression test that monitors a resource, the file identifying the resource and one or more metrics to monitor associated with accessing the resource;   running, on the build of the test instance and by the one or more processors, the performance regression test based on the file, wherein the build of the test instance is configured to access the resource during the running of the performance regression test;   recording, in an operation log, first measurement data of the one or more metrics triggered by a first operation of a plurality of operations, wherein the first measurement data is logged in response to an incoming request for processing associated with accessing the resource via the build of the test instance, and wherein the operation log comprises second measurement data from running the performance regression test on one or more other builds of the test instance;   extracting, by the one or more processors, the measurement data from the operation log;   detecting, by the one or more processors, a performance regression for the resource in the build of the test instance based on a comparison of the first measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and   generating, by the one or more processors, a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.   
     
     
         2 . The method of  claim 1 , wherein the one or more metrics comprise a processing time, a central processing unit percentage, a response time, a network time, a structured query language (SQL) time, a SQL count, an enterprise rule time, an enterprise rule count, a client response time, a client network time, a total page load time, a number of requests, a browser time, a client script time, or a user interface policy time, or a combination thereof 
     
     
         3 . The method of  claim 1 , comprising:
 installing, by the one or more processors, a plurality of bundled artifacts corresponding to each of the plurality of commits on the version of the source code corresponding to the build of the test instance in response to detecting performance regression for the resource in the build of the test instance;   running, by the one or more processors, the performance regression test on each of the plurality of bundled artifacts to determine a specific bundled artifact responsible for the performance regression in the build of the test instance; and   generating an additional notification indicating the specific bundled artifact or specific commit of the plurality of commits responsible for the performance regression.   
     
     
         4 . The method of  claim 1 , comprising, transmitting the notification to a computing device. 
     
     
         5 . The method of  claim 1 , comprising running an additional performance regression test on the build of the test instance. 
     
     
         6 . The method of  claim 1 , wherein the performance regression test is run on the one or more other builds of the test instance before the performance regression test is run on the build of the test instance. 
     
     
         7 . A system comprising:
 a continuous integration (CI) server device configured to perform actions comprising:
 installing a build of a test instance, wherein the build of the test instance is representative of a version of a portion of a source code updated by a plurality of commits; 
 receiving a file for a performance regression test that monitors a resource, the file identifying the resource and one or more metrics to monitor associated with accessing the resource; 
 running, on the build of the test instance, the performance regression test based on the file, wherein the build of the test instance is configured to access the resource during the running of the performance regression test; and 
 recording, in an operation log, first measurement data of the one or more metrics triggered by a first operation of a plurality of operations, wherein the first measurement data is logged in response to an incoming request for processing associated with accessing the resource via the build of the test instance, wherein the operation log comprises second measurement data from running the performance regression test on one or more other builds of the test instance, and wherein the first measurement data and the second measurement data can be compared to identify a performance regression. 
   
     
     
         8 . The system of  claim 7 , wherein the one or more metrics comprise a processing time, a central processing unit percentage, a response time, a network time, a structured query language (SQL) time, a SQL count, an enterprise rule time, an enterprise rule count, a client response time, a client network time, a total page load time, a number of requests, a browser time, a client script time, or a user interface policy time, or a combination thereof. 
     
     
         9 . The system of  claim 7 , wherein the CI server device is configured to perform actions comprising:
 extracting the first measurement data and the second measurement data from the operation log;   detecting the performance regression for the resource in the build of the test instance based on a comparison of the first measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and   generating a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.   
     
     
         10 . The system of  claim 9 , wherein the CI server device is configured to perform actions comprising:
 transmitting the notification to a computing device.   
     
     
         11 . The system of  claim 10 , wherein the CI server device is configured to perform actions comprising:
 installing a plurality of bundled artifacts corresponding to each of the plurality of commits on the version of the source code corresponding to the build of the test instance in response to detecting performance regression for the resource in the build of the test instance;   running the performance regression test on each of the plurality of bundled artifacts to determine a specific bundled artifact of the plurality of bundled artifacts responsible for the performance regression in the build of the test instance; and   generating a notification indicating the specific bundled artifact or specific commit of the plurality of commits responsible for the performance regression.   
     
     
         12 . The system of  claim 7 , comprising an additional server device, wherein the additional server device is configured to perform actions comprising:
 extracting the first measurement data and the second measurement data from the operation log;   detecting the performance regression for the resource in the build of the test instance based on a comparison of the first measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and   generating a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.   
     
     
         13 . The system of  claim 12 , wherein the additional server device comprises a remote client device. 
     
     
         14 . The system of  claim 13 , wherein generating the notification comprises executing logic to visualize, on the remote client device, one or more widgets, wherein the one or more widgets comprise a graphical visualization of measurement data of the resource for a plurality of builds including the regressed build and the one or more other builds of the test instance. 
     
     
         15 . One or more non-transitory, tangible, and computer-readable media, storing instructions that, when executed by one or more processors of a continuous integration (CI) server, cause the CI server to perform operations comprising:
 installing a build of a test instance, wherein the build of the test instance is representative of a version of a portion of a source code updated by a plurality of commits;   receiving a file for a performance regression test that monitors a resource, the file identifying the resource and one or more metrics to monitor associated with accessing the resource;   running, on the build of the test instance, the performance regression test based on the file, wherein the build of the test instance is configured to access the resource during the running of the performance regression test;   recording, in the operation log, first measurement data of the one or more metrics triggered by a first operation of a plurality of operations, wherein the first measurement data is logged in response to an incoming request for processing associated with accessing the resource via the build of the test instance, and wherein the operation log comprises second measurement data from running the performance regression test on one or more other builds of the test instance;   extracting the measurement data from the operation log;   detecting a performance regression for the resource in the build of the test instance based on a comparison of the measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and   generating, by the one or more processors, a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.   
     
     
         16 . The one or more non-transitory, tangible, and computer-readable media of  claim 15 , wherein the one or more metrics comprise a processing time, a central processing unit percentage, a response time, a network time, a structured query language (SQL) time, a SQL count, an enterprise rule time, an enterprise rule count, a client response time, a client network time, a total page load time, a number of requests, a browser time, a client script time, or a user interface policy time, or a combination thereof. 
     
     
         17 . The one or more non-transitory, tangible, and computer-readable media of  claim 15 , wherein the operations comprise:
 installing a plurality of bundled artifacts corresponding to each of the plurality of commits on the version of the source code corresponding to the build of the test instance in response to detecting regression for the resource in the build of the test instance;   running the performance regression test on each of the plurality of bundled artifacts to determine a specific bundled artifact of the plurality of bundled artifacts responsible for the performance regression in the build of the test instance; and   generating an additional notification indicating the specific bundled artifact or specific commit of the plurality of commits responsible for the performance regression.   
     
     
         18 . The one or more non-transitory, tangible, and computer-readable media of  claim 15 , wherein the performance regression test is run on the one or more other builds of the test instance before the performance regression test is run on the build of the test instance. 
     
     
         19 . The one or more non-transitory, tangible, and computer-readable media of  claim 15 , wherein the operations comprise transmitting the notification to a remote client device. 
     
     
         20 . The one or more non-transitory, tangible, and computer-readable media of  claim 15 , wherein running the performance regression test on the build of the test instance and the one or more other builds of the test instance comprises loading the resource for a predetermined number of iterations.