Automated detection of code regressions from time-series data

Abstract

In non-limiting examples of the present disclosure, systems, methods, and devices for detecting and classifying service issues associated with a cloud-based service are presented. Operational event data for a plurality of operations associated with the cloud-based application service may be monitored. A statistical-based unsupervised machine learning model may be applied to the operational event data. A subset of the operational event data may be tagged as potentially being associated with a code regression, wherein the subset comprises a time series of operational event data. A neural network may be applied to the time series of operational event data, and the time series of operational event data may be flagged for follow-up if the neural network classifies the time series as relating to a positive code regression category.

What is claimed is: 
     
         1 . A method for detecting code regressions associated with a cloud-based application service, the method comprising:
 monitoring operational event data for a plurality of operations associated with the cloud-based application service to generate a time series of the operational event data;   generating an input function from the time series of the operational event data comprising a function of one or more of a number of successes in the time series, a number of unexpected failures in the time series, and a number of expected failures in the time series;   applying a statistical-based unsupervised machine learning model to the input function, wherein the statistical-based unsupervised machine learning model is trained to identify datapoints that fall outside of a normalized pattern for a relevant operation of the plurality of operations in the time series;   providing the identified datapoints to a neural network trained to classify the operations associated with the identified datapoints into one of a first category indicating the associated operation is related to a code regression and a second category indicating the associated operation is not related to a code regression; and   flagging one or more of the operations associated with the identified datapoints for follow-up based on a classification into the first category.   
     
     
         2 . The method of  claim 1 , wherein the neural network is a convolutional neural network, and wherein the convolutional neural network is applied to a graphical image representation of the time series of the operational event data. 
     
     
         3 . The method of  claim 2 , wherein applying the neural network to the graphical image representation of the time series of the operational event data comprises classifying the time series of the operational event data into one of a plurality of categories comprising: a positive code regression category; a negative code regression category; and a possible code regression category. 
     
     
         4 . The method of  claim 3 , further comprising:
 ignoring the time series of the operational event data based on the neural network classifying the time series in the negative code regression category.   
     
     
         5 . The method of  claim 1 , wherein the statistical-based unsupervised machine learning model is one of: a Gaussian model; a boxplot method model; a one class support vector machine model; and a density-based spatial clustering of applications with noise model. 
     
     
         6 . The method of  claim 1 , further comprising:
 identifying a plurality of error logs corresponding to the flagged one or more of the operations;   applying a language processing model to each of the error logs; and   classifying, based on the applying the language processing model, each of the flagged one or more of the operations into a severity category.   
     
     
         7 . The method of  claim 6 , wherein the classifying each of the flagged one or more of the operations is based on one or more of: a type of operational event; a number of users that are impacted by the code regression; and a number of sites that are currently still experiencing errors associated with the code regression. 
     
     
         8 . The method of  claim 6 , wherein the language processing model is one of: a recurrent neural network; a convolutional neural network; a bag-of-words model; a rule-based model; a random forest model; and a word segmentation model. 
     
     
         9 . The method of  claim 1 , further comprising:
 applying a language processing model to the flagged one or more of the operations to generate a cause of the flagging; and   providing the flagged one or more of the operations and the cause of the flagging for the follow-up.   
     
     
         10 . A system, comprising:
 one or more processors; and   a memory having stored thereon instructions that, upon execution by the one or more processors, cause the one or more processors to:
 monitor operational event data for a plurality of operations associated with a cloud-based application service to generate a time series of the operational event data; 
 generate an input function from the time series of the operational event data comprising a function of one or more of a number of successes in the time series, a number of unexpected failures in the time series, and a number of expected failures in the time series; 
 apply a statistical-based unsupervised machine learning model to the input function, wherein the statistical-based unsupervised machine learning model is trained to identify datapoints that fall outside of a normalized pattern for a relevant operation of the plurality of operations in the time series; 
 provide the identified datapoints to a neural network trained to classify the operations associated with the identified datapoints into one of a first category indicating the associated operation is related to a code regression and a second category indicating the associated operation is not related to a code regression; and 
 flag one or more of the operations associated with the identified datapoints for follow-up based on a classification into the first category. 
   
     
     
         11 . The system of  claim 10 , wherein the neural network is a convolutional neural network, wherein the convolutional neural network is applied to a graphical image representation of the time series of the operational event data, and wherein the instructions to apply the neural network to the graphical image representation of the time series of the operational event data comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 classify the time series of the operational event data into one of a plurality of categories comprising: a positive code regression category, a negative code regression category, and a possible code regression category; and   ignore the time series of the operational event data based on the neural network classifying the time series in the negative code regression category.   
     
     
         12 . The system of  claim 10 , wherein the statistical-based unsupervised machine learning model is one of: a Gaussian model; a boxplot method model; a one class support vector machine model; and a density-based spatial clustering of applications with noise model. 
     
     
         13 . The system of  claim 10 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 identify a plurality of error logs corresponding to the flagged one or more of the operations;   apply a language processing model to each of the error logs, wherein the language processing model is one of a recurrent neural network, a convolutional neural network, a bag-of-words model, a rule-based model, a random forest model, and a word segmentation model; and   classify each of the flagged one or more of the operations into a severity category based on the applying the language processing model and based on one or more of a type of operational event, a number of users that are impacted by the code regression, and a number of sites that are currently still experiencing errors associated with the code regression.   
     
     
         14 . The system of  claim 10 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 apply a language processing model to the flagged one or more of the operations to generate a cause of the flagging; and   provide the flagged one or more of the operations and the cause of the flagging for the follow-up.   
     
     
         15 . A computer-readable storage device comprising executable instructions that, when executed by one or more processors, cause the one or more processors to:
 monitor operational event data for a plurality of operations associated with a cloud-based application service to generate a time series of the operational event data;   generate an input function from the time series of the operational event data comprising a function of one or more of a number of successes in the time series, a number of unexpected failures in the time series, and a number of expected failures in the time series;   apply a statistical-based unsupervised machine learning model to the input function, wherein the statistical-based unsupervised machine learning model is trained to identify datapoints that fall outside of a normalized pattern for a relevant operation of the plurality of operations in the time series;   provide the identified datapoints to a neural network trained to classify the operations associated with the identified datapoints into one of a first category indicating the associated operation is related to a code regression and a second category indicating the associated operation is not related to a code regression; and   flag one or more of the operations associated with the identified datapoints for follow-up based on a classification into the first category.   
     
     
         16 . The computer-readable storage device of  claim 15 , wherein the neural network is a convolutional neural network, and wherein the convolutional neural network is applied to a graphical image representation of the time series of the operational event data. 
     
     
         17 . The computer-readable storage device of  claim 16 , wherein the instructions to apply the neural network to the graphical image representation of the time series of operational event data comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 classify the time series of the operational event data into one of a plurality of categories comprising: a positive code regression category, a negative code regression category, and a possible code regression category.   
     
     
         18 . The computer-readable storage device of  claim 17 , wherein the instructions to apply the neural network to the graphical image representation of the time series of operational event data comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 ignoring the time series of operational event data if the neural network classifies the time series in the negative code regression category.   
     
     
         19 . The computer-readable storage device of  claim 15 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 identify a plurality of error logs corresponding to the flagged one or more of the operations;   apply a language processing model to each of the error logs, wherein the language processing model is one of a recurrent neural network, a convolutional neural network, a bag-of-words model, a rule-based model, a random forest model, and a word segmentation model; and   classify each of the flagged one or more of the operations into a severity category based on the applying the language processing model and based on one or more of a type of operational event, a number of users that are impacted by the code regression, and a number of sites that are currently still experiencing errors associated with the code regression.   
     
     
         20 . The computer-readable storage device of  claim 15 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
 apply a language processing model to the flagged one or more of the operations to generate a cause of the flagging; and   provide the flagged one or more of the operations and the cause of the flagging for the follow-up.