Feature engineering for web-based anomaly detection

Abstract

The present disclosure is directed towards systems and methods for detecting anomalous network traffic. Network traffic corresponding to an application executed by a server can be received. Application characteristics of the application can be identified to select an anomaly detection profile. The anomaly detection profile can be selected based on the identified application characteristics. The anomaly detection profile can include a set of detection features for the anomaly and one or more predetermined threshold values of the detection features. One or more feature values of the set of one or more detection features can be determined. An anomaly in the network traffic can be detected responsive to comparing the feature values and the predetermined threshold values of the detection features.

What is claimed is: 
     
         1 . A method comprising:
 establishing, by a device, a plurality of anomaly detection profiles, each of the plurality of anomaly detection profiles identifying an anomaly and one or more detection features for the anomaly;   selecting, by the device, an anomaly detection profile for an application from the plurality of anomaly detection profiles based at least on application characteristics of an application identified using network traffic of the application traversing the device;   setting, by the device for the anomaly detection profile, one or more values for each of the one or more detection features based at least on a range of values identified via the network traffic of the application that is non-anomalous;   detecting, by the device, an anomaly in the network traffic of the application responsive to comparing values of one or more detection features identified in the network traffic of the application to one or more thresholds; and   blocking, by the device responsive to detecting the anomaly, at least a portion of the network traffic of the application.   
     
     
         2 . The method of  claim 1 , further comprising communicating, by the device responsive to detecting the anomaly, an alert regarding the detected anomaly. 
     
     
         3 . The method of  claim 1 , wherein each of the plurality of anomaly detection profiles identifies the one or more thresholds to use for comparing the values of the one or more detection features. 
     
     
         4 . The method of  claim 1 , wherein the one or more thresholds are specific to the application. 
     
     
         5 . The method of  claim 1 , further comprising establishing values for the one or more thresholds by using an anomaly detection model. 
     
     
         6 . The method of  claim 5 , further comprising monitoring, by the anomaly detection model, network traffic corresponding to the application to establish the values for the one or more thresholds. 
     
     
         7 . The method of  claim 1 , further comprising identifying the application characteristics that correspond to the one or more detection features used to detect at least one of a denial of service attack, web scraping, a brute force attempt at determining login credentials associated with the application or anomalous packet payloads. 
     
     
         8 . The method of  claim 1 , further comprising identifying the application characteristics from a log of network traffic received by the device. 
     
     
         9 . The method of  claim 1 , further comprising monitoring, by the device, network traffic of the application to identify the values of the one or more detection features. 
     
     
         10 . The method of  claim 1 , wherein the device is intermediary to a plurality of clients and the application. 
     
     
         11 . A system comprising:
 a device comprising one or more processors, coupled to memory and configured to:   establish a plurality of anomaly detection profiles, each of the plurality of anomaly detection profiles identifying an anomaly and one or more detection features for the anomaly;   select an anomaly detection profile for an application from the plurality of anomaly detection profiles based at least on application characteristics of an application identified using network traffic of the application traversing the device;   set, for the anomaly detection profile, one or more values for each of the one or more detection features based at least on a range of values identified via the network traffic of the application that is non-anomalous;   detect an anomaly in the network traffic of the application responsive to comparing values of one or more detection features identified in the network traffic of the application to one or more thresholds; and   block, responsive to detecting the anomaly, at least a portion of the network traffic of the application.   
     
     
         12 . The system of  claim 11 , wherein the device is further configured to communicate, responsive to detecting the anomaly, an alert regarding the detected anomaly. 
     
     
         13 . The system of  claim 11 , wherein each of the plurality of anomaly detection profiles identifies the one or more thresholds to use for comparing the values of the one or more detection features. 
     
     
         14 . The system of  claim 11 , wherein the one or more thresholds are specific to the application. 
     
     
         15 . The system of  claim 11 , wherein the device is further configured to use an anomaly detection model to establish values for the one or more thresholds. 
     
     
         16 . The system of  claim 15 , wherein the the anomaly detection model is further configured to monitor network traffic corresponding to the application to establish the values for the one or more thresholds. 
     
     
         17 . The system of  claim 11 , wherein the device is further configured to identify the application characteristics that correspond to the one or more detection features used to detect at least one of a denial of service attack, web scraping, a brute force attempt at determining login credentials associated with the application or anomalous packet payloads. 
     
     
         18 . The system of  claim 11 , wherein the device is further configured to identify the application characteristics from a log of network traffic received by the device. 
     
     
         19 . The system of  claim 11 , wherein the device is further configured to monitor network traffic of the application to identify the values of the one or more detection features. 
     
     
         20 . The system of  claim 11 , wherein the device is intermediary to a plurality of clients and the application.