Machine Learning Model Evaluation

Abstract

Testing machine learning sensors by adding obfuscated training data to test data, and performing real time model fit analysis on live network traffic to determine whether to retrain.

What is claimed is: 
     
         1 . A process for improving a machine learning sensor for defending a computer network, comprising:
 obtaining samples of normal network traffic from said network;   providing samples of cyber attacks from said network or from a repository of cyber attacks, wherein said samples of cyber attacks constitute known attacks;   sample classifying each of said samples as either said normal network traffic or said known attacks, to create ground truths for said samples;   splitting said samples into a training set and a test set, with each of said sets containing samples of said normal network traffic and of said known attacks;   using a model generating algorithm to generate a variety of models for distinguishing between said normal network traffic and said known attacks in said training set;   obfuscating a portion of samples of said known attacks in said training set to create obfuscated attack samples;   adding said obfuscated attack samples to said test set to form an enhanced test set;   performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;   selecting one of said models that optimizes a desired model parameter as an optimal model for attack classifying between said normal network traffic and classified cyber attacks consisting of said known attacks, said obfuscated attack samples, and unknown attacks that are unrelated to said training set;   installing said optimal model in said sensor;   determining model fit using live data from real-time network traffic on said computer network, to determine more accurately whether and when to react to intrusion detection alert logs.   
     
     
         2 . A process according to  claim 1 , further comprising:
 determining model fit of said sensor by applying anomaly detection, measuring similarity between live network traffic and said training set, and determining model overfit;   assigning thresholds for said anomaly detection, said measuring similarity between live network traffic and said training set, and said determining model overfit, and aggregating results of whether said sensor is above or below said thresholds to identify model fit in real time on a scale of model fits; and   activating model retraining based on said scale of model fits.   
     
     
         3 . A process according to  claim 1 , wherein said selecting step is performed using a desired model parameter selected from the group consisting of intrusion detection capability error and test error. 
     
     
         4 . A system, comprising:
 a network of computers; and   a machine-learning based sensor deployed in said network, wherein said machine-learning based sensor is trained by a process comprising:   obtaining samples of normal network traffic from said network;   providing samples of cyber attacks from said network or from a repository of cyber attacks, wherein said samples of cyber attacks constitute known attacks;   sample classifying each of said samples as either said normal network traffic or said known attacks, to create ground truths for said samples;   splitting said samples into a training set and a test set, with each of said sets containing samples of said normal network traffic and of said known attacks;   using a model generating algorithm to generate a variety of models for distinguishing between said normal network traffic and said known attacks in said training set;   obfuscating a portion of samples of said known attacks in said training set to create obfuscated attack samples;   adding said obfuscated attack samples to said test set to form an enhanced test set;   performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;   selecting one of said models that optimizes a desired model parameter as an optimal model for attack classifying between said normal network traffic and classified cyber attacks consisting of said known attacks, said obfuscated attack samples, and unknown attacks that are unrelated to said training set;   installing said optimal model in said sensor;   determining model fit using live data from real-time network traffic on said computer network, to determine more accurately whether and when to react to intrusion detection alert logs.   
     
     
         5 . A system according to  claim 4 , wherein said selecting step is performed using a desired model parameter selected from the group consisting of intrusion detection capability error and test error. 
     
     
         6 . A computer program product stored in a computer readable medium for protecting a computer network, comprising:
 a machine learning based sensor that is trained by a process comprising:   obtaining samples of normal network traffic from said network;   providing samples of cyber attacks from said network or from a repository of cyber attacks, wherein said samples of cyber attacks constitute known attacks;   sample classifying each of said samples as either said normal network traffic or said known attacks, to create ground truths for said samples;   splitting said samples into a training set and a test set, with each of said sets containing samples of said normal network traffic and of said known attacks;   using a model generating algorithm to generate a variety of models for distinguishing between said normal network traffic and said known attacks in said training set;   obfuscating a portion of samples of said known attacks in said training set to create obfuscated attack samples;   adding said obfuscated attack samples to said test set to form an enhanced test set;   performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;   selecting one of said models that optimizes a desired model parameter as an optimal model for attack classifying between said normal network traffic and classified cyber attacks consisting of said known attacks, said obfuscated attack samples, and unknown attacks that are unrelated to said training set;   installing said optimal model in said sensor; and   determining model fit using live data from real-time network traffic on said computer network, to determine more accurately whether and when to react to intrusion detection alert logs.   
     
     
         7 . A computer program product according to  claim 6 , wherein said selecting step is performed using a desired model parameter selected from the group consisting of intrusion detection capability error and test error. 
     
     
         8 . A process for improving a machine learning sensor, comprising:
 obtaining samples of normal data from a set of data containing normal and abnormal data;   providing samples of abnormal data from said set or from a repository of abnormal data, wherein said samples of abnormal data constitute known abnormal data;   sample classifying each of said samples as either said normal data or said known abnormal data, to create ground truths for said samples;   splitting said samples into a training set and a test set, with each of said sets containing samples of said normal data and of said known abnormal data;   using a model generating algorithm to generate a variety of models for distinguishing between said normal data and said known abnormal data in said training set;   obfuscating a portion of samples of said known abnormal data in said training set to create obfuscated abnormal data samples;   adding said obfuscated abnormal data samples to said test set to form an enhanced test set;   performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;   selecting one of said models that optimizes a desired model parameter as an optimal model for abnormal classifying between said normal data and classified abnormal data consisting of said known abnormal data, said obfuscated abnormal data samples, and unknown abnormal data that is unrelated to said training set;   determining model fit using live data from real-time network traffic on said computer network;   installing said optimal model in said sensor;   wherein said set of data containing normal and abnormal data is selected from the group consisting of network traffic, DNA sequences, a composer's, author's or artist's collected works, and financial transactions; and   wherein said sensor is used for a function selected from the group consisting of detecting cyber attacks in said network traffic, detecting obfuscated DNA sequences in said DNA sequences, detecting a falsely attributed work from among said collected works, or detecting obfuscated transactions in said financial transactions.   
     
     
         9 . A system, comprising:
 a computer; and   a machine-learning based sensor deployed in said computer, wherein said machine-learning based sensor has been trained by a process comprising:   obtaining samples of normal data from a set of data containing normal and abnormal data;   providing samples of abnormal data from said set or from a repository of abnormal data, wherein said samples of abnormal data constitute known abnormal data;   sample classifying each of said samples as either said normal data or said known abnormal data, to create ground truths for said samples;   splitting said samples into a training set and a test set, with each of said sets containing samples of said normal data and of said known abnormal data;   using a model generating algorithm to generate a variety of models for distinguishing between said normal data and said known abnormal data in said training set;   obfuscating a portion of samples of said known abnormal data in said training set to create obfuscated abnormal data samples;   adding said obfuscated abnormal data samples to said test set to form an enhanced test set;   performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;   selecting one of said models that optimizes a desired model parameter as an optimal model for abnormal classifying between said normal data and classified abnormal data consisting of said known abnormal data, said obfuscated abnormal data samples, and unknown abnormal data that is unrelated to said training set;   installing said optimal model in said computer;   wherein said set of data containing normal and abnormal data is selected from the group consisting of network traffic, DNA sequences, a composer's, author's or artist's collected works, and financial transactions; and   wherein said sensor is used for a function selected from the group consisting of detecting cyber attacks in said network traffic, detecting obfuscated DNA sequences in said DNA sequences, detecting a falsely attributed work from among said collected works, or detecting obfuscated transactions in said financial transactions.   
     
     
         10 . A computer program product stored in a computer readable medium that, when executed by a computer, provides said computer with a machine learning based sensor that has been trained by a process comprising:
 obtaining samples of normal data from a set of data containing normal and abnormal data;   providing samples of abnormal data from said set or from a repository of abnormal data, wherein said samples of abnormal data constitute known abnormal data;   sample classifying each of said samples as either said normal data or said abnormal data, to create ground truths for said samples;   splitting said samples into a training set and a test set, with each of said sets containing samples of said normal data and of said known abnormal data;   using a model generating algorithm to generate a variety of models for distinguishing between said normal data and said known abnormal data in said training set;   obfuscating a portion of samples of said known abnormal data in said training set to create obfuscated abnormal data samples;   adding said obfuscated abnormal data samples to said test set to form an enhanced test set;   performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;   selecting one of said models that optimizes a desired model parameter as an optimal model for distinguishing between said normal data and classified abnormal data consisting of said known abnormal data, said obfuscated abnormal data samples, and unknown abnormal data that is unrelated to said training set;   wherein said set of data containing normal and abnormal data is selected from the group consisting of network traffic, DNA sequences, a composer's, author's or artist's collected works, and financial transactions; and   wherein said sensor is used for a function selected from the group consisting of detecting cyber attacks in said network traffic, detecting obfuscated DNA sequences in said DNA sequences, detecting a falsely attributed work from among said collected works, or detecting obfuscated transactions in said financial transactions.   
     
     
         11 . A process according to  claim 8 , wherein said selecting step is performed using a desired model parameter selected from the group consisting of intrusion detection capability error and test error. 
     
     
         12 . A system according to  claim 9 , wherein said selecting step is performed using a desired model parameter selected from the group consisting of intrusion detection capability error and test error. 
     
     
         13 . A computer program product according to  claim 10 , wherein said selecting step is performed using a desired model parameter selected from the group consisting of intrusion detection capability error and test error.