Method and system for detecting malicious payloads

Abstract

Disclosed is an improved method, system, and computer program product for identifying malicious payloads. The disclosed approach identifies potentially malicious payload exchanges which may be associated with payload injection or root-kit magic key usage.

What is claimed is: 
     
         1 . A method for detecting a malicious network payload, comprising:
 collecting network traffic corresponding to communications within a network;   extracting metadata from the network traffic for a client-server session;   in a learning phase, identifying from the metadata any representative sequences for payloads sent between a client and a server for the client-server session; and   in a monitoring phase, detecting for any client-server sequences corresponding to a payload that are different from the representative sequences identified in the learning phase.   
     
     
         2 . The method of  claim 1 , wherein the representative sequences are identified on the basis of a unique combination of the server and a port, where baseline payload characteristics are identified for the unique combination of the server and the port. 
     
     
         3 . The method of  claim 2 , further comprising:
 receiving a first communication for the unique combination of the server and the port;   setting the baseline payload characteristics in correspondence to the first communications;   for an additional communication, comparing the additional communication against the baseline payload characteristics to obtain a similarity score, and identifying new baseline payload characteristics that corresponds to the additional communication if the similarity score is not within a threshold distance and a minimum number of sessions or time has elapsed.   
     
     
         4 . The method of  claim 1 , wherein identification of the representative sequences in the learning phase is performed by constructing one or more models corresponding to baseline behavior for the client and the server. 
     
     
         5 . The method of  claim 4 , wherein new normal client-server sequences are identified even after an initial learning phase by updating the one or more model. 
     
     
         6 . The method of  claim 1 , wherein the monitoring phase is implemented by:
 receiving a new communication for analysis;   comparing an extracted sequence for the new communication against the representative sequences identified in the learning phase to generate a score;   determining if the score exceeds a threshold; and   identifying the new communication as a potential threat for malicious payload if the score exceeds the threshold.   
     
     
         7 . The method of  claim 1 , wherein the metadata extracted from the network traffic comprises some or more of IP address information for the client and the server, payload information, quantity information for transferred data, duration of communications, or length of time delay between a client request and a server response. 
     
     
         8 . A system for detecting a malicious network payload, comprising:
 a processor;   a memory for holding programmable code; and   wherein the programmable code includes instructions collecting network traffic corresponding to communications within a network; extracting metadata from the network traffic for a client-server session; in a learning phase, identifying from the metadata any representative sequences for payloads sent between a client and a server for the client-server session; and in a monitoring phase, detecting for any client-server sequences corresponding to a payload that are different from the representative sequences identified in the learning phase.   
     
     
         9 . The system of  claim 7 , wherein the representative sequences are identified on the basis of a unique combination of the server and a port, where baseline payload characteristics are identified for the unique combination of the server and the port. 
     
     
         10 . The system of  claim 9 , wherein the programmable code further includes instructions for:
 receiving a first communication for the unique combination of the server and the port;   setting the baseline payload characteristics in correspondence to the first communications;   for an additional communication, comparing the additional communication against the baseline payload characteristics to obtain a similarity score, and identifying new baseline payload characteristics that corresponds to the additional communication if the similarity score is not within a threshold distance and a minimum number of sessions or time has elapsed.   
     
     
         11 . The system of  claim 7 , wherein identification of the representative sequences in the learning phase is performed by constructing one or more models corresponding to baseline behavior for the client and the server. 
     
     
         12 . The system of  claim 11 , wherein new normal client-server sequences are identified even after an initial learning phase by updating the one or more model. 
     
     
         13 . The system of  claim 7 , wherein the monitoring phase is implemented by:
 receiving a new communication for analysis;   comparing an extracted sequence for the new communication against the representative sequences identified in the learning phase to generate a score;   determining if the score exceeds a threshold; and   identifying the new communication as a potential threat for malicious payload if the score exceeds the threshold.   
     
     
         14 . The system of  claim 7 , wherein the metadata extracted from the network traffic comprises some or more of IP address information for the client and the server, payload information, quantity information for transferred data, duration of communications, or length of time delay between a client request and a server response. 
     
     
         15 . A computer program product embodied on a computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, executes a method detecting a malicious network payload, comprising:
 collecting network traffic corresponding to communications within a network;   extracting metadata from the network traffic for a client-server session;   in a learning phase, identifying from the metadata any representative sequences for payloads sent between a client and a server for the client-server session; and   in a monitoring phase, detecting for any client-server sequences corresponding to a payload that are different from the representative sequences identified in the learning phase.   
     
     
         16 . The computer program product of  claim 15 , wherein the representative sequences are identified on the basis of a unique combination of the server and a port, where baseline payload characteristics are identified for the unique combination of the server and the port. 
     
     
         17 . The computer program product of  claim 16 , wherein the sequence of instructions which, when executed by a processor, further performs:
 receiving a first communication for the unique combination of the server and the port;   setting the baseline payload characteristics in correspondence to the first communications;   for an additional communication, comparing the additional communication against the baseline payload characteristics to obtain a similarity score, and identifying new baseline payload characteristics that corresponds to the additional communication if the similarity score is not within a threshold distance and a minimum number of sessions or time has elapsed.   
     
     
         18 . The computer program product of  claim 15 , wherein identification of the representative sequences in the learning phase is performed by constructing one or more models corresponding to baseline behavior for the client and the server. 
     
     
         19 . The computer program product of  claim 18 , wherein new normal client-server sequences are identified even after an initial learning phase by updating the one or more model. 
     
     
         20 . The computer program product of  claim 15 , wherein the monitoring phase is implemented by:
 receiving a new communication for analysis;   comparing an extracted sequence for the new communication against the representative sequences identified in the learning phase to generate a score;   determining if the score exceeds a threshold; and   identifying the new communication as a potential threat for malicious payload if the score exceeds the threshold.   
     
     
         21 . The computer program product of  claim 15 , wherein the metadata extracted from the network traffic comprises some or more of IP address information for the client and the server, payload information, quantity information for transferred data, duration of communications, or length of time delay between a client request and a server response.