Systems and methods for large-scale system log analysis, deduplication and management

Abstract

System and methods for parsing raw log data into structured log data, and removing duplicate entries, storing the deduplicated log data into binary format, and managing system events. The subject matter can increase speed of log data analysis and storage, reduce data storage for log data, and easily manage system events.

What is claimed is: 
     
         1 . A computer-implemented system for log data analysis and deduplication, the system comprising:
 (a) one or more digital processing devices comprising an operating system configured to perform executable instructions;   (b) one or more data storages storing one or more binary files;   (c) a computer program including instructions executable by the one or more digital processing devices to create an application, the application comprising:
 (1) a parser module configured to: receive raw log data, and transform the raw log data into structured log data; and 
 (2) a storage module without a database service configured to:
 (i) identify duplicated data entries in the structured log data; and 
 (ii) generate a serialized data record and store the serialized data record into the one or more binary files, wherein the serialized data record comprises one or more timestamps of the duplicated data entries and a single representative of the duplicated data entries. 
 
   
     
     
         2 . The system of  claim 1 , provided that the structured log data is in a binary format and comprises (a) the one or more timestamps and (b) one or more system event identifiers. 
     
     
         3 . The system of  claim 2 , provided that the one or more system event identifiers comprise an alphanumeric representation of a system event. 
     
     
         4 . The system of  claim 3 , provided that the system event comprises one or more of the following: a network event, an actionable network event, a system failure, a network failure, a user activity, a phishing activity, a spam, an advertisement, a fake message, a hack, an attack, an intrusion, a malicious activity, an unnatural activity, and a virus. 
     
     
         5 . The system of  claim 2 , provided that the one or more system event identifiers comprise one or more checksums. 
     
     
         6 . The system of  claim 2 , provided that the one or more system event identifiers are derived within the time window. 
     
     
         7 . The system of  claim 2 , provided that the one or more system event identifiers are accessible by the storage module within the time window, or are inaccessible by the storage module when a time lapses the time window. 
     
     
         8 . The system of  claim 1 , provided that the parser module is further configured to transmit the structured log data to the parser module through a push. 
     
     
         9 . The system of  claim 1 , provided that the transforming the raw log data into the structured log data comprises one or more of the following: (a) processing the raw log data by a hash function; (b) deriving one or more system event identifiers from one or more data fields of the raw log data; and (c) generating the structured log data. 
     
     
         10 . The system of  claim 1 , provided that the duplicated data entries are associated with a same system event identifier in the structured log data. 
     
     
         11 . The system of  claim 1 , provided that the storage module is further configured to perform one or more of the following: (a) count a number of occurrences in the duplicated data entries; (b) retain timestamps of the duplicated data entries; and (c) retrieve the serialized data record from the one or more binary files. 
     
     
         12 . The system of  claim 1 , provided that the storage module further comprises an online analytical processing tool configured to aggregate a data field in the one or more binary files. 
     
     
         13 . The system of  claim 12 , provided that the data field in the one or more binary files comprises one of the following: a number of occurrences of a host server, a number of occurrences of a program, a number of occurrences of a severity level, a number of occurrences of an system event, a list of a last active host server, and a summary of system events within a time window. 
     
     
         14 . The system of  claim 1 , provided that the application further comprises a mapping module configured to create, record, modify or delete a map between one or more system event identifiers and the storage module. 
     
     
         15 . The system of  claim 14 , provided that the mapping module maps one or more system events associated with a same system event identifier to a same storage module. 
     
     
         16 . The system of  claim 1 , provided that the application further comprises a detection module configured to detect a system event from one or more of the following: the structured log data, and the one or more binary files. 
     
     
         17 . The system of  claim 16 , provided that the system event comprises one or more of the following: a user activity, a phishing activity, a spam, an advertisement, a fake message, a hack, an attack, an intrusion, a malicious activity, an unnatural activity, and a virus. 
     
     
         18 . The system of  claim 16 , provided that the detection module generates an alert when a system event meets one or more alert rules. 
     
     
         19 . The system of  claim 1 , provided that the application further comprises a query module configured to interact with a user to enter, adjust or retrieve one or more data parameters. 
     
     
         20 . The system of  claim 19 , provided that the one or more data parameters comprise one or more data fields or data tokens extracted from the raw log data. 
     
     
         21 . The system of  claim 19 , provided that the one or more data parameters comprise one or more of the following: a number of occurrence of hosts, a number of occurrence of programs, a number of occurrence of system events, a level of severity, a list of active hosts, a list of last active hosts, and a time period for the storage module receiving the structured log data. 
     
     
         22 . The system of  claim 1 , provided that the application further comprises a full text search indexer configured to:
 (a) index the serialized data records stored in the one or more binary files; and   (b) allow searching or filtering on the serialized data records.   
     
     
         23 . The system of  claim 22 , provided that the full text search indexer comprises an index list. 
     
     
         24 . The system of  claim 23 , provided that configuration of the index list is automatic or dynamic. 
     
     
         25 . The system of  claim 1 , provided that the application further comprises a manager module configured to configure the parser module or the storage module. 
     
     
         26 . The system of  claim 25 , provided that configuring the parser module or the storage module is achieved remotely or locally. 
     
     
         27 . The system of  claim 25 , provided that configuring the parser module or the storage module is achieved in a distributive way. 
     
     
         28 . The system of  claim 25 , provided that the manager module further comprises a user interface, the interface allowing a user to
 (a) enter or modify one or more configuration parameters for the parser module or the storage module; or   (b) receive one or more existing configuration parameters of the parser module or the storage module.   
     
     
         29 . Non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for log data analysis and deduplication, the application comprising:
 (a) a parser module configured to: receive raw log data, and transform the raw log data into structured log data, wherein the transformation comprises: processing the raw log data by a hash function, deriving one or more system event identifiers from one or more data fields of the raw log data, and generating the structured log data in a binary format; and   (b) a storage module without a database service configured to:
 (1) identify duplicated data entries in the structured log data; and 
 (2) generate a serialized data record and store the serialized data record into one or more binary files, wherein the serialized data record comprises one or more timestamps of the duplicated data entries and a single representative of the duplicated data entries.