Methods and apparatus for monitoring, collecting, storing, processing and using network traffic data of overlapping time periods

Abstract

Methods and apparatus for collecting, storing, processing and using data are described. Network traffic probes are identified and attempts are made to configure the probes to generate network traffic data sets which are as close to a preselected common data format as possible. Application layer traffic data is collected in addition to network layer traffic data. The common data format uses delta count values, and terminal count mode format. Network data is obtained from a probe using one of the available table formats which is selected in the following order of preference: alMatrixTopN (Terminal Mode), alMatrixTopN (AllMode), alMatrix, nlMatrixTopN and nlMatrix. A database of collected network traffic information which includes multiple parallel sets of data stored at different resolutions is created. The data sets for each individual resolution are stored in a separate FIFO data structure and the oldest data records are overwritten when allocated data space becomes fully utilized.

What is claimed is:  
     
         1 . A method of processing and storing data in a computer system including processor circuitry, and a data storage device, the method comprising the steps of: 
 storing first and second sets of records on the data storage device, the first and second sets of records being of different data resolutions and corresponding to overlapping periods of time;    operating the processor circuitry to receive data collected over a period of time; and    operating the processor circuitry to update at least one record in each of the stored first and second sets of records with the received data.    
     
     
         2 . The method of  claim 1 , 
 wherein the first and second sets of records are stored in separate first-in, first-out data structures on the data storage device; and    wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records, includes the step of replacing a previous record included in each of the first and second data structures.    
     
     
         3 . The method of  claim 2 , further comprising the step of: 
 allocating fixed amounts of storage space on the data storage device for storing each one of the first and second first-in, first-out data structures used to store the first and second sets of records.    
     
     
         4 . The method of  claim 2 , wherein the first set of records include hourly records and the second set of records includes daily records.  
     
     
         5 . The method of  claim 2 , further comprising the step of: 
 periodically collecting network traffic data;    storing the collected network traffic data in a buffer; and    operating the processor circuitry to retrieve network traffic data from the buffer, the retrieved network traffic data being received by the processor circuitry.    
     
     
         6 . The method of  claim 5 , 
 wherein the network traffic data stored in the buffer includes time stamp information indicating the period of time in which the network traffic data was collected; and    wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records includes the step of: 
 examining at least one time stamp included in the buffered network traffic data.  
   
     
     
         7 . The method of  claim 5 , wherein the collected network traffic data includes byte and packet count information associated with each of a plurality of monitored conversations between devices included in the computer system, the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records including the steps of: 
 updating a record corresponding to a first conversation in the first set of records; and    updating a record corresponding to the first conversation the second set of records.    
     
     
         8 . The method of  claim 5 , 
 wherein the processor circuitry includes first and second central processing units, and    wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records includes the step of operating the first processor to update the first set of records while operating the second processor to update the second set of records.    
     
     
         9 . The method of  claim 1 , 
 wherein the processor circuitry includes first and second central processing units, and    wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records includes the step of operating the first processor to update the first set of records while operating the second processor to update the second set of records.    
     
     
         10 . The method of  claim 5 , wherein the computer system further includes a display device, the method further comprising the step of: 
 displaying data corresponding to overlapping periods of time at different resolutions on the display device.    
     
     
         11 . The method of  claim 1 , further comprising the step of: 
 allocating storage space for storing the first and second sets of records in first and second first-in, first-out data structures, respectively.    
     
     
         12 . A method of collecting and processing network traffic data, comprising the steps of: 
 periodically collecting network traffic data from a data probe,    generating a database of network traffic information from the collected network traffic data, the database comprising a plurality of network traffic data sets of differing degrees of data resolution corresponding to overlapping network traffic time periods.    
     
     
         13 . The method of  claim 12 , wherein the differing degrees of resolution correspond to measurement time periods of different duration.  
     
     
         14 . The method of  claim 12 , 
 wherein the collected network traffic data includes a plurality of traffic data counter values; and    wherein each traffic data counter value in the collected network traffic data includes information corresponding to an individual monitored conversation, the step of generating a database including the step of generating from the information on each different monitored conversation, a different record in each set of the plurality of network traffic data sets.    
     
     
         15 . The method of  claim 14 , further comprising the step of storing each of the plurality of network traffic data sets in a different first-in, first-out data structure.  
     
     
         16 . The method of  claim 15 , wherein a limited amount of data storage space is used for each of the different first-in, first out data structures, the method further comprising the step of: 
 overwriting the oldest data records in the first-in, first-out data structure used to store one of the network traffic data sets, when the limited amount of data storage space used for said first-in, first-out data structure is filled with records.    
     
     
         17 . A system for monitoring network traffic data, comprising: 
 a plurality of network traffic data probes for collecting network traffic information;    processor circuitry coupled to the network traffic probes for receiving data therefrom; and    a data storage device for storing a network traffic database generated by the processor circuitry using data collected by the network traffic data probes, the data storage device including:    a plurality of data structures, each one of the plurality of data structures including network traffic data: 
 a) stored at a different resolution than the resolution at which network traffic data is stored in the other ones of the plurality of data structures; and  
 b) corresponding to a period of time which overlaps the period of time for which network traffic data is stored in the other ones of the plurality of data structures.  
   
     
     
         18 . The system of  claim 17 , wherein each of the plurality of data structures is a first-in, first-out data structure.  
     
     
         19 . The system of  claim 18 , wherein each one of the plurality of data structures includes a plurality of data records, each data record corresponding to a monitored network conversation.  
     
     
         20 . The system of  claim 18 , wherein data records are arranged within each individual data structure as a function of the time the conversation to which the record corresponds was monitored.  
     
     
         21 . The system of  claim 20 , wherein records which were monitored during the same time interval are grouped together within each individual data structure.  
     
     
         22 . The system of  claim 21 , further comprising: 
 means for modifying at least one network traffic data record included in each one of the plurality of data structures to reflect collected information about an individual network conversation.    
     
     
         23 . The system of  claim 18 , further comprising: 
 means for modifying at least one network traffic data record included in each one of the plurality of data structures to reflect collected information about an individual network conversation.    
     
     
         24 . The system of  claim 18 , wherein the processor circuitry includes a plurality of separate central processing units which operate in parallel.  
     
     
         25 . The system of  claim 24 , wherein each one of the plurality of data structures includes a plurality of data records, each data record corresponding to a monitored network conversation.  
     
     
         26 . The system of  claim 24 , wherein data records are arranged within each individual data structure as a function of the time the conversation to which the record corresponds was monitored.  
     
     
         27 . The system of  claim 26 , wherein records which were monitored during the same time interval are grouped together within each individual data structure.  
     
     
         28 . The system of  claim 27 , further comprising: 
 means for modifying at least one network traffic data record included in each one of the plurality of data structures to reflect collected information about an individual network conversation.    
     
     
         29 . The system of  claim 24 , further comprising: 
 means for modifying at least one network traffic data record included in each one of the plurality of data structures to reflect collected information about an individual network conversation.