Identifying and localizing equipment failures

Abstract

The disclosed technology is directed towards automatically detecting failure states and the cause of the failure. For a network, the technology collects status messages from equipment and customers into batches as they occur. The technology groups and aggregates messages, then transforms the aggregations to the frequency domain. Anomalies induce detectable changes in the particle distribution of a trained particle filter, from which an anomalous spectrogram is generated. The status messages of each device are iteratively removed from the larger set of messages, resulting in reduced subsets that are each aggregated, transformed into a modified spectrogram and compared against the anomalous spectrogram to obtain a distance score. The distance score for each device is used to rank the devices with respect to being the cause of the failure.

What is claimed is: 
     
         1 . A system, comprising:
 a processor; and   a memory that stores executable instructions which, when executed by the processor of the system, facilitate performance of operations, the operations comprising:   in response to a determination, according to a defined criterion, that a first spectrogram represents a low probability hypothesis, indicating that a failure state exists among a group of network devices, removing, from aggregated status messages associated with the group of network devices, status messages associated with a selected network device, of the group of network devices, to obtain a reduced aggregated message batch;   transforming the reduced aggregated message batch to a defined frequency domain data representation to obtain a second spectrogram;   comparing the second spectrogram with the first spectrogram to determine a similarity score; and   based on the similarity score, generating a failure probability score for the selected network device.   
     
     
         2 . The system of  claim 1 , wherein the comparing of the second spectrogram with the first spectrogram to determine the similarity score comprises determining a relative distance value between the first spectrogram and the second spectrogram. 
     
     
         3 . The system of  claim 1 , wherein the operations further comprise iteratively repeating the obtaining of the failure probability score for respective selected network devices, and ranking the group of network devices based on respective failure probability scores for the respective selected network devices. 
     
     
         4 . The system of  claim 3 , wherein a tie in the ranking is determined based on the failure probability scores of two most likely failed network devices, and wherein the operations further comprise selecting a most likely failed device from the two most likely failed network devices based on respective proximities of the two most likely failed network devices to a customer device associated with a customer. 
     
     
         5 . The system of  claim 1 , wherein the status messages comprise network protocol messages. 
     
     
         6 . The system of  claim 1 , wherein the status messages comprise customer indications received from customer devices. 
     
     
         7 . The system of  claim 1 , wherein the status messages associated with the group of network devices are grouped from a larger message dataset, comprising more status messages than the status messages, based on at least one of: a customer group served by the group of network devices, or network device service type information. 
     
     
         8 . The system of  claim 1 , wherein the operations further comprise:
 processing the first spectrogram with a particle filter trained with normal operational data, corresponding to normal operating conditions of the group of network devices according to a defined criterion, to determine whether the first spectrogram represents a low probability hypothesis; and   testing the particle filter with input data corresponding to known failure conditions.   
     
     
         9 . The system of  claim 1 , wherein the status messages comprise error codes. 
     
     
         10 . The system of  claim 1 , wherein the status messages comprise alarms output by the group of network devices. 
     
     
         11 . A method, comprising:
 in response to a determination that a spectrogram is an anomalous spectrogram indicative of a failure condition relative to a non-anomalous operating condition, the determination having been made by processing the spectrogram with a particle filter trained with data corresponding to known non-anomalous operating conditions to determine that the spectrogram represents a low probability hypothesis, removing, by a system comprising a processor, respective status messages associated with respective network devices to obtain respective reduced aggregated message datasets;   transforming, by the system, the respective reduced aggregated message datasets to respective frequency domain data representations corresponding to respective spectrograms; and   ranking, by the system, the respective network devices, based on the respective spectrograms, to determine which network device of the respective network devices is most probable as being responsible for the failure condition.   
     
     
         12 . The method of  claim 11 , wherein the ranking of the respective network devices, based on the respective spectrograms, to determine which network device of the respective network devices is most probable as being responsible for the failure condition comprises comparing the respective spectrograms with the anomalous spectrogram to determine respective similarity scores. 
     
     
         13 . The method of  claim 12 , wherein two network devices of the respective network devices are determined to be candidates for being most probable as being responsible for the failure condition, and further comprising modifying the similarity scores of at least one candidate based on device proximity to a customer location. 
     
     
         14 . The method of  claim 11 , further comprising:
 outputting, by the system, a list indicating ranks of the respective network devices according to the ranking.   
     
     
         15 . The method of  claim 11 , further comprising:
 selecting, by the system, a group of status messages associated with the respective network devices from a larger message dataset based on at least one selection criterion.   
     
     
         16 . A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, the operations comprising:
 removing respective status messages associated with respective network devices, of network devices, from an anomalous spectrogram;   generating a modified spectrogram based on non-removed status messages of the network devices remaining after the removing, wherein the generating the modified spectrogram comprises applying a Fourier transform to counts of the status messages, collected during a time window, of the non-removed status messages remaining after the removing; and   ranking the network devices, based on respective similarity scores for the respective modified spectrograms relative to the anomalous spectrogram, to determine which network device of the respective network devices is most probable as being responsible for an anomalous operating condition.   
     
     
         17 . The non-transitory machine-readable medium of  claim 16 , wherein the operations further comprise:
 transforming message counts of the status messages collected during a time window into an operating condition spectrogram.   
     
     
         18 . The non-transitory machine-readable medium of  claim 17 , wherein the operations further comprise:
 processing the operating condition spectrogram via a particle filter trained with data corresponding to known normal operating conditions to determine that the operating condition spectrogram represents a low probability hypothesis, resulting in the anomalous spectrogram.   
     
     
         19 . The non-transitory machine-readable medium of  claim 16 , wherein the operations further comprise selecting the network devices based on a selection criterion. 
     
     
         20 . The non-transitory machine-readable medium of  claim 16 , wherein the operations further comprise:
 using proximity data representative of respective proximities of two respective network devices to a customer as a tiebreaker of two respective similarity scores within a difference range of the two respective network devices.