System and Method for Continuous Low-Overhead Monitoring of Distributed Applications Running on a Cluster of Data Processing Nodes

Abstract

Embodiments of the present invention provide an improvement over known approaches for monitoring of and taking action on observations associated with distributed applications. Application event reporting and application resource monitoring is unified in a manner that significantly reduces storage and aggregation overhead. For example, embodiments of the present invention can employ hardware and/or software support that reduces storage and aggregation overhead. In addition to providing for fine-grained, continuous, decentralized monitoring of application activity and resource consumption, embodiments of the present invention can also provide for decentralized filtering, statistical analysis, and derived data streaming. Furthermore, embodiments of the present invention are securely implemented (e.g., for use solely under the control of an operator) and can use a separate security domain for network traffic.

1 - 20 . (canceled) 
     
     
         21 . A method for monitoring one or more distributed applications running on a cluster comprising a plurality of interconnected data processing nodes, the plurality of data processing nodes comprising respective ones of a plurality of application processing modules each comprising a respective application running thereon, the method comprising:
 receiving, at an application monitoring module of the cluster, at least one monitor command, wherein the application monitoring module is in data communication with each of the plurality of application processing modules via at least a management plane;   configuring the application monitoring module to selectively implement one or more processes for assessing activity of a particular one of the plurality of application processing modules that arise from execution of the respective application running thereon; and   using at least the configured application monitoring module, causing the application monitoring module to selectively implement the one or more processes for the assessing activity of a particular one of the plurality of application processing modules that arise from execution of the respective application running thereon;   wherein, the assessing activity is performed via the management plane such that the one or more processes implemented by the monitoring service module are out-of-band relative to processes of the respective applications running on each one of the application processing modules.   
     
     
         22 . The method of  claim 21 , wherein:
 the configuring the application monitoring module further comprises configuring the application monitoring module to selectively implement one or more processes for outputting event data generated by the particular one of the plurality of application processing modules; and   the using at least the configured application monitoring module further comprises using the application monitoring module to selectively implement the one or more processes for outputting event data generated by the particular one of the plurality of application processing modules.   
     
     
         23 . The method of  claim 22 , wherein the assessing comprises using at least one filter function to manipulate execution of the respective application running thereon to configure at least one aspect of the generation of the event data. 
     
     
         24 . The method of  claim 22 , wherein the outputting the event data comprises applying a time stamp to the event data based upon a global time to which a local time of each one of the nodes is synchronized. 
     
     
         25 . The method of  claim 22 , further comprising implementing, by one or more of the plurality of data processing nodes, a first security domain not accessible by an external entity which issued the at least one monitor command, and implementing, by another one of the plurality of data processing nodes, a second security domain accessible by the external entity; and
 wherein outputting the event data includes transmitting the event data to the another node, thereby enabling the events to be monitored by the external entity without allowing the external entity to interact with the respective application by which the events were generated.   
     
     
         26 . The method of  claim 21 , wherein the configuring is responsive to receiving the command. 
     
     
         27 . The method of  claim 21 , wherein the assessing activity includes alerting an external entity of resource consumption by a particular one of the plurality of application processing modules exceeding a prescribed limit. 
     
     
         28 . The method of  claim 27 , wherein the application monitoring module comprises a resource assessor process, and is present on the management plane. 
     
     
         29 . The method of  claim 27 , further comprising implementing, by one or more of the plurality of data processing nodes, a first security domain not accessible by the external entity, and implementing, by another one of the plurality of data processing nodes, a second security domain accessible by the external entity. 
     
     
         30 . The method of  claim 21 , wherein the management plane comprises one or more dedicated system resources. 
     
     
         31 . The method of  claim 30 , wherein the one or more dedicated system resources comprise one or more dedicated system resources with respect to the application processing modules; and
 wherein the one or more dedicated system resources do not consume resources dedicated to the application processing modules.   
     
     
         32 . A non-transitory computer readable apparatus comprising a storage medium, the storage medium comprising at least one compute program, the at least one computer program configured to, when executed on an out-of-band application monitoring system of a cluster, the cluster comprising a plurality of data processing nodes interconnected by at least the out-of-band application monitoring system, monitor one or more distributed applications running on the plurality of interconnected data processing nodes by at least:
 receipt, at the application monitoring system of the cluster, at least one monitor command; and   responsive to the received at least one monitor command, configuration of the application monitoring system to cause implementation of one or more processes for assessing activity of an application processing module associated with at least one of the plurality of data processing nodes, the activity resulting from execution of an application running on the at least one data processing node; and   wherein, the assessing activity is performed via the out-of-band application monitoring system such that the implemented one or more processes are out-of-band relative to processes of the application running on the at least one data processing node.   
     
     
         33 . The non-transitory computer readable apparatus of  claim 32 , wherein:
 the configuration of the application monitoring system further comprises configuration of the application monitoring system to implement one or more processes for output of event data generated by the application processing module associated with at least one of the plurality of data processing nodes; and   the at least one computer program is further configured to, when executed, monitor, use at least the configured application monitoring system to selectively implement the one or more processes for output of the event data generated by the application processing module associated with at least one of the plurality of data processing nodes to a domain accessible by a computerized user process which issued the at least one monitor command.   
     
     
         34 . The non-transitory computer readable apparatus of  claim 33 , wherein the assessing activity comprises use of at least one filter function to manipulate execution of the application running on the at least one data processing node to configure at least one aspect of the generation of the event data. 
     
     
         35 . The non-transitory computer readable apparatus of  claim 33 , wherein the output of the event data comprises application of time stamp data to the event data based upon a global time to which a local time of each one of the plurality of nodes is synchronized. 
     
     
         36 . The non-transitory computer readable apparatus of  claim 33 , wherein the at least one computer program is further configured to, when executed, implement, by one or more of the plurality of data processing nodes, a first security domain not accessible by the computerized user process, and implementing, by another one of the plurality of data processing nodes, a second security domain accessible by the computerized user process; and
 wherein outputting the event data includes transmitting the event data to the another node, thereby enabling the events to be monitored by the computerized user process without allowing the computerized user process to interact with the application running on the at least one data processing node.   
     
     
         37 . The non-transitory computer readable apparatus of  claim 32 , wherein the assessing activity includes alert of a computerized user process external to the cluster of resource consumption by a particular one of the application processing module associated with the at least one of the plurality of data processing nodes exceeding a prescribed limit. 
     
     
         38 . The non-transitory computer readable apparatus of  claim 37 , wherein the at least one computer program is further configured to, when executed, implement a resource assessor process, and resource assessor process utilizing at least a portion of the out-of-band management system to assess resource utilization of the at least one data processing node relating to execution of the application running on the at least one data processing node. 
     
     
         39 . The non-transitory computer readable apparatus of  claim 37 , wherein the at least one computer program is further configured to, when executed, implement, by one or more of the plurality of data processing nodes, a first security domain not accessible by a computerized user process which issued the at least one monitor command, and implementing, by another one of the plurality of data processing nodes, a second security domain accessible by the computerized user process. 
     
     
         40 . The non-transitory computer readable apparatus of  claim 32 , wherein the out-of-band management system comprises one or more dedicated system resources. 
     
     
         41 . The non-transitory computer readable apparatus of  claim 40 , wherein:
 the one or more dedicated system resources comprise one or more dedicated system resources with respect to at least the application processing module; and   the one or more dedicated system resources do not consume resources dedicated to the application processing module.   
     
     
         42 . A computerized system comprising:
 a cluster of interconnected data processing nodes, the plurality of data processing nodes comprising respective ones of a plurality of central processing units each configured to execute a respective application thereon, and   an out-of-band monitoring system, the out-of-band monitoring system comprising an application monitoring module in data communication with each of the plurality of central processing units, the application monitoring module configured to provide one or more application monitoring services out-of-band of processes executed on each one of the central processing units;   wherein the application monitoring module is configured to:
 selectively implement, responsive to one or more commands received thereby and issued by a computerized user process external to the system, one or more processes to assess activity of a particular one of the plurality of central processing units that arise from execution of the respective application thereof; and 
 based at least on the assessment of the activity, cause forwarding of generated event data related to the assessment to the computerized user process. 
   
     
     
         43 . The system of  claim 42 , wherein the provision of the one or more application monitoring services out-of-band of processes executed on each one of the central processing units is enabled at least in part via configuration of at least the application monitoring module to use dedicated hardware and software resources for provision of the application monitoring services, the dedicated hardware and software resources not being part of the cluster of interconnected data processing nodes. 
     
     
         44 . The system of  claim 43 , wherein the one or more commands comprise resource assessment specification data, the resource assessment specification data configured to enable configuration of a protocol, the protocol used in the assessment of the activity. 
     
     
         45 . The system of  claim 43 , wherein the respective applications comprise at least a portion of a common distributed application. 
     
     
         46 . The system of  claim 42 , wherein the assessment of the activity comprises assessment of consumption of one or more compute resources within at least the particular one of the plurality of central processing units. 
     
     
         47 . The system of  claim 43 , wherein the assessment of the consumption of one or more compute resources within at least the particular one of the plurality of central processing units comprises assessment relative to an operator-defined limit. 
     
     
         48 . The system of  claim 47 , wherein the one or more commands comprise resource assessment specification data, the resource assessment specification data configured to enable configuration of a protocol, the protocol used in the assessment of the activity, the resources assessment specification data comprising data indicative of the operator-defined limit. 
     
     
         49 . The system of  claim 43 , wherein the assessment of the consumption of one or more compute resources within at least the particular one of the plurality of central processing units comprises assessment relative to a limit computed or derived by the out-of-band monitoring system itself. 
     
     
         50 . The system of  claim 42 , wherein the one or more application monitoring services out-of-band of processes executed on each one of the central processing units comprise a plurality of decentralized services that can be provided to respective ones of a plurality of computerized user processes disposed on decentralized computerized apparatus external to the cluster. 
     
     
         51 . The system of  claim 48 , wherein the out-of-band monitoring system further comprises at least one module configured to implement an event data store or log which is inaccessible to the plurality of computerized user processes. 
     
     
         52 . The system of  claim 49 , wherein the implementation of the event data store or log which is inaccessible to the plurality of computerized user processes comprises use of a first security domain wherein the event data store or log is maintained and is inaccessible to the plurality of computerized user processes, and at least one second security domain whereby one or more of the computerized user processes may obtain at least portions of the event data without interaction of the application via the forwarded generated event data.