Methods and systems of function-specific tracing

Abstract

A system and methods are provided for function-specific tracing of a program. In one embodiment, a method includes generating a trace profile identifying one or more functions of a target program, wherein the trace profile identifies one or more functions to trace and depth of tracing for each function to be traced, loading the trace profile and the target program, identifying traced functions in the target program based on the trace profile, patching the target program to call a trace parameter for one or more functions, wherein traced functions are declared at runtime, and observing function calls for traced functions of the application. In this regard, individual functions are traced and debugged on a function-by-function basis without modifying the code or pre-arranging functions so they are traceable. As such, the scope of tracing may be dynamically limited to yield only information that is desired.

1 - 27 . (canceled) 
     
     
         1 . A method of function-specific tracing in a distributed application, the method comprising:
 generating a trace profile by observing the processing of a set of one or more inputs to a target program, the trace profile corresponding to the control flow between a plurality of functions invoked in the processing of said inputs;   evaluating the applicability of one or more user-specified pruning criteria, the criteria selected from a group including a number of invocations, the total time spent in a particular function, the number of times a function is called, and a depth of the function call stack;   pruning the trace profile according to the applicable evaluated pruning criteria;   modifying the target program of the distributed application to provide tracing information from the functions identified in the trace profile;   loading, into a compute controller in a distributed computing system, the modified target program;   executing, by the compute controller, the modified target program;   emitting metadata from the modified target program according to the trace profile; and   collecting emitted metadata by a monitoring program.   
     
     
         2 . The method of  claim 1 , wherein the monitoring program runs in a different process within the same execution environment. 
     
     
         3 . The method of  claim 1 , wherein the monitoring program runs in a different execution environment accessible over a network. 
     
     
         4 . The method of  claim 1 , further comprising:
 generating a second trace profile by observing the processing of a second set of one or more inputs to a second target program, the second trace profile corresponding to the control flow between a plurality of functions invoked in the processing of the second set of inputs, the second set of inputs being associated with the first set of inputs;   evaluating the applicability of the user-specified pruning criteria,   pruning the second trace profile according to the applicable evaluated pruning criteria;   modifying the second target program of the distributed application to provide tracing information from the functions identified in the second trace profile;   loading, into a compute controller in a distributed computing system, the second modified target program;   executing, by the compute controller, the second modified target program;   emitting metadata from the second modified target program according to the trace profile;   collecting second emitted metadata by a monitoring program; and   associating the emitted data and the second emitted data.   
     
     
         5 . The method of  claim 4 , wherein the modified target program and the second modified target program are executed by the same compute controller. 
     
     
         6 . The method of  claim 4 , wherein the modified target program and the second modified target program are executed by different compute controllers within the same distributed system. 
     
     
         7 . The method of  claim 4 , wherein the monitoring program runs in an execution environment accessible to the modified target programs over a network 
     
     
         8 . The method of  claim 1 , wherein elements of the emitted metadata is grouped and/or correlated. 
     
     
         9 . The method of  claim 4 , wherein elements of the emitted metadata and the second emitted metadata are grouped and/or correlated. 
     
     
         10 . A system comprising:
 a trace profiler to generate a trace profile identifying one or more functions of a target program, wherein the trace profile identifies one or more functions to trace by observing the execution of the target program in response to a first set of one or more inputs, and correlates an invoked function address with a function name, and outputs a list of functions corresponding to the control structures traversed within the target program while evaluating the first set of one or more inputs;   a pruner to modify the trace profile by removing or correlating one or more function invocations from the trace profile according to one or more applicable user-specified pruning criteria, the criteria selected from a group including a number of invocations, the total time spent in a particular function, the number of times a function is called, and a depth of the function call stack;   a compiler operable to take an input representing the target program and the trace profile and output an instrumented target program;   a module to load, into a controller module in a first node of distributed computing system, the instrumented target program;   a controller operable to execute the instrumented target program responsive to the reception of a received set of inputs and output program execution metadata received during the operation of the program; and   a monitor for receiving the program execution metadata.   
     
     
         11 . The system of  claim 10 , further comprising:
 a second generated trace profile generated by the trace profiler's observation of the execution of a second target program in response to a second set of one or more inputs;
 a second instrumented target program output by the compiler given an input representing the second target program and the second trace profile; 
   a module to load, into a second controller module in a second node of distributed computing system, the instrumented target program;   a second controller operable to execute the instrumented target program responsive to the reception of a second set of received inputs and outputs second program execution metadata.   
     
     
         12 . The system of  claim 11 , wherein the first node is the second node. 
     
     
         13 . The system of  claim 11 , wherein the received set of inputs and the second set of received inputs are associated with a distributed calling tree representing the execution of a distributed application spanning at least two nodes. 
     
     
         14 . The system of  claim 10 , wherein the monitor is co-located on the first node. 
     
     
         15 . The system of  claim 11 , wherein the monitor is not co-located on the first or second node. 
     
     
         16 . The system of  claim 11 , wherein the monitor correlates the program execution metadata with the second program execution metadata. 
     
     
         17 . A system for profiling distributed applications, the system comprising:
 first, second, and third computing nodes, the first, second, and third nodes communicably coupled via a network;   a first instrumented program executed in a context on the first node, and a second instrumented program executed in a context on the second node, wherein the instrumented programs have been modified to record metadata associated with one or more monitored function invocations, the monitored functions corresponding to a set of functions identified in a first trace profile for the first instrumented program and a second trace profile for the second instrumented program, the trace profiles associating internal function identifiers with function names;   a monitoring program executed in a context on the third node;   wherein the monitoring program associates function call metadata reported from the first and second instrumented program with the multi-node calling tree associated with the execution of a distributed application comprising the first and second instrumented programs.   
     
     
         18 . The system of  claim 17 , further comprising a network monitor, communicably coupled to the third node, operable to record network traffic between the first and second nodes. 
     
     
         19 . The system of  claim 18 , wherein the monitoring program further associates information from the network monitor with the execution of the distributed application. 
     
     
         20 . The system of  claim 17 , wherein elements of the function call metadata reported from the first and the second instrumented programs are grouped and/or correlated.