Adaptive resource management in distributed computing systems

Abstract

The present disclosure provides a framework for adaptive resource handling of applications being executed in distributed systems so as to ensure efficient resource utilization. The present disclosure provides a framework: to enable a client to participate in identifying application uniquely using tags such that resource adaptation is more effective, to collect and store resource statistics for an application task against various parameters, to monitor resources done based on historical statistics collected, to resource management is done based on historically identified resource usage limits, and if exact match for the records are not found then based on confidence score the limits are identified, and to enable each resource manager (RM) agent to receives the workers to be launched with both limits: given by the application (hard limit) and by the historical statistics (soft limit).

We claim: 
     
       1. A system for adaptive resource management, the system comprising a resource manager (RM) and at least one RM Agent, wherein:
 the resource manager (RM) is configured to:
 receive at least one resource request from at least one application, wherein the application comprises an application name and a tag, and the resource request comprises a resource requirement of an application task for execution; 
 provide at least a soft limit for the application task based on at least a historical resource usage statistics pre-stored in a history data store or return to a hard limit for the application task configured by at least one user; and 
 allocate at least one resource from the RM Agent identified for execution of the application task, wherein the RM is configured to allocate at least one other resource for the execution of the application task from the RM Agent identified when resource usage of the resource allocated for execution exceeds the soft limit; and 
 
 the RM Agent is configured to:
 execute the application task based at least on the soft limit or the hard limit; 
 monitor the application task during execution; and 
 store a resource usage statistics of the resource of the application task against the tag. 
 
 
     
     
       2. The system as claimed in  claim 1 , wherein the resource requirement of the application task is selected from at least one of a memory, input/output (I/O) process, or central processing unit (CPU). 
     
     
       3. The system as claimed in  claim 1 , wherein the soft limit or the hard limit for the application task is provided by matching the resource requirement of an application task with at least one of the application name, the application task, the RM Agent, or a RM Agent type, pre-stored in the history data store. 
     
     
       4. The system as claimed in  claim 3 , wherein the resource manager is further configured to: apply a configurable formula associated with resource usage to identify the soft limit of the application task when a match is found. 
     
     
       5. The system as claimed in  claim 3 , wherein when a match is not found, the resource manager is further configured to:
 generate a confidence score for different set of historical resource usage pre-stored in the history data store associated with the application; 
 select a best confidence score from the confidence score generated; and 
 apply a configurable formula associated with resource usage to identify the soft limit of the application task. 
 
     
     
       6. The system as claimed in  claim 1 , wherein if resource usage of the resource allocated for execution exceeds the hard limit, the execution of the application task is stopped or killed. 
     
     
       7. The system as claimed in  claim 1 , wherein the tag comprises a task type identifying a given task of the application. 
     
     
       8. The system as claimed in  claim 1 , wherein the resource usage statistics comprises at least one of the application name, an application identification, application task type, the tag, or a resource provided type. 
     
     
       9. A resource manager (RM) for adaptive resource management, the resource manager comprising:
 a processor; 
 a memory coupled to the processor for executing a plurality of instructions stored in the memory, wherein the processor executes the instructions to: 
 receive at least one resource request from at least one application, wherein the application comprises an application name and a tag, and the resource request comprises a resource requirement of an application task for execution; 
 provide at least a soft limit for the application task based on at least a historical resource usage statistics pre-stored in a history data store or return to a hard limit for the application task configured by at least one user; and 
 allocate, based on the soft limit or a hard limit, at least one resource for execution of the application task from at least one RM Agent identified, wherein the processor executes the instructions to allocate at least one other resource for the execution of the application task from at least one RM Agent identified when resource usage of the resource allocated for execution exceeds the soft limit. 
 
     
     
       10. The resource manager as claimed in  claim 9 , wherein the resource requirement of the application task is selected from at least one of a memory, input/output (I/O) process, or central processing unit (CPU). 
     
     
       11. The resource manager as claimed in  claim 9 , the soft limit or the hard limit for the application task is provided by matching the resource requirement of the application task with at least one of the application name, the application task, the RM Agent, or a RM Agent type, pre-stored in the history data store. 
     
     
       12. The resource manager as claimed in  claim 11 , wherein if a match is found, the processor executes the instructions to: apply a configurable formula associated with resource usage to identify the soft limit of the application task. 
     
     
       13. The resource manager as claimed in  claim 11 , wherein if match not found, the processor executes the instructions to:
 generate a confidence score for different set of historical resource usage pre-stored in the history data store associated with the application; 
 select a best confidence score from the confidence score generated; and 
 apply a configurable formula associated with resource usage to identify the soft limit of the application task. 
 
     
     
       14. The resource manager as claimed in  claim 9 , wherein if resource usage of the resource allocated for execution exceeds the hard limit, an execution of the application task is stopped or killed. 
     
     
       15. The resource manager as claimed in  claim 9 , wherein the tag comprises a task type identifying a given task of the application. 
     
     
       16. The resource manager as claimed in  claim 9 , wherein the historical resource usage statistics comprises at least one of the application name, an application identification, application task type, the tag, or a resource provided type. 
     
     
       17. A method for adaptive resource management (RM), the method comprising:
 receiving at least one resource request from at least an application, wherein the application comprises an application name and a tag, and the resource request contains at least a resource requirement of an application task for execution; 
 identifying at least one RM Agent for the execution of the application task; 
 providing at least a soft limit for the application task based on at least a historical resource usage statistics pre-stored in a history data store or a hard limit for the application task configured by at least one user; 
 allocating at least one resource from the RM Agent identified for execution of the application task, wherein the allocating at least one resource from the RM Agent identified for execution of the application task comprises: when resource usage of the resource allocated for execution exceeds the soft limit, allocating at least one other resource from the RM Agent identified for the execution of the application task; 
 scheduling the application task based at least on the soft limit or the hard limit; 
 monitoring a resource usage of the application task during for execution; and 
 storing a resource usage statistics of the application task in the history data store against tag. 
 
     
     
       18. The method as claimed in  claim 17 , wherein the soft limit or the hard limit for the application task is provided by matching the resource requirement of the application task with at least one of the application name, the application task, the RM Agent, or a RM Agent type, pre-stored in the history data store. 
     
     
       19. The method as claimed in  claim 18 , wherein the method further comprises applying a configurable formula associated with resource usage to identify the soft limit of the application task when a match is found.