System and Method for a Self-Optimizing Reservation in Time of Compute Resources

Abstract

A system and method of dynamically controlling a reservation of resources within a cluster environment to maximize a response time are disclosed. The method embodiment of the invention comprises receiving from a requestor a request for a reservation of resources in the cluster environment, reserving a first group of resources, evaluating resources within the cluster environment to determine if the response time can be improved and if the response time can be improved, then canceling the reservation for the first group of resources and reserving a second group of resources to process the request at the improved response time.

1 - 20 . (canceled) 
     
     
         21 . A method of operating a compute environment for processing of one or more compute workloads, the method comprising:
 receiving a workload submission for processing by the compute environment, the workload submission including at least a first resource request and a second resource request;   based on at least on the first resource request, causing creation of a reservation of a first resource within the compute environment;   based at least on the second resource request, causing determination of a second resource, the second resource at least being able to meet one or more requirements associated with the second resource request; and   causing the determined second resource to be available for use by at least the first resource during at least a portion of the reservation.   
     
     
         22 . The method of  claim 21 , wherein the first resource request comprises a request relating to one or more compute resources, and the second resource request comprises a request relating to one or more data storage resources. 
     
     
         23 . The method of  claim 22 , wherein:
 the method further comprises processing at least a portion of the workload submission using at least the one or more data storage resources and the one or more compute resources; and   the one or more data storage resources are accessible to the one or more compute resources such that data stored on the one or more storage resources can be processed by the one or more compute resources as part of the processing of the at least portion of the workload submission.   
     
     
         24 . The method of  claim 22 , wherein the one or more storage resources comprises pre-staged data for use by the one or more compute resources. 
     
     
         25 . The method of  claim 24 , further comprising pre-staging the pre-staged data so that it is accessible to the one or more compute resources concurrent with at least a start of an execution of at least portion of the workload during the reservation. 
     
     
         26 . The method of  claim 25 , wherein the pre-staging the pre-staged data so that it is accessible to the one or more compute resources concurrent with at least a portion of the reservation comprises pre-staging the pre-staged data concurrent with execution of at least a portion of the submitted workload by the one or more compute resources. 
     
     
         27 . The method of  claim 21 , wherein the causing determination of a second resource, the second resource at least being able to meet one or more requirements associated with the second resource request, comprises evaluating a plurality of ones of second resources to determine at least one of the plurality which has an availability;
 wherein the availability enables the first resources to process at least part of the submitted workload during the reservation.   
     
     
         28 . The method of  claim 21 , wherein:
 the first resource request comprises a request relating to one or more compute resources, and the second resource request comprises a request relating to one or more data storage resources; and   the one or more requirements comprise one or more data storage requirements.   
     
     
         29 . The method of  claim 21 , wherein:
 the first resource request comprises a request relating to one or more compute resources, and the second resource request comprises a request relating to one or more data storage resources; and   the one or more requirements comprise one or more ordering requirements relating to performance of one or more actions for the one or more compute resources relative to performance of one or more actions for the one or more data storage resources.   
     
     
         30 . The method of  claim 29 , wherein the performance of one or more actions for the one or more compute resources comprises performing the one or more actions for the one or more data storage resources as a condition precedent to performance of the one or more actions for the one or more compute resources. 
     
     
         31 . The method of  claim 21 , further comprising causing reservation of the determined second resource. 
     
     
         32 . The method of  claim 31 , wherein the reservation of the determined second resource and the reservation of the first resource temporally overlap with one another for at least a period of time. 
     
     
         33 . The method of  claim 21 , wherein:
 the workload submission specifies one or more quality of service (QoS) requirements;   the first resource request comprises a request relating to one or more compute resources, and the second resource request comprises a request relating to one or more data storage resources; and   the method further comprises processing at least a portion of the workload submission using at least the one or more data storage resources and the one or more compute resources so that the one or more QoS requirements are met.   
     
     
         34 . The method of  claim 21 , wherein:
 the first resource request comprises a request relating to one or more compute resources, and the second resource request comprises a request relating to one or more data storage resources; and   the processing at least a portion of the workload submission using at least the one or more data storage resources and the one or more compute resources so that the one or more QoS requirements are met comprises pre-staging data needed for the processing on the one or more data storage resources in advance of the processing.   
     
     
         35 . The method of  claim 21 , wherein:
 the first resource request comprises a request relating to one or more compute resources, and the second resource request comprises a request relating to one or more data storage resources; and   further comprising releasing at least a portion of the one or more data storage resources which are no longer required by the one or more compute resources.   
     
     
         36 . The method of  claim 35 , wherein the releasing at least a portion of the one or more data storage resources which are no longer required by the one or more compute resources occurs prior to a termination or expiration of the reservation for the one or more compute resources. 
     
     
         37 . A method of operating a compute environment for processing of one or more compute workloads, the method comprising:
 receiving a workload submission for processing by the compute environment;   based on at least on the received workload submission, causing creation of a reservation of at least one compute resource within the compute environment;   identifying at least one data resource that is subject to a reservation and which meets one or more requirements associated with the workload submission, the identified at least one data resource having an availability concurrent with at least a portion of the reservation of the at least one compute resource; and   utilizing the at least one compute resource and the identified at least one data resource to process at least a portion of the workload submission at least during the portion of the reservation of the at least one compute resource.   
     
     
         38 . The method of  claim 37 , wherein:
 the one or more requirements comprises one or more data storage requirements;   the causing creation of a reservation of at least one compute resource within the compute environment comprises causing creation of a reservation of at least one node within the compute environment; and   the identifying at least one data resource that is subject to a reservation and which meets one or more requirements associated with the workload submission comprises identifying at least one data storage resource in data communication with the at least one node.   
     
     
         39 . The method of  claim 37 , wherein the utilizing the at least one compute resource and the identified at least one data resource to process at least a portion of the workload submission comprises pre-staging data stored on the identified at least one data resource in advance of use of the pre-staged data by the at least one compute resource to process the at least portion of the workload submission. 
     
     
         40 . The method of  claim 37 , further comprising pre-staging data stored on the identified at least one data resource in advance of use of the pre-staged data by the at least one compute resource during the processing of the at least portion of the workload submission. 
     
     
         41 . The method of  claim 40 , wherein the pre-staging of the data stored on the identified at least one data resource in advance of use of the pre-staged data by the at least one compute resource during the processing of the at least portion of the workload submission comprises prestaging the stored data by at least a prescribed period of time in advance of execution of the at least portion of the workload submission. 
     
     
         42 . The method of  claim 37 , wherein the receiving a workload submission for processing by the compute environment comprises receiving at least one request specifying at least: (i) one or more processor requests, and (ii) one or more data storage requests; and
 wherein the one or more requirements are based at least in part on a computerized evaluation of at least one of (i) or (ii).   
     
     
         43 . A computer readable apparatus comprising a non-transitory storage medium having a plurality of instructions thereon, the plurality of instructions configured to, when executed by a processing apparatus, cause operation of a compute environment so as to process one or more compute workloads by at least:
 receipt of data relating to the one or more compute workloads, the received data comprising data requesting at least one compute resource and at least one data storage resource;   creation, based on at least on the received data, of at least one reservation of at least one compute resource within the compute environment;   creation, based at least on the received data, of a reservation of at least one data resource, wherein:
 (i) the at least one data resource comprising data pre-staged thereon and necessary for processing at least part of the one or more compute workloads; and 
 (ii) at least a portion of the reservation of the at least one data resource is concurrent with at least a portion of the reservation of the at least one compute resource; and 
   utilizing the at least one compute resource and the at least one data resource to process at least a portion of the one or more compute workloads at least during the concurrent at least portion of the reservation of the at least one compute resource and the at least portion of the of the reservation of the at least one data resource.   
     
     
         44 . A computer readable apparatus comprising a non-transitory storage medium having a plurality of instructions thereon, the plurality of instructions configured to, when executed by a processing apparatus, cause operation of a compute environment so as to process one or more compute workloads by at least:
 receipt of data relating to the one or more compute workloads, the received data comprising data requesting one or more compute resources and data requesting one or more data resources;   configuration, based at least on the data requesting the one or more data resources, of at least one of a plurality of data resources accessible to the compute environment, the configuration comprising pre-staging data necessary for processing of at least one of the one or more compute workloads, the pre-staging performed in advance of the processing of the at least one compute workload;   creation, based on at least on the data requesting the one or more compute resources, of at least one reservation of at least one of a plurality of compute resources within the compute environment, the reservation for performance of the processing of the at least one compute workload; and   utilizing the at least one compute resource and at least the pre-staged data to process the at least one compute workload during at least a portion of the reservation of the at least one compute resource.   
     
     
         45 . The computer readable apparatus of  claim 44 , wherein:
 the creation of the at least one reservation is performed at least by a computerized scheduler process of the compute environment; and   the at least one data resource comprises a data storage device in network data communication with the compute environment.   
     
     
         46 . The computer readable medium of  claim 44 , wherein the plurality of instructions are further configured to, when executed by a processing apparatus, release at least a portion of the at least data resource which is no longer used for processing of the at least one compute workload. 
     
     
         47 . The computer readable medium of  claim 46 , wherein the plurality of instructions are further configured to, when executed by a processing apparatus, configure one of the plurality of data resources accessible to the compute environment, the configuration comprising pre-staging second data necessary for processing of at least one of the one or more compute workloads, the pre-staging of the second data performed after the release but in advance of processing of the at least one compute workload which requires the second data. 
     
     
         48 . The computer readable medium of  claim 44 , wherein the plurality of instructions are further configured to, when executed by a processing apparatus, utilize one of the plurality of data resources accessible to the compute environment to stage out data generated by the processing of the at least one compute workload, the staging out of the generated data performed after the release but in advance of any release of the one or more compute resources from the reservation.