System and method for optimization and load balancing of computer clusters

Abstract

A system and methods for optimization and load balancing for computer clusters, comprising a distributed computational graph, a server architecture using multi-dimensional time-series databases for continuous load simulation and forecasting, a server architecture using traditional databases for discrete load simulation and forecasting, and using a combination of real-time data and records of previous activity for continuous and precise load forecasting for computer clusters, datacenters, or servers.

What is claimed is: 
     
         1 . A system for optimization and load balancing for computer clusters, comprising:
 a distributed computational graph module comprising a first plurality of programming instructions stored in the memory and operating on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
 receive a distributed computational graph defining a data processing workflow wherein:
 the vertices of the distributed computational graph represent data transformation stages and the edges of the distributed computational graph represent messaging between the data transformation stages; and 
 the data processing workflow comprises one or more data pipelines for analysis of load balancing of a plurality of devices, each data pipeline comprising a series of nodes and edges of the directed computational graph; 
 
 maintain a plurality of connections with each of the plurality of devices over a network, wherein each connection provides the ability to send data to, and receive data from, the respective device over the network; and 
 analyze a dataset using the data pipeline as requested by a load forecasting application to produce a load balancing result; 
   the load forecasting application comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
 query a multidimensional time-series database for portions of the recorded data for one or more of the plurality of devices; 
 utilize the received response data to produce a load simulation, wherein the load simulation comprises a the distributed computational graph; 
 provide the distributed computational graph to the directed computational graph module for execution; 
 receive the load balancing result from the directed computational graph; and 
 create a redistribution of a processing load among the plurality of devices. 
   
     
     
         2 . The system of  claim 1 , wherein the load forecasting application queries data from a database other than a multidimensional time-series database. 
     
     
         3 . The system of  claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on the same computing device as the load forecasting application. 
     
     
         4 . The system of  claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on a device connected by a network.