High performance software-defined core network

Abstract

A system of nodes configured to form a network comprising virtual links in an overlay network provisioned over an underlay network including servers of a public network. The system includes virtual routers (VRs) at each node. Each VR is coupled to the network and to a tenant of the node, and configured to form in the network a set of virtual links corresponding to the tenant. One or more VRs includes a feedback control system comprising an objective function that characterizes the network. The VR is configured to receive link state data of the set of virtual links and control routing of a tenant traffic flow of each tenant according to a best route of the network determined by the objective function using the link state data.

What is claimed is: 
     
         1 . A system comprising:
 a plurality of nodes configured to form a network comprising a plurality of virtual links in an overlay network provisioned over an underlay network including servers of a public network; and   a plurality of virtual routers (VRs) at each node, wherein each VR is coupled to the network and to a tenant of a plurality of tenants of the node, and configured to form in the network a set of virtual links corresponding to the tenant, wherein at least one VR includes a feedback control system comprising at least one objective function that characterizes the network, wherein the at least one VR is configured to receive link state data of the set of virtual links and control routing of a tenant traffic flow of each tenant according to a best route of the network determined by the at least one objective function using the link state data.   
     
     
         2 . The system of  claim 1 , wherein each node includes a plurality of virtual machines (VMs), wherein each VM includes a VR of the plurality of VRs and corresponds to the tenant. 
     
     
         3 . The system of  claim 2 , wherein each VM is configured to isolate at least one of a control plane and a data plane of each tenant from each other tenant of the plurality of tenants. 
     
     
         4 . The system of  claim 2 , wherein the control of the routing of the tenant traffic flow comprises routing the tenant traffic flow from an ingress note to an egress node of the plurality of nodes. 
     
     
         5 . The system of  claim 4 , wherein the control of the routing of the tenant traffic comprises each VR separately controlling routing of each tenant traffic flow to at least one next node of the best route. 
     
     
         6 . The system of  claim 2 , wherein the plurality of virtual links is a component of the overlay network and utilizes the underlay network for delivery of the tenant traffic flow. 
     
     
         7 . The system of  claim 2 , wherein the set of virtual links are configured to form a private tenant network corresponding to the tenant. 
     
     
         8 . The system of  claim 2 , wherein the plurality of virtual links comprises a plurality of single-hop virtual links coupled between each node of the plurality of nodes. 
     
     
         9 . The system of  claim 2 , wherein the plurality of virtual links include a plurality of sets of virtual links, wherein each set of virtual links forms a private tenant network of a corresponding tenant of the plurality of tenants. 
     
     
         10 . The system of  claim 9 , wherein the network includes a plurality of private tenant networks corresponding to the plurality of tenants, wherein each private tenant network is isolated from each other private tenant network of the plurality of private tenant networks. 
     
     
         11 . The system of  claim 10 , wherein the plurality of private tenant networks is configured to maintain separation of multi-tenant traffic flows throughout the network. 
     
     
         12 . The system of  claim 10 , wherein each private tenant network is configured with a tenant configuration of a corresponding tenant to control routing of tenant traffic flows of the tenant. 
     
     
         13 . The system of  claim 12 , wherein the tenant configuration includes traffic classification data, route data, and bandwidth. 
     
     
         14 . The system of  claim 2 , wherein each VR comprises a plurality of routing control algorithms representing a plurality of routing behaviors. 
     
     
         15 . The system of  claim 14 , wherein each routing control algorithm is configured to determine at least one path through the network for routing the tenant traffic flow from an ingress node of the plurality of nodes to an egress node of the plurality of nodes. 
     
     
         16 . The system of  claim 14 , wherein each routing behavior corresponds to a traffic classification of the tenant traffic flow. 
     
     
         17 . The system of  claim 16 , wherein each routing behavior is defined by an objective function of a plurality of objective functions, wherein the plurality of objective functions include the at least one objective function. 
     
     
         18 . The system of  claim 17 , wherein the VR is configured to apply a corresponding objective function to the link state data and generate a link weight for each link of the set of links. 
     
     
         19 . The system of  claim 18 , wherein the VR is configured to determine the best route of the tenant traffic flow according to link weights of the set of links. 
     
     
         20 . The system of  claim 19 , wherein the control of the routing of the tenant traffic flow comprises continually adapting the at least one route in response to changes in the link state data as processed by the corresponding objective function. 
     
     
         21 . The system of  claim 20 , wherein the VR is configured to periodically receive link state updates that include updated link state data of the set of virtual links. 
     
     
         22 . The system of  claim 21 , wherein the continually adapting of the at least one route comprises applying the corresponding objective function to the updated link state data. 
     
     
         23 . The system of  claim 22 , wherein the VR is configured to apply the corresponding objective function to the updated link state data and generate an updated link weight for each link of the set of links. 
     
     
         24 . The system of  claim 23 , wherein the VR is configured to determine an updated best route of the tenant traffic flow according to updated link weights of the set of links. 
     
     
         25 . The system of  claim 24 , wherein each VR of a set of VRs each includes the feedback control system comprising the at least one objective function that characterizes the network. 
     
     
         26 . The system of  claim 24 , wherein each VR of the plurality of VRs includes the feedback control system comprising the at least one objective function that characterizes the network. 
     
     
         27 . The system of  claim 14 , wherein the plurality of routing behaviors includes routing a tenant traffic flow via multiple paths of the network. 
     
     
         28 . The system of  claim 14 , wherein the plurality of routing behaviors includes routing a tenant traffic flow directly via a shortest path of the network. 
     
     
         29 . The system of  claim 14 , wherein the plurality of routing behaviors includes routing a tenant traffic flow on a path and maintaining the tenant traffic flow on the path until detection of an network event. 
     
     
         30 . The system of  claim 29 , wherein the network event includes at least one of a network topology change and a variation in the link state data exceeding a pre-specified threshold. 
     
     
         31 . The system of  claim 2 , wherein the best route includes at least one lowest cost path based on the link state data. 
     
     
         32 . The system of  claim 31 , wherein the link state data of each link represents at least one link metric of the link. 
     
     
         33 . The system of  claim 32 , wherein the at least one link metric includes at least one of latency, jitter, packet loss, throughput, utilization, link state, and link status. 
     
     
         34 . The system of  claim 2 , wherein each VR is configured to maintain configuration data for the corresponding tenant, and to use the configuration data in the control of the routing of the tenant traffic flow. 
     
     
         35 . The system of  claim 34 , wherein the configuration data includes traffic class configuration data. 
     
     
         36 . The system of  claim 35 , wherein the traffic class configuration data identifies traffic classes, and configures MCN behavior corresponding to each traffic class. 
     
     
         37 . The system of  claim 34 , wherein the configuration data includes route configuration data. 
     
     
         38 . The system of  claim 37 , wherein the route configuration data includes data of a service that is a recipient of the tenant traffic flow. 
     
     
         39 . The system of  claim 34 , wherein each VR is configured to maintain topology data including a logical view of the overlay network for the corresponding tenant, and to use the topology data in the control of the routing of the tenant traffic flow. 
     
     
         40 . The system of  claim 2 , wherein each VM includes a monitoring agent coupled to the VR, wherein the monitoring agent is configured to collect data representing the link state data of the set of virtual links of the overlay network. 
     
     
         41 . The system of  claim 40 , wherein the data representing the link state data of the set of virtual links includes at least one link metric of the set of virtual links. 
     
     
         42 . The system of  claim 41 , wherein the at least one link metric includes at least one of latency, jitter, packet loss, throughput, utilization, link state, and link status. 
     
     
         43 . The system of  claim 40 , wherein each monitoring agent is configured to collect the at least one link metric from at least one of a plurality of monitoring agents and a plurality of VRs of the plurality of VMs. 
     
     
         44 . The system of  claim 43 , wherein the VR is configured to configure the set of virtual links for use by the monitoring agent. 
     
     
         45 . The system of  claim 40 , wherein each monitoring agent is configured to collect the at least one link metric using probe signals exchanged with others of the plurality of VMs. 
     
     
         46 . The system of  claim 45 , wherein each VM is configured to send the at least one link metric to the monitoring agent transmitting the probe signals in response to receipt of the probe signals. 
     
     
         47 . The system of  claim 40 , wherein the monitoring agent is configured to generate the link state data of the set of virtual links by processing the at least one link metric. 
     
     
         48 . The system of  claim 47 , wherein the VR is configured to receive from the monitoring agent the link state data of the set of virtual links. 
     
     
         49 . The system of  claim 40 , wherein each VR is configured to receive the link state data of others of the plurality of links from others of the plurality of VRs. 
     
     
         50 . The system of  claim 40 , wherein each VM includes a virtual gateway coupled to the corresponding tenant and the corresponding VR, wherein the virtual gateway is configured to control tenant traffic flows incoming to the VM from the corresponding tenant. 
     
     
         51 . The system of  claim 50 , wherein the virtual gateway is coupled to the monitoring agent. 
     
     
         52 . The system of  claim 50 , wherein the VR is configured to generate at least one set of flow rules configured to control the routing of the tenant traffic flow through the overlay network. 
     
     
         53 . The system of  claim 52 , wherein the at least one set of flow rules corresponds to the at least one objective function. 
     
     
         54 . The system of  claim 50 , wherein the virtual gateway is configured to attract tenant traffic flows of the corresponding tenant. 
     
     
         55 . The system of  claim 54 , wherein the virtual gateway is configured to reject traffic flows arriving from sources other than the corresponding tenant. 
     
     
         56 . The system of  claim 54 , wherein each VM includes a set of public IP addresses, wherein the set of public IP addresses is dedicated to the corresponding tenant, wherein the corresponding tenant accesses the virtual gateway of the VM using the set of public IP addresses. 
     
     
         57 . The system of  claim 52 , comprising at least one virtual switch coupled to the VR and the virtual gateway of each VM. 
     
     
         58 . The system of  claim 57 , wherein the at least one virtual switch includes a set of routing tables representing the at least one set of flow rules. 
     
     
         59 . The system of  claim 58 , wherein the set of routing tables is configured to manage the control of the routing of the tenant traffic flow through the network. 
     
     
         60 . The system of  claim 57 , wherein the at least one virtual switch is configured to transfer the tenant traffic flow between the virtual gateway and the VR. 
     
     
         61 . The system of  claim 57 , wherein at least one of the virtual gateway, the VR, and the at least one virtual switch are configured to form the set of virtual links. 
     
     
         62 . The system of  claim 57 , comprising an aggregator coupled to the at least one virtual switch. 
     
     
         63 . The system of  claim 62 , wherein the aggregator is configured to route via the set of virtual links the tenant traffic flow received at the virtual gateway from the corresponding tenant. 
     
     
         64 . The system of  claim 62 , wherein the aggregator is configured to route to the corresponding tenant the tenant traffic flow received at the node via the network. 
     
     
         65 . The system of  claim 64 , wherein the tenant traffic flow arriving at the aggregator via the network is routed to the corresponding tenant via at least one of the corresponding VR and the virtual gateway. 
     
     
         66 . The system of  claim 64 , wherein the virtual gateway routes the tenant traffic flow arriving at the aggregator to the tenant via a coupling over a public network. 
     
     
         67 . The system of  claim 62 , wherein each node includes an aggregator coupled to the at least one virtual switch and the network. 
     
     
         68 . The system of  claim 67 , wherein the aggregator is configured to route via the network the tenant traffic flows of the plurality of tenants corresponding to the node. 
     
     
         69 . The system of  claim 62 , wherein each node includes a hypervisor, wherein the hypervisor is configured as an operating system of the plurality of VMs of the node. 
     
     
         70 . The system of  claim 69 , wherein the hypervisor is configured to include the at least one virtual switch. 
     
     
         71 . The system of  claim 70 , wherein the hypervisor is configured to include the aggregator. 
     
     
         72 . The system of  claim 2 , comprising a provisioner coupled to the plurality of VMs of the plurality of nodes, wherein the provisioner is configured to control provisioning of at least one of the overlay network and the underlay network. 
     
     
         73 . The system of  claim 72 , wherein the provisioner is coupled to a queue comprising at least one pre-provisioned network, wherein the control of the provisioning of the underlay network includes use of a pre-provisioned network of the queue as the underlay network. 
     
     
         74 . The system of  claim 72 , wherein the provisioner is configured to control configuration of the plurality of VMs of the plurality of nodes. 
     
     
         75 . The system of  claim 74 , wherein the provisioner is configured to control configuration of components of each VM of the plurality of VMs using a tenant configuration of the corresponding tenant. 
     
     
         76 . The system of  claim 75 , wherein the provisioner is configured to generate routes corresponding to each of the plurality of tenants. 
     
     
         77 . The system of  claim 72 , wherein the provisioner is configured to maintain network data of at least one of the overlay network and the underlay network, wherein the network data includes data representing the overlay network, the underlay network, route configurations, topology data of the network including the plurality of virtual links, and tenant configurations of the plurality of tenants. 
     
     
         78 . The system of  claim 77 , comprising a web application coupled to the provisioner, wherein the web application is configured to generate a user interface configured to generate for presentation prompts for data representing the tenant configuration, and to receive data input of the tenant. 
     
     
         79 . The system of  claim 78 , wherein the web application is configured to maintain link state data of the plurality of virtual links, and link metrics represented by the link state data. 
     
     
         80 . The system of  claim 79 , wherein the web application includes an alerts engine configured to generate and manage alerts and notifications, wherein the alerts and notifications correspond to at least one of the link state data and the link metrics. 
     
     
         81 . A system comprising:
 a plurality of nodes configured to form a network comprising virtual links;   a plurality of virtual machines (VMs) at each node, wherein each VM is coupled to the network and to a tenant of a plurality of tenants of the node;   a feedback control system in at least one VM, comprising a plurality of objective functions representing a plurality of routing behaviors, wherein each objective function is configured to continually characterize the network per traffic flow based on link state data of the virtual links received from a set of the VMs, wherein the feedback control system determines based on the characterization a best route through the network, and controls routing of each traffic flow to at least one next node of the best route.