Enabling memory transactions across a lossy network

Abstract

Methods and systems for enabling remote programmed I/O to be carried out across a “lossy” network are provided. According to one embodiment, a node maps a portion of a remote memory of a remote node into its physical address space. MTMs conforming to a processor bus protocol are received by a network interface of the node. The MTMs destined for the remote node are encapsulated within network packets. Each network packet is assigned a sending priority based upon a transaction type of the encapsulated MTM and based upon ordering rules associated with the processor bus protocol. The network packets are organized into groups based upon sending priority and transmitted to the remote node via a lossy network according to the sending priorities. It is ensured that a particular subset of the network packets having a particular sending priority is received by the remote node in a proper sequence.

1 . A computer-implemented method for a local computer system to access a remote memory of a remote computer system by performing remote programmed input/output (I/O), the method comprising:
 mapping, by the local computer system, a portion of the remote memory into a physical address space of the local computer system;   receiving, by a local network interface of the local computer system coupled in communication with the remote computer system via a lossy network, a plurality of memory transaction messages (MTMs) from a local memory controller of the local computer system, wherein each MTM of the plurality of MTMs comprises a set of information and conforms to a processor bus protocol that is used by one or more local processors of the local computer system or the local memory controller to access a local memory of the local computer system;   determining, by the local network interface, that the plurality of MTMs is destined for the remote computer system;   determining, by the local network interface, for each MTM of the plurality of MTMs, a transaction type;   generating a plurality of network packets, by the local network interface, within which at least a portion of the set of information for each MTM is encapsulated;   assigning to each network packet of the plurality of network packets, by the local network interface, one of a plurality of sending priorities based upon the transaction type of the MTM of the plurality of MTMs whose said at least a portion of the set of information is encapsulated therein and based upon ordering rules associated with the processor bus protocol that defines relative priorities among MTMs exchanged between (i) one or more local processors or the local memory controller and (ii) the local memory;   organizing, by the local network interface, the plurality of network packets into groups based upon sending priority;   transmitting, by the local network interface, the plurality of network packets to the remote computer system via the lossy network according to an order determined based, at least partially, upon the sending priorities of the plurality of network packets; and   ensuring, by the local network interface, that a particular subset of the plurality of network packets having a particular sending priority of the sending priorities is received by the remote computer system in a proper sequence.   
   
   
       2 . The computer-implemented method of  claim 1 , wherein the local memory controller controls the local memory, and wherein an MTM of the plurality of MTMs comprises a request to access the remote memory which is local to and controlled by the remote computer system, or a response to an access request received from the remote computer system to access the local memory. 
   
   
       3 . The computer-implemented method of  claim 2 , wherein each MTM of the plurality of MTMs is one of the following transaction types: (a) posted request; (b) response and (c) non-posted request. 
   
   
       4 . The computer-implemented method of  claim 3 , wherein said assigning to each network packet of the plurality of network packets, by the local network interface, one of a plurality of sending priorities comprises:
 assigning a network packet of the plurality of network packets a first sending priority if the MTM that that packet is encapsulating is of the posted request type;   assigning the network packet a second sending priority, which is lower than the first sending priority, if the MTM that that packet is encapsulating is of the response type; and   assigning the network packet a third sending priority, which is lower than the second sending priority, if the MTM that that packet is encapsulating is of the non-posted request type.   
   
   
       5 . The computer-implemented method of  claim 4 , wherein said organizing, by the local network interface, the plurality of network packets into groups comprises:
 placing all of the network packets of the plurality of network packets having the first sending priority into a first queue, wherein ordering of the network packets within the first queue is determined based upon when the MTMs encapsulated by the network packets within the first queue were received by the local network interface from the local memory controller;   placing all of the network packets of the plurality of network packets having the second sending priority into a second queue, wherein ordering of the network packets within the second queue is determined based upon when the MTMs encapsulated by the network packets within the second queue were received by the local network interface from the local memory controller; and   placing all of the network packets of the plurality of network packets having the third sending priority into a third queue, wherein ordering of the network packets within the third queue is determined based upon when the MTMs encapsulated by the network packets within the third queue were received by the local network interface from the local memory controller.   
   
   
       6 . The computer-implemented method of  claim 5 , wherein said transmitting, by the local network interface, the plurality of network packets to the remote computer system via the lossy network comprises:
 transmitting all of the network packets in the first queue;   transmitting the network packets in the second queue after all of the network packets in the first queue have been transmitted; and   transmitting the network packets in the third queue after all of the network packets in the first queue and the second queue have been transmitted.   
   
   
       7 . The computer-implemented method of  claim 1 , wherein said ensuring, by the local network interface, that a particular subset of the plurality of network packets having a particular sending priority of the sending priorities are received by the remote computer system in a proper sequence comprises:
 determining whether an acknowledgement has been received from the remote computer system indicating that the remote computer system has received a particular network packet within the particular subset of the network packets; and   in response to a determination that the acknowledgement has not been received: (a) retransmitting the particular network packet to the remote computer system; and (b) retransmitting, to the remote computer system, all subsequent network packets in the particular subset of the network packets that were transmitted to the remote computer system after the particular network packet was transmitted.   
   
   
       8 . The computer-implemented method of  claim 7 , wherein the subsequent network packets are retransmitted in the same order as they were sent previously. 
   
   
       9 . The computer-implemented method of  claim 1 , further comprising:
 receiving, by the local network interface, an incoming network packet from the remote computer system, wherein the incoming packet is part of a certain sequence of incoming network packets from the remote computer system;   extracting from the incoming network packet, by the local network interface, an associated priority and a sequence number;   determining, by the local network interface, whether the sequence number matches an expected sequence number for the associated priority; and   in response to a determination that the sequence number matches the expected sequence number for the associated priority, transmitting, by the local network interface, an acknowledgement packet to the remote node.   
   
   
       10 . The computer-implemented method of  claim 9 , further comprising:
 processing, by the local network interface, the incoming network packet; and   updating, by the local network interface, the expected sequence number for the associated priority to a new expected sequence number.   
   
   
       11 . The computer-implemented method of  claim 10 , further comprising:
 in response to a determination that the sequence number does not match the expected sequence number for the associated priority, dropping, by the local network interface, the incoming network packet.   
   
   
       12 . A computer-readable storage medium tangibly embodying a set of instructions, which when executed by one or more processors of a local computer system cause the local computer system to perform a method for accessing a remote memory of a remote computer system by performing remote programmed input/output (I/O), the method comprising:
 mapping a portion of the remote memory into a physical address space of the local computer system;   receiving, by a local network interface of the local computer system coupled in communication with the remote computer system via a lossy network, a plurality of memory transaction messages (MTMs) from a local memory controller of the local computer system, wherein each MTM of the plurality of MTMs comprises a set of information and conforms to a processor bus protocol that is used by one or more local processors of the local computer system or the local memory controller to access a local memory of the local computer system;   determining the plurality of MTMs are destined for the remote computer system;   determining for each MTM of the plurality of MTMs, a transaction type;   generating a plurality of network packets within which an MTM of the plurality of MTMs encapsulated;   assigning to each network packet of the plurality of network packets one of a plurality of sending priorities based upon the transaction type of the MTM encapsulated therein and based upon ordering rules associated with the processor bus protocol that defines relative priorities among MTMs exchanged between (i) one or more local processors or the local memory controller and (ii) the local memory;   organizing the plurality of network packets into groups based upon sending priority;   transmitting the plurality of network packets to the remote computer system via the lossy network according to an order determined based, at least partially, upon the sending priorities of the plurality of network packets; and   ensuring that a particular subset of the plurality of network packets having a particular sending priority of the sending priorities are received by the remote computer system in a proper sequence.   
   
   
       13 . The computer-readable storage medium of  claim 12 , wherein the local memory controller controls the local memory, and wherein an MTM of the plurality of MTMs comprises a request to access the remote memory which is local to and controlled by the remote computer system, or a response to an access request received from the remote computer system to access the local memory. 
   
   
       14 . The computer-readable storage medium of  claim 13 , wherein each MTM of the plurality of MTMs is one of the following transaction types: (a) posted request; (b) response and (c) non-posted request. 
   
   
       15 . The computer-readable storage medium of  claim 14 , wherein said assigning to each network packet of the plurality of network packets one of a plurality of sending priorities comprises:
 assigning a network packet of the plurality of network packets a first sending priority if the MTM that that packet is encapsulating is of the posted request type;   assigning the network packet a second sending priority, which is lower than the first sending priority, if the MTM that that packet is encapsulating is of the response type; and   assigning the network packet a third sending priority, which is lower than the second sending priority, if the MTM that that packet is encapsulating is of the non-posted request type.   
   
   
       16 . The computer-readable storage medium of  claim 15 , wherein said organizing the plurality of network packets into groups comprises:
 placing all of the network packets of the plurality of network packets having the first sending priority into a first queue, wherein ordering of the network packets within the first queue is determined based upon when the MTMs encapsulated by the network packets within the first queue were received by the local network interface from the local memory controller;   placing all of the network packets of the plurality of network packets having the second sending priority into a second queue, wherein ordering of the network packets within the second queue is determined based upon when the MTMs encapsulated by the network packets within the second queue were received by the local network interface from the local memory controller; and   placing all of the network packets of the plurality of network packets having the third sending priority into a third queue, wherein ordering of the network packets within the third queue is determined based upon when the MTMs encapsulated by the network packets within the third queue were received by the local network interface from the local memory controller.   
   
   
       17 . The computer-readable storage medium of  claim 16 , wherein said transmitting the plurality of network packets to the remote computer system via the lossy network comprises:
 transmitting all of the network packets in the first queue;   transmitting the network packets in the second queue after all of the network packets in the first queue have been transmitted; and   transmitting the network packets in the third queue after all of the network packets in the first queue and the second queue have been transmitted.   
   
   
       18 . The computer-readable storage medium of  claim 12 , wherein said ensuring that a particular subset of the plurality of network packets having a particular sending priority of the sending priorities are received by the remote computer system in a proper sequence comprises:
 determining whether an acknowledgement has been received from the remote computer system indicating that the remote computer system has received a particular network packet within the particular subset of the network packets; and   in response to a determination that the acknowledgement has not been received: (a) retransmitting the particular network packet to the remote computer system; and (b) retransmitting, to the remote computer system, all subsequent network packets in the particular subset of the network packets that were transmitted to the remote computer system after the particular network packet was transmitted.   
   
   
       19 . The computer-readable storage medium of  claim 18 , wherein the subsequent network packets are retransmitted in the same order as they were sent previously. 
   
   
       20 . A network device comprising:
 a network interface operable to be coupled in communication to a remote node via a lossy network;   a local memory having stored therein one or more routines for performing a method of accessing a remote memory of the remote node by performing remote programmed input/output (I/O);   one or more processors coupled to the network interface and the storage device, operable to execute the one or more routines;   wherein the method comprises:   mapping, by the network device, a portion of a remote memory of the remote node into a physical address space of the network device;   receiving, by the network interface, a plurality of memory transaction messages (MTMs) from a local memory controller of the network device, wherein each MTM of the plurality of MTMs comprises a set of information and conforms to a processor bus protocol that is used by the one or more processors or the local memory controller to access the local memory of the network device;   determining the plurality of MTMs are destined for the remote node;   determining for each MTM of the plurality of MTMs, a transaction type;   generating a plurality of network packets within which an MTM of the plurality of MTMs is encapsulated;   assigning to each network packet of the plurality of network packets one of a plurality of sending priorities based upon the transaction type of the MTM encapsulated therein and based upon ordering rules associated with the processor bus protocol that defines relative priorities among MTMs exchanged between (i) the one or more processors or the local memory controller and (ii) the local memory;   organizing the plurality of network packets into groups based upon sending priority;   transmitting the plurality of network packets to the remote node via the lossy network according to an order determined based, at least partially, upon the sending priorities of the plurality of network packets; and   ensuring a particular subset of the plurality of network packets having a particular sending priority of the sending priorities are received by the remote node in a proper sequence.