Rules-Based, Mode-Driven Manager for Timer Bounded Arbitration Protocol Based Resource Control

Abstract

An example apparatus includes a processor, a memory, and an interface that connects the processor, the memory, and a set of components. The set of components includes a first component configured to acquire a mode from members of an HA cluster and a second component configured to enforce mode pairing rules for members of the HA cluster. Once the desired mode pairing has been determined, a third component takes actions configured to either achieve the mode pairing according to rules for members of the HA cluster or to selectively force a hardware reset of one or more members of the HA cluster upon determining that a split brain scenario is possible based, at least in part, on the mode of the members of the HA cluster. The example apparatus therefore implements a rules-based manager for timer bounded arbitration protocol based resource control.

1 . A non-transitory computer readable medium storing computer executable instructions that when executed by a computer in a high availability (HA) cluster controls the computer to perform a method, the method comprising:
 upon detecting an occurrence of a decision point in the lifecycle of the HA cluster, acquiring data describing an operating condition of a set of servers comprising the HA cluster, the data including a mode of a server in the HA cluster;   controlling at least one member of the HA cluster to selectively change mode to a target paired mode, where the target paired mode is selected based, at least in part, on mode pairing rules associated with the HA cluster; and   selectively forcing a hardware reset of one or more members of the HA cluster upon determining that the mode pairing rules have been violated.   
     
     
         2 . The computer readable medium of  claim 1 , where the mode is one of: a default mode, a single mode, a configuration mode, a locked mode, a peerdown mode, and a failed startup mode. 
     
     
         3 . The computer readable medium of  claim 2 , where:
 in the default mode, HA monitoring for a server is ON and SMITH (Shoot Myself In The Head) reset is enabled;   in the single mode, HA monitoring for a server is OFF and a communicating peer server is communicating and in locked mode;   in the configuration mode, HA monitoring for a server is OFF and a communicating peer server is communicating and in locked mode;   in the locked mode, a storage area network (SAN) application on the server is stopped and prevented from starting;   in the peerdown mode, a peer server is OFF; and   in the failed startup mode, attempts to start an SAN application are blocked until a failure indicator is cleared.   
     
     
         4 . The computer readable medium of  claim 1 , the data also including a status of the server in the HA cluster. 
     
     
         5 . The computer readable medium of  claim 4 , where the status is one of: unknown, stopped, running, and primary. 
     
     
         6 . The computer readable medium of  claim 5 , where:
 the unknown status is reported when the server is not communicating;   the stopped status is reported when a status command returns a first pre-determined code;   the running status is reported when a status command returns a second, different pre-determined code; and   the primary status is reported when the server status is running and the FSMPM (file system port mapper) in the server is in the primary state.   
     
     
         7 . The computer readable medium of  claim 1 , where controlling at least one member of the HA cluster to selectively change mode comprises issuing one or more of: a status, stop, start, config, clear, primary, and force reset command. 
     
     
         8 . The computer readable medium of  claim 7 , where:
 the status command causes members of the HA cluster to report their status;   the stop command causes the non-primary server in the HA cluster to be transitioned to locked mode, the primary server in the HA cluster to be transitioned to config mode, HA monitoring to be turned off on the primary server in the HA cluster, the SAN application to be stopped, and both servers in the HA cluster to be transitioned to default mode;   the start command also causes the stop command to be run, the SAN application and HA monitor to be started on the local server in the HA cluster, SMITH reset to be enabled on the local server in the HA cluster, the SAN application and HA monitor to be started on the peer server in the HA cluster, and SMITH reset to be enabled on the peer server in the HA cluster;   the configuration command causes the peer server in the HA cluster to be transitioned to the locked mode and the local server in the HA cluster to be transitioned to the configuration mode;   the clear command clears an indicator that was set by failure of a start command;   the primary command sets the status of the FSMPM on the local server in the HA cluster to primary, and   the force reset command triggers an immediate HA reset of one or more servers in the HA cluster.   
     
     
         9 . The computer readable medium of  claim 1 , the method comprising:
 storing values for the modes associated with members of the HA cluster to maintain modes through a hardware reset.   
     
     
         10 . The computer readable medium of  claim 1 , the method comprising:
 monitoring a file that indicates that a previous initialization of the HA cluster has failed, and   granting permission, prior to initialization of the HA cluster, for the HA cluster to initialize.   
     
     
         11 . The computer readable medium of  claim 1 , the method comprising controlling members of the HA cluster to be in a mode pairing selected from an allowed set of mode pairings comprising:
 default-default, default-locked, default-peerdown, single-peerdown, single-locked, config-locked, locked-default, locked-single, locked-config, and locked-locked.   
     
     
         12 . The computer readable medium of  claim 11 , the method comprising controlling members of the HA cluster to not be in a mode pairing selected from the prohibited set of paired mode states comprising:
 single-default, single-single, single-config, config-default, config-single, and config-config.   
     
     
         13 . An apparatus, comprising;
 a processor,   a memory, and   an interface that connects the processor, the memory, and a set of components, the set of components comprising:
 a first component configured to acquire a mode from a member of an HA cluster; 
 a second component configured to determine a desired mode pairing for the member of the HA cluster; and 
 a third component configured to take an action configured to either achieve the desired mode pairing for the member of the HA cluster or to selectively force a hardware reset of the member of the HA cluster upon determining that a split brain scenario is possible based, at least in part, on the mode of the member of the HA cluster. 
   
     
     
         14 . The apparatus of  claim 13 , the mode being one of:
 a default mode, a single mode, a configuration mode, a locked mode, a peerdown mode, and a failed startup mode, where:   in the default mode, HA monitoring for a member of the HA cluster is ON and SMITH reset is enabled;   in the single mode, HA monitoring for a member of the HA cluster is OFF and a communicating peer member of the HA cluster is communicating and in locked mode or not communicating and in peerdown mode;   in the configuration mode, HA monitoring for a member of the HA cluster is OFF and a communicating peer member of the HA cluster is communicating and in locked mode or not communicating and in peerdown mode;   in the locked mode, an SAN application on the member of the HA cluster is stopped and prevented from starting;   in the peerdown mode, a peer member of the HA cluster is OFF; and   in the failed startup mode, attempts to start the SAN application are blocked until a failure indicator is cleared.   
     
     
         15 . The apparatus of  claim 14 , the first component being configured to acquire a status from the member of the HA cluster, the status being one of:
 unknown, stopped, running, and primary, and where:   the unknown status is reported when the server is not communicating;   the stopped status is reported when a status command returns a first pre-determined code;   the running status is reported when a status command returns a second, different pre-determined code; and   the primary status is reported when the member of the HA cluster status is running and the FSMPM in the member of the HA cluster is in the primary state.   
     
     
         16 . The apparatus of  claim 15 , where the action performed by the third component is one of:
 status, stop, start, configuration, clear, primary, and force reset, and where:   the status action causes a cluster member to report status;   the stop action causes the non-primary member of the HA cluster to be transitioned to locked mode, the primary member of the HA cluster to be transitioned to configuration mode, HA monitoring to be turned off on the primary member of the HA cluster, the SAN application to be stopped, and both members of the HA cluster to be transitioned to default mode;   the start action also causes the stop command to be run, the SAN application and HA monitor to be started on the local member of the HA cluster, SMITH reset to be enabled on the local member of the HA cluster, the SAN application and HA monitor to be started on the peer member of the HA cluster, and SMITH reset to be enabled on the peer member of the HA cluster;   the configuration action causes the peer member of the HA cluster to be transitioned to the locked mode and the local member of the HA cluster to be transitioned to the configuration mode;   the clear action clears an indicator that was set by failure of a start command;   the primary action sets the status of the FSMPM on the local member of the HA cluster to primary, and   the force reset action triggers an immediate HA reset.   
     
     
         17 . The apparatus of  claim 16 , the desired mode pairings comprising:
 default-default, default-locked, default-peerdown, single-peerdown, single-locked, config-locked, locked-default, locked-single, locked-config, and locked-locked.   
     
     
         18 . The apparatus of  claim 17 , where prohibited mode pairings comprise:
 single-default, single-single, single-config, config-default, config-single, and config-config.   
     
     
         19 . The apparatus of  claim 18 , the third component being configured to force a hardware reset upon determining that the HA cluster is in a prohibited paired mode and is in danger of an SBS. 
     
     
         20 . A high availability (HA) cluster manager apparatus, comprising:
 a logic configured to acquire a substantially instantaneous state of an HA cluster, the state comprising at least a mode and status for members of the HA cluster;   a mode rules logic configured to select a mode for a member of the HA cluster, the mode being selected to make the HA cluster comply with a set of allowed mode pairings and to prevent the HA cluster from attaining a prohibited mode pairing, and   an action logic configured to prevent a split brain scenario in the HA cluster by transforming an HA cluster member state by performing one or more of, a status action, a stop action, a start action, a configuration action, a clear action, a primary action, and a force hardware reset action.