Usually, a failover results from a node failure, but there are
other reasons that can also generate a failover.
It is possible for a problem to affect only a single cluster resource group
that can cause a failover for that cluster resource group (CRG) but not for
any other CRG.
The following table shows various types of failures and the category they
fit in:
| Failure |
General category |
|---|
| CEC hardware failure (CPU, for example) |
2 |
| Communications adapter, line, or router failure; or ENDTCPIFC affecting
all IP interface addresses defined for the node |
4 |
| Power loss to the CEC |
1 |
| Operating system software machine check |
2 |
| ENDTCP(*IMMED or *CNTRLD with a time limit) is issued |
1 |
| ENDSBS QSYSWRK(*IMMED or *CNTRLD) is issued |
1 |
| ENDSBS(*ALL, *IMMED, or *CNTRLD) is issued |
1 |
| ENDSYS (*IMMED or *CNTRLD) is issued |
1 |
| PWRDWNSYS(*IMMED or *CNTRLD) is issued |
1 |
| Initial program load (IPL) button is pushed while cluster resource
services is active on system |
1 |
| Cancel Job (*IMMED or *CNTRLD with a time limit) of the QCSTCTL job
is issued |
1 |
| Cancel Job (*IMMED or *CNTRLD with a time limit) of the QCSTCRGM job
is issued |
1 |
| Cancel Job (*IMMED or *CNTRLD with a time limit) of a cluster resource
group job is issued |
3 |
| End Cluster Node API is called |
1 |
| Remove Cluster Node API is called |
1 |
| Cluster resource group job has a software error that causes it to end
abnormally |
3 |
| Enters function 8 or function 3 from control panel to power down system |
2 |
| Enters function 7 for a delayed power down of a partition |
1 |
| Application program failure for an application cluster resource group |
3 |
General category:
- All of cluster resource services (CRS) fails on a node and is detected
as a node failure. The node may actually be operational or the node may have
failed (for example, a system failure due to power loss). If all of cluster
resource services fails, then the resources that are managed by CRS will go
through the failover process.
- All of CRS fails on a node but it is detected as a cluster partition.
The node may or may not be operational.
- A failure occurs on an individual cluster resource group. These conditions
are always detected as a failure.
- A failure occurs but the node and cluster resource services are still
operational and it is detected as a cluster partition.
|
When a failure occurs, the action taken by cluster resource services for
a specific cluster resource group depends on the type of failure and the state
of the cluster resource group. However, in all cases, the exit program is
called. A failover may must work with a list of failed nodes. When the exit
program is called, it needs to determine if it must deal with only a single
node failure or with a list of failed nodes.
If the cluster resource group is inactive, the membership
status of the failed node in the cluster resource group's recovery domain
is changed to either an Inactive or Partition status.
However, the node roles are not changed, and the backup nodes are not reordered.
The backup nodes are reordered in an inactive cluster resource group when
the Start
Cluster Resource Group (STRCRG) command or the Start Cluster Resource
Group (QcstStartClusterResourceGroup) API is called. But,
the Start Cluster Resource Group API will fail if the primary node is not
active. You must issue the Change
Cluster Resource Group (CHGCRG) command or the Change Cluster Resource
Group (QcstChangeClusterResourceGroup) API to designate
an active node as the primary node, then call the Start Cluster Resource Group
API again.
If the cluster resource group is active and the failing
node is not the primary node, the failover updates the
status of the failed recovery domain member in the cluster resource group's
recovery domain. If the failing node is a backup node, the list of backup
nodes is reordered so that active nodes are at the beginning of the list.
If the cluster resource group is active and the recovery
domain member is the primary node, the following actions are performed based
on which type of failure has occurred:
- Category 1 Failure
- Failover occurs. The primary node is marked inactive in
each cluster resource group and made the last backup node. The node that was
the first backup becomes the new primary node. All device cluster resource
groups failover first. Then, all data cluster resource groups failover. Finally,
all application cluster resource groups failover. If a failover for any CRG
detects that none of the backup nodes are active, the status of the CRG is
set to indoubt.
- Category 2 Failure
- Failover occurs but the primary node does not change. All nodes in the
cluster partition that do not have the primary node as a member of the partition
will end the active cluster resource group. The status of the nodes in the
recovery domain in the cluster resource group are set to a partition status
for each node that is in the primary partition. If a node really failed but
is detected only as a partition problem and the failed node was the primary
node, you lose all the data and application services on that node and no automatic
failover is started. You must either declare the node as failed or bring the
node back up and start clustering on that node again. See Change partitioned
nodes to failed for more information.
- Category 3 Failure
- If only a single cluster resource group is affected, failover occurs on
an individual basis because cluster resource groups are independent of each
other. It may happen that several cluster resource groups are affected at
the same time due to someone canceling several cluster resource jobs. However,
the type of failure is handled on a CRG by CRG basis, and no coordinated failover
between CRGs is performed. The primary node is marked as inactive in
each cluster resource group and made the last backup node. The node that was
the first backup node becomes the new primary node. If there is no active
backup node, the status of the cluster resource group is set to indoubt.
- Category 4 Failure
- This category is similar to category 2. However, while all nodes and cluster
resource services on the nodes are still operational, not all nodes can communicate
with each other. The cluster is partitioned, but the primary node or nodes
are still providing services. However, because of the partition, you may experience
various problems. For example, if the primary node is in one partition and
all the backup nodes or replicate nodes are in another partition, you are
no longer replicating data and have no protection should the primary node
fail. In the partition that contains the primary node, the failover process
updates the status of the nodes in the cluster resource group's recovery domain
to partition for all nodes in the other partition. In the
partition that does not contain the primary node, the status of the nodes
in the cluster resource group's recovery domain for all nodes in the other
partition is set to partition.
