Certain cluster conditions are easily corrected. If a cluster partition
has occurred, you can learn how to recover. This topic also tells you how
to avoid a cluster partition and gives you an example of how to merge partitions
back together.
A cluster partition occurs in a cluster whenever contact is lost between
one or more nodes in the cluster and a failure of the lost nodes cannot be
confirmed. This is not to be confused with a partition in a logical partition
(LPAR) environment.
If you receive error message CPFBB20 in either the history log (QHST) or
the QCSTCTL joblog, a cluster partition has occurred and you need to know
how to recover. The following example shows a cluster partition that involves
a cluster made up of four nodes: A, B, C, and D. The example shows a loss
of communication between cluster nodes B and C has occurred, which results
in the cluster dividing into two cluster partitions. Before the cluster partition
occurred, there were four cluster resource groups, which can be of any type,
called CRG A, CRG B, CRG C, and CRG D. The example shows the recovery domain
of each cluster resource group.
Table 1. Example of a recovery domain during a cluster partition| Node A |
Node B |
x |
Node C |
Node D |
|---|
| CRG A (backup1) |
CRG A (primary) |
|
|
|
| |
CRG B (primary) |
CRG B (backup1) |
|
| |
CRG C (primary) |
CRG C (backup1) |
CRG C (backup2) |
| CRG D (backup2) |
CRG D (primary) |
CRG D (backup1) |
|
| Partition 1 |
|
Partition 2 |
A cluster may partition if the maximum transmission unit (MTU) at any point
in the communication path is less than the cluster communications tuneable
parameter, message fragment size. MTU for a cluster IP address can be verified
using the Work with TCP/IP Network Status (WRKTCPSTS) command
on the subject node. The MTU must also be verified at each step along the
entire communication path. If the MTU is less than the message fragment size,
either raise the MTU of the path or lower the message fragment size. You can
use the Retrieve Cluster Resource Services Information (QcstRetrieveCRSInfo) API
to view the current settings of the tuning parameters and the Change
Cluster Resource Services (QcstChgClusterResourceServices) API to
change the settings.
Once the cause of the cluster partition condition has been corrected, the
cluster will detect the re-established communication link and issue the message
CPFBB21 in either the history log (QHST) or the QCSTCTL joblog. This informs
the operator that the cluster has recovered from the cluster partition. Be
aware that once the cluster partition condition has been corrected, it may
be a few minutes before the cluster merges back together.
In order to determine the types of cluster resource
group actions that you can take within a cluster partition, you need to know
whether the partition is a primary or a secondary cluster partition. When
a partition is detected, each partition is designated as a primary or secondary
partition for each cluster resource group defined in the cluster.