Investigate a problem with CLUSTERINFO macro

The CLUSTERINFO macro displays the information that cluster resource services contains about nodes, CRGs and active cluster IP addresses.

The CLUSTERINFO macro creates a snapshot of information about the current cluster. The command navigates through clustering objects and creates a description of the cluster on the local node. The CLUSTERINFO macro provides a flight recorder for different cluster objects and can aid in determining the source of a problem within the cluster. To access the CLUSTERINFO macro, complete the following steps:

  1. On a character-based interface, enter STRSST.
  2. Sign on with your Service Tools user profile.
  3. On the Start Service Tool, select Option 1 (Start a service tool).
  4. Select Option 4 (Display/Alter/Dump).
  5. Select Option 2 (Dump to Printer).
  6. Select Option 2 (Licensed Internal Code (LIC) data).
  7. Select Option 14 (Advanced Analysis).
  8. Enter a 1 in front of CLUSTERINFO macro option. Press Enter.
Once the CLUSTERINFO macro displays, use the -H option to display help for all the options that are available with this macro. The following usage diagram describes each of the options that are available for the CLUSTERINFO macro:
Table 1. Options for CLUSTERINFO macro
Option Description
-H Displays help screen for options
-A Displays all information
-FR Displays flight recorder entries
-HB Displays heartbeat information
-PERF Displays performance counters
-Q Displays outgoing message queue status
-G Unsuppresses display of all CC broadcast groups
-T Displays tuning parameters
-M Displays matrices for all DAs
-DP Display dataPort information

Interpreting CLUSTERINFO macro results

In this example, the -A option is specified so all fields are dumped. The flight recorders are the main tools for debug. Note that these flight recorders are deleted when the cluster node is ended or deleted. For problem analysis the CLUSTERINFO macro must be run before the cluster is ended or deleted. In some cases the flight recorders may be written to a vlog when the cluster is deleted or ended. The flight recorders record various events that affect the structure and performance of the cluster. It is beyond the scope of this information to provide detailed interpretation of flight recorder data.
Note: Start of changeVertical ellipses indicate that a portion of the results was removed and is not displayed in the output.End of change

Section 1 of CLUSTERINFO macro results

DISPLAY/ALTER/DUMP    CLUSTERINFO -NEW2     08/23/05  13:36:37   PAGE      1
Running macro: CLUSTERINFO               -A
Use -H for command information
Cluster Name   : MYCLUSTER
Local Node Name: SYSTEM1
CC/CTS Version : 5
Macro Timestamp: 08/23/05 13:36:37.079

Section 1 contains generic information about the cluster, such as the cluster name, cluster version, and the timestamp when the report was generated. In this example, the cluster name is MYCLUSTER and the local node name is SYSTEM1.

Section 2 of CLUSTERINFO macro results

Cluster Object Addresses
CstcClusterServices Address: DBF08681C9161580
Cluster Address            : FC5B04B0D4001000
Cluster Task Address       : B00010000E932000
Cluster Task Q Address     : DBF08681C9169A00
Clue Group Services Address: CDAB6D0339001000
CC Services Address        : FC5B04B0D4008000

Section 2 provides pointers to the locations of key cluster objects.

Section 3 of CLUSTERINFO macro results

Message Statistics
Number of non-fragmented messages: 250
Number of fragmented messages    : 1
Number of fragments              : 7
Number of acks                   : 148

Section 3 contains messaging statistics for the cluster, such as the number of fragments and the number of acknowledgements (acks).

Section 4 of CLUSTERINFO macro results

Node Map
Node ID : SYSTEM1
 GenesisSubnetId : 9.5.251.0
 CCNode *   : FC5B04B0D4007000
 CCSrvNode *: FC5B04B0D404F000
 Adapter 1  : 9.5.251.46 Primary
  Status    : 0x01 Reachable
  Line Type : 0x09 Ethernet
Node ID : SYSTEM2
 GenesisSubnetId : 9.5.251.0
 CCNode *   : FC5B04B0D4060000
 CCSrvNode *: FC5B04B0D4061000
 Adapter 1  : 9.5.251.47 Primary
  Status    : 0x01 Reachable
  Line Type : 0x09 Ethernet

Section 4 lists all the current active cluster nodes in a node map. In this example there are two active nodes, SYSTEM1 and SYSTEM2. T

Section 5 of CLUSTERINFO macro results

Subnet Map
Subnet ID: 127.0.0.0
 CCSubnet *            : FC5B04B0D4006000
 CCSrvSubnet*          : FC5B04B0D400C000
 Retries               : 0
 Msg Timeouts          : 0
 Bad Msg Counter       : 0
 Failed Default Address: 0
Subnet ID: 226.5.5.5
 CCSubnet *            : FC5B04B0D405B100
 CCSrvSubnet*          : FC5B04B0D405C000
 Retries               : 0
 Msg Timeouts          : 0
 Bad Msg Counter       : 0
 Failed Default Address: 0

Section 5 contains a list of all subnet objects that are in the cluster.

Section 6 of CLUSTERINFO macro results

Group Map
Group ID: 0x0000000000000001
 Name  : CTS
 CCGroup *   : FC5B04B0D405FF00
 CCSrvGroup *: FC5B04B0D4064B00
 Member Nodes
  SYSTEM1
  SYSTEM2
Group ID: 0x0000000000000002
 Name  : CTS
 CCGroup *   : FC5B04B0D4055100
 CCSrvGroup *: FC5B04B0D4055200
 Member Nodes
  SYSTEM1
  SYSTEM2
.
.
.

Section 6 list of all the current cluster groups. Each group is called a distributed activity group. These groups are used for communication between the groups on each active node in the cluster. Majority of the groups deal are for licensed internal code (LIC). These are identifiable by a group name of CTS and BADA. You will also see a group for CCTL (QCSTCTL job in the operating system), CRGM (QCSTCRGM job in the operating system) and each cluster resource group (CRG) job. Groups for CRG jobs will not have a group name. Each group has member nodes. Member nodes are the nodes which are communicating together for this group.

Section 7 of CLUSTERINFO macro results

Partition Map
Partition Map is empty
Section 7 contains a list of all the nodes in the SLIC partition list.
Note: This is not the same concept as XPF partitioned nodes.

Section 8 of CLUSTERINFO macro results

CTS Client List
CTS Client List is empty

Section 8 contains a list of all the registered cluster clients, such as data ports.

Section 9 of CLUSTERINFO macro results

Flight Recorder        : CSTCSVFR
Flight Recorder Address: DBF08681C9161620

Section 9 contains the cluster services flight recorder (CSTCSVFR) which remains on the system until an IPL.

Section 10 of CLUSTERINFO macro results

Message Statistics
Number of non-fragmented messages: 250
Number of fragmented messages    : 1
Number of fragments              : 7
Number of acks                   : 148
Time Stamp: 08/18/05 14:00:15.329
Trace Point: 0x0010 CstcClusterServicesTracePtCreatedFlightRecorder
C3D9C5C1E3C5C6D9  							 <CREATEFR>
Time Stamp: 08/22/05 16:43:28.912
Trace Point: 0x0020 CstcClusterServicesTracePtCreatedClusterObject
D4D6D9C5E8404040 4040C5F8D3770500	 <MYCLUSTER 	 E8L...>
1000														 <..								>
Time Stamp: 08/23/05 13:33:40.935
Trace Point: 0x0030 CstcClusterServicesTracePtDeletedClusterObject
D4D6D9C5E8404040 404040E2E3       <MYCLUSTER      ST 	>
Time Stamp: 08/23/05 13:33:41.204
Trace Point: 0x0030 CstcClusterServicesTracePtDeletedClusterObject
C3D4D7E3                          <CMPT         			>
Time Stamp: 08/23/05 13:33:55.122
Trace Point: 0x0020 CstcClusterServicesTracePtCreatedClusterObject
D4D6D9C5E8404040 4040FC5B04B0D400 <MYCLUSTER   ....M.>
1000														<..							  >

	                

Section 10 contains the flight recorder for CSTCCCFR. This cluster flight recorder remains on the system until clustering is ended on this node.

Section 11 of CLUSTERINFO macro results

Flight Recorder        : CSTCCLFR
Flight Recorder Address: FC5B04B0D4001E80
-------------------------------------------------------------------------------
Time Stamp: 08/23/05 13:33:54.944
Trace Point: 0x1010 CstcClusterTracePtCreatedSubnetObject
7F000000FC5B04B0 D4006000         <........M.-.    >
Time Stamp: 08/23/05 13:33:55.062
Trace Point: 0x1000 CstcClusterTracePtCreatedNodeObject
C3E2E3D9D9C3C8C3 E2E3F5F2FC5B04B0 <CSTRSYSTEM1....>
D4007000                          <M...            >
Time Stamp: 08/23/05 13:33:55.122
Trace Point: 0x1020 CstcClusterTracePtCreatedMCGroupObject
0000000000000001 00000000D9C3C8C3 <............RCHC>
E2E3F5F2                          <ST52            >
.
.
.

Section 11 contains cluster communication flight recorder (CSTECLFR). This cluster flight recorder remains on the system until clustering is ended on this node.

Section 12 of CLUSTERINFO macro results

Flight Recorder        : CSTCCCFR
Flight Recorder Address: FC5B04B0D4006380
-------------------------------------------------------------------------------
Time Stamp: 08/23/05 13:33:55.080
Trace Point: 0x3000 CstcCCScamTracePtScamOpen
FC5B04B0D400E480 0000000000000000 <....M.U.........>
Time Stamp: 08/23/05 13:33:55.097
Trace Point: 0x3010 CstcCCScamTracePtScamBind
FC5B04B0D400E480 0000000000000000 <....M.U.........>
Time Stamp: 08/23/05 13:33:55.100
Trace Point: 0x3000 CstcCCScamTracePtScamOpen
FC5B04B0D400E480 0000000000000000 <....M.U.........>
D6E4E3                            <OUT             >
Time Stamp: 08/23/05 13:33:55.100
Trace Point: 0x3010 CstcCCScamTracePtScamBind
FC5B04B0D400E480 0000000000000000 <....M.U.........>
.
.
.

Section 12 contains the clue flight recorder (CSTCCCFR), which remains on the system until clustering is ended on this node.

Section 13 of CLUSTERINFO macro results

Time Stamp: 08/23/05 13:33:55.201
C3A2A385C7E27A7A C3A2A385C7E24082 <CsteGS::CsteGS b>
85878995A2                        <egins           >
Time Stamp: 08/23/05 13:33:55.201
C3A2A385C4C14083 9695A2A399A483A3 <CsteDA construct>
85847A40C2C1C4C1 404040404040     <ed: BADA        >
Time Stamp: 08/23/05 13:33:55.201
C3A2A385C7E27A7A C3A2A385C7E24081 <CsteGS::CsteGS a>
8484408281848140 A39640C4C16D9389 <dd bada to DA_li>
A2A3                              <st              >
.
.
.

Section 13 displays the contents of the send queues and active message queues. If this section is not empty for an extended period of time it indicates a problem in the cluster.

Section 14 of CLUSTERINFO macro results

Flight Recorder        : CSTECLF2
Flight Recorder Address: CDAB6D0339001300
-------------------------------------------------------------------------------
Time Stamp: 08/23/05 13:33:55.201
C3A2A385C4C17A7A C3A2A385C4C16B40 <CsteDA::CsteDA, >
83998581A385D4C3 C79996A49740C9C4 <createMCGroup ID>
407E40F0A7F1F5                    < = 0x15         >
Time Stamp: 08/23/05 13:33:55.209
C3A2A385E2C3D985 977A7A84859389A5 <CsteSCRep::deliv>
85994094A287E3A8 97857EF0A7F140A2 <er msgType=0x1 s>
A482E3A897857EF0 A7F240C4C17EC2C1 <ubType=0x2 DA=BA>
C4C1404040404040                  <DA              >
Time Stamp: 08/23/05 13:33:55.209
C3A2A385C4C17A7A A58985A66B409985 <CsteDA::view, re>
9496A585D4C3C799 96A497D485948285 <moveMCGroupMembe>
99A240C9C4407E40 F0A7F1F540969384 <rs ID = 0x15 old>
6D959684856D9389 A2A340A289A98540 <_node_list size >
7E40F0A7F0                        <= 0x0           >
.
.
.

Section 14 contains flight recorder information.

Section 15 of CLUSTERINFO macro results

Message Queues
Send Queues:
Send Queue: 00 Messages: 00 MessageQueue*: FC5B04B0D400BF80
Send Queue: 01 Messages: 00 MessageQueue*: FC5B04B0D400DF80
Send Queue: 02 Messages: 00 MessageQueue*: FC5B04B0D400E600
Send Queue: 03 Messages: 00 MessageQueue*: FC5B04B0D400E680
Send Queue: 04 Messages: 00 MessageQueue*: FC5B04B0D400E700
Send Queue: 05 Messages: 00 MessageQueue*: FC5B04B0D400E780
Send Queue: 06 Messages: 00 MessageQueue*: FC5B04B0D400E800
Send Queue: 07 Messages: 00 MessageQueue*: FC5B04B0D400E880
Send Queue: 08 Messages: 00 MessageQueue*: FC5B04B0D400E900
Send Queue: 09 Messages: 00 MessageQueue*: FC5B04B0D400E980
Send Queue: 10 Messages: 00 MessageQueue*: FC5B04B0D400EA00
Send Queue: 11 Messages: 00 MessageQueue*: FC5B04B0D400EA80
Send Queue: 12 Messages: 00 MessageQueue*: FC5B04B0D400EB00
Send Queue: 13 Messages: 00 MessageQueue*: FC5B04B0D400EB80
Send Queue: 14 Messages: 00 MessageQueue*: FC5B04B0D400EC00
Send Queue: 15 Messages: 00 MessageQueue*: FC5B04B0D400EC80
Send Queue: 16 Messages: 00 MessageQueue*: FC5B04B0D400ED00
Send Queue: 17 Messages: 00 MessageQueue*: FC5B04B0D400ED80
Send Queue: 18 Messages: 00 MessageQueue*: FC5B04B0D400EE00
Send Queue: 19 Messages: 00 MessageQueue*: FC5B04B0D400EE80
Active Message Queues:
Active Message Queue: 00 Messages: 00 MessageQueue*: FC5B04B0D4008570
Active Message Queue: 01 Messages: 00 MessageQueue*: FC5B04B0D4008640
Active Message Queue: 02 Messages: 00 MessageQueue*: FC5B04B0D4008710
Active Message Queue: 03 Messages: 00 MessageQueue*: FC5B04B0D40087E0
Active Message Queue: 04 Messages: 00 MessageQueue*: FC5B04B0D40088B0
Active Message Queue: 05 Messages: 00 MessageQueue*: FC5B04B0D4008980
Active Message Queue: 06 Messages: 00 MessageQueue*: FC5B04B0D4008A50
Active Message Queue: 07 Messages: 00 MessageQueue*: FC5B04B0D4008B20
Active Message Queue: 08 Messages: 00 MessageQueue*: FC5B04B0D4008BF0
Active Message Queue: 09 Messages: 00 MessageQueue*: FC5B04B0D4008CC0
Active Message Queue: 10 Messages: 00 MessageQueue*: FC5B04B0D4008D90
Active Message Queue: 11 Messages: 00 MessageQueue*: FC5B04B0D4008E60
Active Message Queue: 12 Messages: 00 MessageQueue*: FC5B04B0D4008F30
Active Message Queue: 13 Messages: 00 MessageQueue*: FC5B04B0D4009000
Active Message Queue: 14 Messages: 00 MessageQueue*: FC5B04B0D40090D0
Active Message Queue: 15 Messages: 00 MessageQueue*: FC5B04B0D40091A0
Active Message Queue: 16 Messages: 00 MessageQueue*: FC5B04B0D4009270
Active Message Queue: 17 Messages: 00 MessageQueue*: FC5B04B0D4009340
Active Message Queue: 18 Messages: 00 MessageQueue*: FC5B04B0D4009410
Active Message Queue: 19 Messages: 00 MessageQueue*: FC5B04B0D40094E0
-------------------------------------------------------------------------------
Tuning Values
cstcRcvSendTimerRatio        : 2 Default: 2
cstcMcastRelayTimerRatio     : 8 Default: 8
cstcMcastRelayHBTimerRatio   : 4 Default: 4
cstcHeartbeatBaseTimer       : 12288000000 Default: 12288000000
cstcHeartbeatBasePrecision   : 4096000000 Default: 4096000000
cstcRetryPrecision           : 4096000000 Default: 4096000000
cstcRetryTimerVal            : 4096000000 Default: 4096000000
cstcCDATBaseTimer            : 491520000000 Default: 491520000000
cstcCDATBasePrecision        : 40960000000 Default: 40960000000
cstcRecoveryBaseTimer        : 3686400000000 Default: 3686400000000
cstcRecoveryBasePrecision    : 491520000000 Default: 491520000000
cstcMaxRetryTime             : 32768000000 Default: 32768000000
cstcCCFragmentSize           : 1464 Default: 1464
cstcCCSendQOverflow          : 1024 Default: 1024
cstcBadMsgCtrThreshold       : 3 Default: 3
cstcUnreachableHBAckThreshold: 1 Default: 1
cstcReachableHBAckThreshold  : 3 Default: 3
cstcUnreachableHBThreshold   : 4 Default: 4
cstcReachableHBThreshold     : 4 Default: 4
cstcMaxHBThreshold           : 16 Default: 16
cstcDisableMsgTimer          : 0 Default: 0
cstcRepeatAckThreshold       : 10 Default: 10
     DISPLAY/ALTER/DUMP    CLUSTERINFO -NEW2   08/23/05  13:36:37   PAGE     87
cstcDelayedAckTimer          : 409600000 Default: 409600000
cstcDelayedAckPrecision      : 40960000 Default: 40960000
cstcCCSendWindow             : 2 Default: 2
cstcCCEnableMcast            : 1 Default: 1
cstcCCPerfClass              : 2 Default: 2
-------------------------------------------------------------------------------
****** END OF DUMP ******

Section 15 contains tuning values. Tuning values are equivilent to the cluster performance and configuration information that is described in the Retrieve Cluster Resource Services Information (QcstRetrieveCRSInfo) API. Section 14 contains the current value and the default for these fields.