<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html lang="en-us" xml:lang="en-us"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="security" content="public" /> <meta name="Robots" content="index,follow" /> <meta http-equiv="PICS-Label" content='(PICS-1.1 "http://www.icra.org/ratingsv02.html" l gen true r (cz 1 lz 1 nz 1 oz 1 vz 1) "http://www.rsac.org/ratingsv01.html" l gen true r (n 0 s 0 v 0 l 0) "http://www.classify.org/safesurf/" l gen true r (SS~~000 1))' /> <meta name="DC.Type" content="concept" /> <meta name="DC.Title" content="Investigate a problem with CLUSTERINFO macro" /> <meta name="abstract" content="The CLUSTERINFO macro displays the information that cluster resource services contains about nodes, CRGs and active cluster IP addresses." /> <meta name="description" content="The CLUSTERINFO macro displays the information that cluster resource services contains about nodes, CRGs and active cluster IP addresses." /> <meta name="DC.Relation" scheme="URI" content="rzaigtroubleshoot.htm" /> <meta name="copyright" content="(C) Copyright IBM Corporation 1998, 2006" /> <meta name="DC.Rights.Owner" content="(C) Copyright IBM Corporation 1998, 2006" /> <meta name="DC.Format" content="XHTML" /> <meta name="DC.Identifier" content="rzaiginvestigateclusterinfo" /> <meta name="DC.Language" content="en-us" /> <!-- All rights reserved. Licensed Materials Property of IBM --> <!-- US Government Users Restricted Rights --> <!-- Use, duplication or disclosure restricted by --> <!-- GSA ADP Schedule Contract with IBM Corp. --> <link rel="stylesheet" type="text/css" href="./ibmdita.css" /> <link rel="stylesheet" type="text/css" href="./ic.css" /> <title>Investigate a problem with CLUSTERINFO macro</title> </head> <body id="rzaiginvestigateclusterinfo"><a name="rzaiginvestigateclusterinfo"><!-- --></a> <!-- Java sync-link --><script language="Javascript" src="../rzahg/synch.js" type="text/javascript"></script> <h1 class="topictitle1">Investigate a problem with CLUSTERINFO macro</h1> <div><p>The CLUSTERINFO macro displays the information that cluster resource services contains about nodes, CRGs and active cluster IP addresses.</p> <p>The CLUSTERINFO macro creates a snapshot of information about the current cluster. The command navigates through clustering objects and creates a description of the cluster on the local node. The CLUSTERINFO macro provides a flight recorder for different cluster objects and can aid in determining the source of a problem within the cluster. To access the CLUSTERINFO macro, complete the following steps:</p> <div class="p"><ol><li>On a character-based interface, enter <span class="cmdname">STRSST</span>.</li> <li>Sign on with your Service Tools user profile.</li> <li>On the Start Service Tool, select Option 1 (Start a service tool).</li> <li>Select Option 4 (Display/Alter/Dump).</li> <li>Select Option 2 (Dump to Printer).</li> <li>Select Option 2 (Licensed Internal Code (LIC) data).</li> <li>Select Option 14 (Advanced Analysis).</li> <li>Enter a 1 in front of CLUSTERINFO macro option. Press Enter.</li> </ol> Once the CLUSTERINFO macro displays, use the -H option to display help for all the options that are available with this macro. The following usage diagram describes each of the options that are available for the CLUSTERINFO macro:</div> <div class="p"> <div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" frame="border" border="1" rules="all"><caption>Table 1. Options for CLUSTERINFO macro</caption><thead align="left"><tr valign="bottom"><th valign="bottom" id="d0e46">Option</th> <th valign="bottom" id="d0e48">Description</th> </tr> </thead> <tbody><tr><td valign="top" headers="d0e46 ">-H</td> <td valign="top" headers="d0e48 ">Displays help screen for options</td> </tr> <tr><td valign="top" headers="d0e46 ">-A </td> <td valign="top" headers="d0e48 ">Displays all information</td> </tr> <tr><td valign="top" headers="d0e46 ">-FR </td> <td valign="top" headers="d0e48 ">Displays flight recorder entries</td> </tr> <tr><td valign="top" headers="d0e46 ">-HB</td> <td valign="top" headers="d0e48 ">Displays heartbeat information</td> </tr> <tr><td valign="top" headers="d0e46 ">-PERF</td> <td valign="top" headers="d0e48 ">Displays performance counters </td> </tr> <tr><td valign="top" headers="d0e46 "> -Q </td> <td valign="top" headers="d0e48 ">Displays outgoing message queue status</td> </tr> <tr><td valign="top" headers="d0e46 ">-G </td> <td valign="top" headers="d0e48 ">Unsuppresses display of all CC broadcast groups</td> </tr> <tr><td valign="top" headers="d0e46 ">-T </td> <td valign="top" headers="d0e48 ">Displays tuning parameters</td> </tr> <tr><td valign="top" headers="d0e46 ">-M</td> <td valign="top" headers="d0e48 ">Displays matrices for all DAs</td> </tr> <tr><td valign="top" headers="d0e46 "> -DP</td> <td valign="top" headers="d0e48 "> Display dataPort information</td> </tr> </tbody> </table> </div> </div> <p> </p> <div class="section"><h4 class="sectiontitle">Interpreting CLUSTERINFO macro results</h4><div class="p">In this example, the -A option is specified so all fields are dumped. The flight recorders are the main tools for debug. Note that these flight recorders are deleted when the cluster node is ended or deleted. For problem analysis the CLUSTERINFO macro must be run before the cluster is ended or deleted. In some cases the flight recorders may be written to a vlog when the cluster is deleted or ended. The flight recorders record various events that affect the structure and performance of the cluster. It is beyond the scope of this information to provide detailed interpretation of flight recorder data.<div class="note"><span class="notetitle">Note:</span> <img src="./delta.gif" alt="Start of change" />Vertical ellipses indicate that a portion of the results was removed and is not displayed in the output.<img src="./deltaend.gif" alt="End of change" /></div> </div> </div> <div class="section"><h4 class="sectiontitle">Section 1 of CLUSTERINFO macro results</h4><pre>DISPLAY/ALTER/DUMP CLUSTERINFO -NEW2 08/23/05 13:36:37 PAGE 1 Running macro: CLUSTERINFO -A Use -H for command information Cluster Name : MYCLUSTER Local Node Name: SYSTEM1 CC/CTS Version : 5 Macro Timestamp: 08/23/05 13:36:37.079</pre> <p>Section 1 contains generic information about the cluster, such as the cluster name, cluster version, and the timestamp when the report was generated. In this example, the cluster name is MYCLUSTER and the local node name is SYSTEM1.</p> </div> <div class="section"><h4 class="sectiontitle">Section 2 of CLUSTERINFO macro results</h4><pre>Cluster Object Addresses CstcClusterServices Address: DBF08681C9161580 Cluster Address : FC5B04B0D4001000 Cluster Task Address : B00010000E932000 Cluster Task Q Address : DBF08681C9169A00 Clue Group Services Address: CDAB6D0339001000 CC Services Address : FC5B04B0D4008000</pre> <p>Section 2 provides pointers to the locations of key cluster objects. </p> </div> <div class="section"><h4 class="sectiontitle">Section 3 of CLUSTERINFO macro results</h4><pre>Message Statistics Number of non-fragmented messages: 250 Number of fragmented messages : 1 Number of fragments : 7 Number of acks : 148</pre> <p>Section 3 contains messaging statistics for the cluster, such as the number of fragments and the number of acknowledgements (acks).</p> </div> <div class="section"><h4 class="sectiontitle">Section 4 of CLUSTERINFO macro results</h4><pre>Node Map Node ID : SYSTEM1 GenesisSubnetId : 9.5.251.0 CCNode * : FC5B04B0D4007000 CCSrvNode *: FC5B04B0D404F000 Adapter 1 : 9.5.251.46 Primary Status : 0x01 Reachable Line Type : 0x09 Ethernet Node ID : SYSTEM2 GenesisSubnetId : 9.5.251.0 CCNode * : FC5B04B0D4060000 CCSrvNode *: FC5B04B0D4061000 Adapter 1 : 9.5.251.47 Primary Status : 0x01 Reachable Line Type : 0x09 Ethernet</pre> <p>Section 4 lists all the current active cluster nodes in a node map. In this example there are two active nodes, SYSTEM1 and SYSTEM2. T</p> </div> <div class="section"><h4 class="sectiontitle">Section 5 of CLUSTERINFO macro results</h4><pre>Subnet Map Subnet ID: 127.0.0.0 CCSubnet * : FC5B04B0D4006000 CCSrvSubnet* : FC5B04B0D400C000 Retries : 0 Msg Timeouts : 0 Bad Msg Counter : 0 Failed Default Address: 0 Subnet ID: 226.5.5.5 CCSubnet * : FC5B04B0D405B100 CCSrvSubnet* : FC5B04B0D405C000 Retries : 0 Msg Timeouts : 0 Bad Msg Counter : 0 Failed Default Address: 0</pre> <p>Section 5 contains a list of all subnet objects that are in the cluster.</p> </div> <div class="section"><h4 class="sectiontitle">Section 6 of CLUSTERINFO macro results</h4><pre>Group Map Group ID: 0x0000000000000001 Name : CTS CCGroup * : FC5B04B0D405FF00 CCSrvGroup *: FC5B04B0D4064B00 Member Nodes SYSTEM1 SYSTEM2 Group ID: 0x0000000000000002 Name : CTS CCGroup * : FC5B04B0D4055100 CCSrvGroup *: FC5B04B0D4055200 Member Nodes SYSTEM1 SYSTEM2 . . .</pre> <p>Section 6 list of all the current cluster groups. Each group is called a distributed activity group. These groups are used for communication between the groups on each active node in the cluster. Majority of the groups deal are for licensed internal code (LIC). These are identifiable by a group name of CTS and BADA. You will also see a group for CCTL (QCSTCTL job in the operating system), CRGM (QCSTCRGM job in the operating system) and each cluster resource group (CRG) job. Groups for CRG jobs will not have a group name. Each group has member nodes. Member nodes are the nodes which are communicating together for this group. </p> </div> <div class="section"><h4 class="sectiontitle">Section 7 of CLUSTERINFO macro results</h4><pre>Partition Map Partition Map is empty</pre> <div class="p">Section 7 contains a list of all the nodes in the SLIC partition list. <div class="note"><span class="notetitle">Note:</span> This is not the same concept as XPF partitioned nodes.</div> </div> </div> <div class="section"><h4 class="sectiontitle">Section 8 of CLUSTERINFO macro results</h4><pre>CTS Client List CTS Client List is empty</pre> <p>Section 8 contains a list of all the registered cluster clients, such as data ports.</p> </div> <div class="section"><h4 class="sectiontitle">Section 9 of CLUSTERINFO macro results</h4><pre>Flight Recorder : CSTCSVFR Flight Recorder Address: DBF08681C9161620</pre> <p>Section 9 contains the cluster services flight recorder (CSTCSVFR) which remains on the system until an IPL. </p> </div> <div class="section"><h4 class="sectiontitle">Section 10 of CLUSTERINFO macro results</h4><pre>Message Statistics Number of non-fragmented messages: 250 Number of fragmented messages : 1 Number of fragments : 7 Number of acks : 148 Time Stamp: 08/18/05 14:00:15.329 Trace Point: 0x0010 CstcClusterServicesTracePtCreatedFlightRecorder C3D9C5C1E3C5C6D9 <CREATEFR> Time Stamp: 08/22/05 16:43:28.912 Trace Point: 0x0020 CstcClusterServicesTracePtCreatedClusterObject D4D6D9C5E8404040 4040C5F8D3770500 <MYCLUSTER E8L...> 1000 <.. > Time Stamp: 08/23/05 13:33:40.935 Trace Point: 0x0030 CstcClusterServicesTracePtDeletedClusterObject D4D6D9C5E8404040 404040E2E3 <MYCLUSTER ST > Time Stamp: 08/23/05 13:33:41.204 Trace Point: 0x0030 CstcClusterServicesTracePtDeletedClusterObject C3D4D7E3 <CMPT > Time Stamp: 08/23/05 13:33:55.122 Trace Point: 0x0020 CstcClusterServicesTracePtCreatedClusterObject D4D6D9C5E8404040 4040FC5B04B0D400 <MYCLUSTER ....M.> 1000 <.. > </pre> <p>Section 10 contains the flight recorder for CSTCCCFR. This cluster flight recorder remains on the system until clustering is ended on this node.</p> </div> <div class="section"><h4 class="sectiontitle">Section 11 of CLUSTERINFO macro results</h4><pre>Flight Recorder : CSTCCLFR Flight Recorder Address: FC5B04B0D4001E80 ------------------------------------------------------------------------------- Time Stamp: 08/23/05 13:33:54.944 Trace Point: 0x1010 CstcClusterTracePtCreatedSubnetObject 7F000000FC5B04B0 D4006000 <........M.-. > Time Stamp: 08/23/05 13:33:55.062 Trace Point: 0x1000 CstcClusterTracePtCreatedNodeObject C3E2E3D9D9C3C8C3 E2E3F5F2FC5B04B0 <CSTRSYSTEM1....> D4007000 <M... > Time Stamp: 08/23/05 13:33:55.122 Trace Point: 0x1020 CstcClusterTracePtCreatedMCGroupObject 0000000000000001 00000000D9C3C8C3 <............RCHC> E2E3F5F2 <ST52 > . . .</pre> <p>Section 11 contains cluster communication flight recorder (CSTECLFR). This cluster flight recorder remains on the system until clustering is ended on this node.</p> </div> <div class="section"><h4 class="sectiontitle">Section 12 of CLUSTERINFO macro results</h4><pre>Flight Recorder : CSTCCCFR Flight Recorder Address: FC5B04B0D4006380 ------------------------------------------------------------------------------- Time Stamp: 08/23/05 13:33:55.080 Trace Point: 0x3000 CstcCCScamTracePtScamOpen FC5B04B0D400E480 0000000000000000 <....M.U.........> Time Stamp: 08/23/05 13:33:55.097 Trace Point: 0x3010 CstcCCScamTracePtScamBind FC5B04B0D400E480 0000000000000000 <....M.U.........> Time Stamp: 08/23/05 13:33:55.100 Trace Point: 0x3000 CstcCCScamTracePtScamOpen FC5B04B0D400E480 0000000000000000 <....M.U.........> D6E4E3 <OUT > Time Stamp: 08/23/05 13:33:55.100 Trace Point: 0x3010 CstcCCScamTracePtScamBind FC5B04B0D400E480 0000000000000000 <....M.U.........> . . .</pre> <p>Section 12 contains the clue flight recorder (CSTCCCFR), which remains on the system until clustering is ended on this node.</p> </div> <div class="section"><h4 class="sectiontitle">Section 13 of CLUSTERINFO macro results</h4><pre>Time Stamp: 08/23/05 13:33:55.201 C3A2A385C7E27A7A C3A2A385C7E24082 <CsteGS::CsteGS b> 85878995A2 <egins > Time Stamp: 08/23/05 13:33:55.201 C3A2A385C4C14083 9695A2A399A483A3 <CsteDA construct> 85847A40C2C1C4C1 404040404040 <ed: BADA > Time Stamp: 08/23/05 13:33:55.201 C3A2A385C7E27A7A C3A2A385C7E24081 <CsteGS::CsteGS a> 8484408281848140 A39640C4C16D9389 <dd bada to DA_li> A2A3 <st > . . .</pre> <p>Section 13 displays the contents of the send queues and active message queues. If this section is not empty for an extended period of time it indicates a problem in the cluster. </p> </div> <div class="section"><h4 class="sectiontitle">Section 14 of CLUSTERINFO macro results</h4><pre>Flight Recorder : CSTECLF2 Flight Recorder Address: CDAB6D0339001300 ------------------------------------------------------------------------------- Time Stamp: 08/23/05 13:33:55.201 C3A2A385C4C17A7A C3A2A385C4C16B40 <CsteDA::CsteDA, > 83998581A385D4C3 C79996A49740C9C4 <createMCGroup ID> 407E40F0A7F1F5 < = 0x15 > Time Stamp: 08/23/05 13:33:55.209 C3A2A385E2C3D985 977A7A84859389A5 <CsteSCRep::deliv> 85994094A287E3A8 97857EF0A7F140A2 <er msgType=0x1 s> A482E3A897857EF0 A7F240C4C17EC2C1 <ubType=0x2 DA=BA> C4C1404040404040 <DA > Time Stamp: 08/23/05 13:33:55.209 C3A2A385C4C17A7A A58985A66B409985 <CsteDA::view, re> 9496A585D4C3C799 96A497D485948285 <moveMCGroupMembe> 99A240C9C4407E40 F0A7F1F540969384 <rs ID = 0x15 old> 6D959684856D9389 A2A340A289A98540 <_node_list size > 7E40F0A7F0 <= 0x0 > . . . </pre> <p>Section 14 contains flight recorder information.</p> </div> <div class="section"><h4 class="sectiontitle">Section 15 of CLUSTERINFO macro results</h4><pre>Message Queues Send Queues: Send Queue: 00 Messages: 00 MessageQueue*: FC5B04B0D400BF80 Send Queue: 01 Messages: 00 MessageQueue*: FC5B04B0D400DF80 Send Queue: 02 Messages: 00 MessageQueue*: FC5B04B0D400E600 Send Queue: 03 Messages: 00 MessageQueue*: FC5B04B0D400E680 Send Queue: 04 Messages: 00 MessageQueue*: FC5B04B0D400E700 Send Queue: 05 Messages: 00 MessageQueue*: FC5B04B0D400E780 Send Queue: 06 Messages: 00 MessageQueue*: FC5B04B0D400E800 Send Queue: 07 Messages: 00 MessageQueue*: FC5B04B0D400E880 Send Queue: 08 Messages: 00 MessageQueue*: FC5B04B0D400E900 Send Queue: 09 Messages: 00 MessageQueue*: FC5B04B0D400E980 Send Queue: 10 Messages: 00 MessageQueue*: FC5B04B0D400EA00 Send Queue: 11 Messages: 00 MessageQueue*: FC5B04B0D400EA80 Send Queue: 12 Messages: 00 MessageQueue*: FC5B04B0D400EB00 Send Queue: 13 Messages: 00 MessageQueue*: FC5B04B0D400EB80 Send Queue: 14 Messages: 00 MessageQueue*: FC5B04B0D400EC00 Send Queue: 15 Messages: 00 MessageQueue*: FC5B04B0D400EC80 Send Queue: 16 Messages: 00 MessageQueue*: FC5B04B0D400ED00 Send Queue: 17 Messages: 00 MessageQueue*: FC5B04B0D400ED80 Send Queue: 18 Messages: 00 MessageQueue*: FC5B04B0D400EE00 Send Queue: 19 Messages: 00 MessageQueue*: FC5B04B0D400EE80 Active Message Queues: Active Message Queue: 00 Messages: 00 MessageQueue*: FC5B04B0D4008570 Active Message Queue: 01 Messages: 00 MessageQueue*: FC5B04B0D4008640 Active Message Queue: 02 Messages: 00 MessageQueue*: FC5B04B0D4008710 Active Message Queue: 03 Messages: 00 MessageQueue*: FC5B04B0D40087E0 Active Message Queue: 04 Messages: 00 MessageQueue*: FC5B04B0D40088B0 Active Message Queue: 05 Messages: 00 MessageQueue*: FC5B04B0D4008980 Active Message Queue: 06 Messages: 00 MessageQueue*: FC5B04B0D4008A50 Active Message Queue: 07 Messages: 00 MessageQueue*: FC5B04B0D4008B20 Active Message Queue: 08 Messages: 00 MessageQueue*: FC5B04B0D4008BF0 Active Message Queue: 09 Messages: 00 MessageQueue*: FC5B04B0D4008CC0 Active Message Queue: 10 Messages: 00 MessageQueue*: FC5B04B0D4008D90 Active Message Queue: 11 Messages: 00 MessageQueue*: FC5B04B0D4008E60 Active Message Queue: 12 Messages: 00 MessageQueue*: FC5B04B0D4008F30 Active Message Queue: 13 Messages: 00 MessageQueue*: FC5B04B0D4009000 Active Message Queue: 14 Messages: 00 MessageQueue*: FC5B04B0D40090D0 Active Message Queue: 15 Messages: 00 MessageQueue*: FC5B04B0D40091A0 Active Message Queue: 16 Messages: 00 MessageQueue*: FC5B04B0D4009270 Active Message Queue: 17 Messages: 00 MessageQueue*: FC5B04B0D4009340 Active Message Queue: 18 Messages: 00 MessageQueue*: FC5B04B0D4009410 Active Message Queue: 19 Messages: 00 MessageQueue*: FC5B04B0D40094E0 ------------------------------------------------------------------------------- Tuning Values cstcRcvSendTimerRatio : 2 Default: 2 cstcMcastRelayTimerRatio : 8 Default: 8 cstcMcastRelayHBTimerRatio : 4 Default: 4 cstcHeartbeatBaseTimer : 12288000000 Default: 12288000000 cstcHeartbeatBasePrecision : 4096000000 Default: 4096000000 cstcRetryPrecision : 4096000000 Default: 4096000000 cstcRetryTimerVal : 4096000000 Default: 4096000000 cstcCDATBaseTimer : 491520000000 Default: 491520000000 cstcCDATBasePrecision : 40960000000 Default: 40960000000 cstcRecoveryBaseTimer : 3686400000000 Default: 3686400000000 cstcRecoveryBasePrecision : 491520000000 Default: 491520000000 cstcMaxRetryTime : 32768000000 Default: 32768000000 cstcCCFragmentSize : 1464 Default: 1464 cstcCCSendQOverflow : 1024 Default: 1024 cstcBadMsgCtrThreshold : 3 Default: 3 cstcUnreachableHBAckThreshold: 1 Default: 1 cstcReachableHBAckThreshold : 3 Default: 3 cstcUnreachableHBThreshold : 4 Default: 4 cstcReachableHBThreshold : 4 Default: 4 cstcMaxHBThreshold : 16 Default: 16 cstcDisableMsgTimer : 0 Default: 0 cstcRepeatAckThreshold : 10 Default: 10 DISPLAY/ALTER/DUMP CLUSTERINFO -NEW2 08/23/05 13:36:37 PAGE 87 cstcDelayedAckTimer : 409600000 Default: 409600000 cstcDelayedAckPrecision : 40960000 Default: 40960000 cstcCCSendWindow : 2 Default: 2 cstcCCEnableMcast : 1 Default: 1 cstcCCPerfClass : 2 Default: 2 ------------------------------------------------------------------------------- ****** END OF DUMP ******</pre> <p>Section 15 contains tuning values. Tuning values are equivilent to the cluster performance and configuration information that is described in the <a href="../apis/clcntrtcrs.htm">Retrieve Cluster Resource Services Information (QcstRetrieveCRSInfo) API</a>. Section 14 contains the current value and the default for these fields. </p> </div> </div> <div> <div class="familylinks"> <div class="parentlink"><strong>Parent topic:</strong> <a href="rzaigtroubleshoot.htm" title="Find error recovery solutions for problems that are specific to clusters.">Troubleshoot clusters</a></div> </div> </div> </body> </html>