Cisco MDS SanOS Troubleshooting Guide -- Overview

From DocWiki

(Difference between revisions)
Jump to: navigation, search
 
(24 intermediate revisions not shown)
Line 5: Line 5:
<br>
<br>
 +
 +
== Generating a PDF of this section==
 +
 +
[[Cisco MDS SAN-OS Troubleshooting Guide -- Overview PDF]]
 +
-----
 +
== Overview of the Troubleshooting Process ==
== Overview of the Troubleshooting Process ==
Line 14: Line 20:
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
-
To identify the possible problems, you need to use a variety of tools and understand the overall storage environment. For this reason, this guide describes a number of general troubleshooting tools in Appendix B, "'''Troubleshooting Tools and Methodology'''," including those that are specific to the Cisco MDS 9000 Family. This chapter also provides a plan for investigating storage issues. See other chapters in this book for detailed explanations of specific issues.  
+
To identify the possible problems, you need to use a variety of tools and understand the overall storage environment. For this reason, this guide describes a number of general troubleshooting tools in Appendix B, "Troubleshooting Tools and Methodology"including those that are specific to the Cisco MDS 9000 Family. This chapter also provides a plan for investigating storage issues. See other chapters in this book for detailed explanations of specific issues.
==Best Practices==
==Best Practices==
Line 51: Line 57:
===Gathering Information Using Common Fabric Manager Tools and CLI Commands===
===Gathering Information Using Common Fabric Manager Tools and CLI Commands===
-
This section highlights the Fabric Manager tools and CLI commands that are commonly used to troubleshoot problems within your fabric. These tools and commands are a subset of what you may use to troubleshoot your specific problem.  
+
This section highlights the Fabric Manager tools and CLI commands that are commonly used to troubleshoot problems within your fabric. These tools and commands are a subset of what you might use to troubleshoot your specific problem.  
Each section in this guide may include additional tools and commands specific to the symptoms and possible problems covered in that chapter.  
Each section in this guide may include additional tools and commands specific to the symptoms and possible problems covered in that chapter.  
Line 57: Line 63:
'''Common Fabric Manager Tools''' <br>
'''Common Fabric Manager Tools''' <br>
Use the following navigation paths in Fabric Manager or Device Manager to access common troubleshooting information:  
Use the following navigation paths in Fabric Manager or Device Manager to access common troubleshooting information:  
-
* Overview of switch status—In Fabric Manager, click the Switch Health Analysis icon.  
+
* Overview of switch status — In '''Fabric Manager''', click the '''Switch Health Analysis''' icon.  
-
* End-to-end connectivity—In Fabric Manager, click the End-to-End Connectivity Analysis icon.  
+
* End-to-end connectivity — In '''Fabric Manager''', click the '''End-to-End Connectivity Analysis''' icon.  
-
* Fabric configuration— In Fabric Manager, click the Fabric Configuration Analysis icon.  
+
* Fabric configuration — In '''Fabric Manager''', click the '''Fabric Configuration Analysis''' icon.  
-
* Module status—In Device Manager, choose Physical > Modules.  
+
* Module status — In '''Device Manager''', choose '''Physical''' > '''Modules'''.  
-
* Cisco SAN-OS version—In Device Manager, choose Physical > System.  
+
* Cisco SAN-OS version — In '''Device Manager''', choose '''Physical''' > '''System'''.  
-
* View logs—In Device Manager, choose Logs > FM Server or Logs > Switch Resident.  
+
* View logs — In '''Device Manager''', choose '''Logs''' > '''FM Server or Logs''' > '''Switch Resident'''.  
-
* View Fabric Manager events—In Fabric Manager, click the Events tab at the bottom of the fabric pane.  
+
* View Fabric Manager events — In '''Fabric Manager''', click the '''Events''' tab at the bottom of the fabric pane.  
-
* Interface status—In Fabric Manager, choose Switches > Interfaces and select the port type you are interested in.  
+
* Interface status — In '''Fabric Manager''', choose '''Switches''' > '''Interfaces''' and select the port type you are interested in.  
-
* View name server information— In Device Manager, choose FC > Name Server.  
+
* View name server information — In '''Device Manager''', choose '''FC''' > '''Name Server'''.  
-
* View FLOGI information—In Fabric Manager, choose Switches > Interfaces > FC Physical > FLOGI.  
+
* View FLOGI information — In '''Fabric Manager''', choose '''Switches''' > '''Interfaces''' > '''FC Physical''' > '''FLOGI'''.  
-
* Analyze the results of merging zones—In Fabric Manager, choose Zone > Merge Analysis.  
+
* Analyze the results of merging zones — In '''Fabric Manager''', choose '''Zone''' > '''Merge Analysis'''.  
-
* View traffic statistics—In Device Manager, click the Device tab, select one or more ports, right-click and choose Monitoring from the pop-up menu.  
+
* View traffic statistics — In '''Device Manager''', click the '''Device''' tab, select one or more ports, right-click the selected port, and choose '''Monitoring''' from the pop-up menu.  
Fabric Manager and Device Manager also provide the following tools to proactively monitor your fabric:  
Fabric Manager and Device Manager also provide the following tools to proactively monitor your fabric:  
-
* ISL performance—In Fabric Manager, click the ISL Performance icon.  
+
* ISL performance — In '''Fabric Manager''', click the '''ISL Performance''' icon.  
-
* Network monitoring—In Device Manager, click the Summary tab.  
+
* Network monitoring — In '''Device Manager''', click the '''Summary''' tab.  
-
* Performance monitoring—In Fabric Manager, choose Performance > Start Collection.  
+
* Performance monitoring — In '''Fabric Manager''', choose '''Performance''' > '''Start Collection'''.  
'''Common CLI Commands''' <br>
'''Common CLI Commands''' <br>
-
Issue the following commands and examine the outputs:  
+
Enter the following commands and examine the outputs:  
* show module  
* show module  
* show version  
* show version  
Line 92: Line 98:
----
----
----
----
-
{{Note| To issue commands with the '''internal''' keyword, you must have an account that is a member of the network-admin group.}}
+
{{Note| To use commands with the '''internal''' keyword, you must have an account that is a member of the network-admin group.}}
----
----
Line 99: Line 105:
* Are you using the correct fiber (SM or MM)?  
* Are you using the correct fiber (SM or MM)?  
* Did you check for a broken fiber?  
* Did you check for a broken fiber?  
-
* Is the Fibre Channel port LED on the connected module green, and do the LEDs on any host bus adapter (HBA)/storage subsystem ports indicate normal functionality?  
+
* Is the Fibre Channel port LED on the connected module green, and do the LEDs on any host bus adapter (HBA) or storage subsystem ports indicate normal functionality?  
* Is there a LUN masking policy applied on the storage subsystem? If yes, is the server allowed to see the LUNs exported by the storage array?  
* Is there a LUN masking policy applied on the storage subsystem? If yes, is the server allowed to see the LUNs exported by the storage array?  
* Is there a LUN masking policy configured on the host? Did you enable the server to see all the LUNs it can access?  
* Is there a LUN masking policy configured on the host? Did you enable the server to see all the LUNs it can access?  
Line 105: Line 111:
* Is the subsystem configured for an N port?  
* Is the subsystem configured for an N port?  
-
Examine the FLOGI database on the two switches that are directly connected to the host HBA and subsystem ports. Also, verify that both ports (attached to MDS-A and MDS-B) are members of the same VSAN. If both devices are listed in the FCNS database then ISLs are not an issue.  
+
Examine the FLOGI database on the two switches that are directly connected to the host HBA and subsystem ports. Also, verify that both ports (attached to MDS-A and MDS-B) are members of the same VSAN. If both devices are listed in the FCNS databasethen ISLs are not an issue.  
-
In Fabric Manager, choose '''Tools''' > '''Ping or Tools''' > '''Traceroute''' (or use the fcping or fctrace CLI commands) to verify connectivity. See the "'''FC Ping and FC Traceroute'''".
+
In '''Fabric Manager''', choose '''Tools''' > '''Ping or Tools''' > '''Traceroute''' (or use the '''fcping''' or '''fctrace''' CLI commands) to verify connectivity. See the "FC Ping and FC Traceroute" section.
===Verifying SAN Element Registration===
===Verifying SAN Element Registration===
Answer the following questions to verify that your end devices are registered to the fabric:  
Answer the following questions to verify that your end devices are registered to the fabric:  
* Are the HBAs and subsystem ports successfully registered with the fabric name server?  
* Are the HBAs and subsystem ports successfully registered with the fabric name server?  
-
** In Device Manager, choose FC > Name Server.  
+
** In Device Manager, choose '''FC''' > '''Name Server'''.  
-
** In the CLI, use the show fcns commands.  
+
** In the CLI, use the '''show fcns''' command.  
* Does the correct pWWN for the HBAs and the storage subsystem ports show up on the correct port in the FLOGI database?  
* Does the correct pWWN for the HBAs and the storage subsystem ports show up on the correct port in the FLOGI database?  
-
** In Fabric Manager, choose Switches > Interfaces > FC Physical > FLOGI.  
+
** In Fabric Manager, choose '''Switches''' > '''Interfaces''' > '''FC Physical''' > '''FLOGI'''.  
-
** In the CLI, use the show flogi commands.  
+
** In the CLI, use the '''show flogi''' command.  
* Are the HBA and storage subsystem on the same VSAN?  
* Are the HBA and storage subsystem on the same VSAN?  
-
** In Fabric Manager, choose End Devices and verify the VSAN IDs are identical.  
+
** In Fabric Manager, choose '''End Devices''' and verify that the VSAN IDs are identical.  
-
** From the CLI, use the show vsan membership command.  
+
** From the CLI, use the '''show vsan membership''' command.  
* Does any single zone contain both devices?  
* Does any single zone contain both devices?  
-
** In Fabric Manager, choose the Zone > Edit Full Zone Database and select the active zone set (in bold) for the VSAN that contains the end devices. Verify that both devices are members of the same zone.  
+
** In Fabric Manager, choose '''Zone''' > '''Edit Full Zone Database''' and select the active zone set (in bold) for the VSAN that contains the end devices. Verify that both devices are members of the same zone.  
-
** From the CLI, use the show zoneset active command.  
+
** From the CLI, use the '''show zoneset active''' command.
===Fibre Channel End-to-End Connectivity===
===Fibre Channel End-to-End Connectivity===
Line 130: Line 136:
* Can the host complete a port login (PLOGI) to the storage subsystem?  
* Can the host complete a port login (PLOGI) to the storage subsystem?  
* Is there any SCSI exchange that takes place between the server and the disk array?  
* Is there any SCSI exchange that takes place between the server and the disk array?  
-
* Is the HBA configured for N port?  
+
* Is the HBA configured for the N port?  
-
You can use the HBA configuration utilities or the host system logs to determine if the subsystem pWWN or FC ID is listed as a device. This can validate that FSPF is working correctly.  
+
To validate whether FSPF is working correctly, you can use the HBA configuration utilities or the host system logs to determine if the subsystem pWWN or FC ID is listed as a device.
'''Fabric Issues''' <br>
'''Fabric Issues''' <br>
Line 145: Line 151:
'''Port Issues''' <br>  
'''Port Issues''' <br>  
-
Initial tasks to perform while investigating port connectivity issues include:  
+
Answer these questions when investigating port connectivity issues:  
-
* Verify correct media: copper or optical; single-mode (SM) or multimode (MM).
+
* Are you using the correct media: copper or optical; single-mode (SM) or multimode (MM)?
* Is the media broken or damaged?  
* Is the media broken or damaged?  
* Is the LED on the switch green?  
* Is the LED on the switch green?  
Line 159: Line 165:
* Black box: An SFP is not present.  
* Black box: An SFP is not present.  
-
'''Figure 1-1 Device Manager: Device View '''
+
<p>'''Figure 1-1 Device Manager: Device View '''</p>
[[Image:1-1.jpg]]
[[Image:1-1.jpg]]
Line 169: Line 175:
* Speed  
* Speed  
* Frames transmitted and received  
* Frames transmitted and received  
-
* Percentage utilization for the CPU, dynamic memory, and Flash memory  
+
* Percentage utilization for the CPU, dynamic memory, and flash memory  
-
'''Figure 1-2 Device Manager: Summary View'''
+
<p>'''Figure 1-2 Device Manager: Summary View'''</p>
[[Image:1-2.jpg]]
[[Image:1-2.jpg]]
Line 178: Line 184:
Additional tabs include the following:  
Additional tabs include the following:  
-
* Rx BB Credit—Configure and view buffer-to-buffer credits (BB_credits).  
+
* Rx BB Credit — Configure and view buffer-to-buffer credits (BB_credits).  
-
* Other—View PortChannel ID, WWN, and maximum transmission unit (MTU), and configure maximum receive buffer size.  
+
* Other — View PortChannel ID, WWN, and maximum transmission unit (MTU), and configure maximum receive buffer size.  
-
* FLOGI—View FC ID, pWWN, nWWN, BB_credits, and class of service for N port connections.  
+
* FLOGI — View FC ID, pWWN, nWWN, BB_credits, and class of service for N port connections.  
-
* ELP—View pWWN, nWWN, BB_credits, and supported classes of service for ISLs.  
+
* ELP — View pWWN, nWWN, BB_credits, and supported classes of service for ISLs.  
-
* Trunk Config—View and configure trunk mode and allowed VSANs.  
+
* Trunk Config — View and configure trunk mode and allowed VSANs.  
-
* Trunk Failure—View the failure cause for ISLs.  
+
* Trunk Failure — View the failure cause for ISLs.  
-
* Physical—Configure beaconing; view SFP information.  
+
* Physical — Configure beaconing; view SFP information.  
-
* Capability—View current port capability for hold-down timers, BB credits, maximum receive buffer size.  
+
* Capability — View current port capability for hold-down timers, BB credits, maximum receive buffer size.  
'''Device Manager: Port Monitoring''' <br>
'''Device Manager: Port Monitoring''' <br>
Line 192: Line 198:
Additional tabs include the following:  
Additional tabs include the following:  
-
* Protocol—View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to- buffer flow control.  
+
* Protocol — View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to-buffer flow control.  
-
* Discards—View the number of frames discarded by the port, including Class 2, Class 3, and Class F frames, EISL frames, and totals.  
+
* Discards — View the number of frames discarded by the port, including Class 2, Class 3, and Class F frames, EISL frames, and totals.  
-
* Link Errors—View the number of link errors, including link failures, signal losses, synchronization failures, invalid transmission words, and delimiter and address identifier errors.  
+
* Link Errors — View the number of link errors, including link failures, signal losses, synchronization failures, invalid transmission words, and delimiter and address identifier errors.  
-
* Frame Errors—View frame error statistics, including the number of frames with invalid CRC, Class 3 frames that were discarded upon reception, FBSY returns for selected situations, and FRJT returns resulting from frame rejection by fabric.  
+
* Frame Errors — View frame error statistics, including the number of frames with invalid CRC, Class 3 frames that were discarded upon reception, FBSY returns for selected situations, and FRJT returns resulting from frame rejection by fabric.  
-
* Class 2 Traffic—View the amount of Class 2 traffic for the selected interval.  
+
* Class 2 Traffic — View the amount of Class 2 traffic for the selected interval.  
-
* Class 2 Errors—View error statistics for Class 2 traffic, including busy frame responses and port rejects.  
+
* Class 2 Errors — View error statistics for Class 2 traffic, including busy frame responses and port rejects.  
-
* FICON—View FICON error statistics, including pacing, disparity, EOF, OOF, and order sets errors.
+
* FICON — View FICON error statistics, including pacing, disparity, EOF, OOF, and order sets errors.
'''Device Manager: Oversubscription Information'''<br>  
'''Device Manager: Oversubscription Information'''<br>  
-
Device Manager provides oversubscription information for supported switching modules and line cards. Use the Device View to view an individual module and then right-click to select from the following tasks:  
+
Device Manager provides oversubscription information for supported switching modules and line cards. Use the Device View to view an individual module, and then right-click to select from the following tasks:  
* Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness)  
* Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness)  
* Check port group oversubscription (display includes group and ports, port speed, and send and receive speed and includes possible oversubscription indicators)  
* Check port group oversubscription (display includes group and ports, port speed, and send and receive speed and includes possible oversubscription indicators)  
Line 219: Line 225:
'''Figure 1-3 Troubleshooting Process Flowchart'''
'''Figure 1-3 Troubleshooting Process Flowchart'''
 +
<br>
 +
 +
[[Image:1-3.jpg]]
[[Image:1-3.jpg]]
==System Messages==
==System Messages==
-
The system software sends these syslog (system) messages to the console (and, optionally, to a logging server on another system) during operation. Not all messages indicate a problem with your system. Some messages are purely informational, while others might help diagnose problems with links, internal hardware, or the system software.  
+
The system software sends these syslog (system) messages to the console (and, optionally, to a logging server on another system) during operation. Not all messages indicate a problem with your system. Some messages are only informational, while others might help diagnose problems with links, internal hardware, or the system software.  
===System Message Text===
===System Message Text===
-
Message-text is a text string that describes the condition. This portion of the message might contain detailed information about the event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory address space. Because the information in these variable fields changes from message to message, it is represented here by short strings enclosed in square brackets <nowiki>([ ])</nowiki>. A decimal number, for example, is represented as <nowiki>[dec]</nowiki>.  
+
Message text is a text string that describes the condition. This portion of the message might contain detailed information about the event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory address space. Because the information in these variable fields changes from message to message, it is represented here by short strings enclosed in square brackets <nowiki>([ ])</nowiki>. A decimal number, for example, is represented as <nowiki>[dec]</nowiki>.  
:<nowiki>PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.</nowiki>
:<nowiki>PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.</nowiki>
-
Use this string to find the matching system message in the Cisco MDS 9000 Family System Messages Reference.   
+
Use this string to find the matching system message in the ''Cisco MDS 9000 Family System Messages Reference''.   
-
Each system message is followed by an explanation and recommended action. The action may be as simple as &quot;No action required.&quot; It may involve a fix or a recommendation to contact technical support as shown in the following example:  
+
Each system message is followed by an explanation and recommended action. The action might be as simple as &quot;No action required.&quot; The message might involve a fix or a recommendation to contact technical support as shown in the following example:  
::'''Error Message'''  PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface <nowiki>[chars] is not supported.</nowiki>
::'''Error Message'''  PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface <nowiki>[chars] is not supported.</nowiki>
Line 237: Line 246:
::'''Explanation'''  Transceiver (SFP) is not from an authorized vendor.  
::'''Explanation'''  Transceiver (SFP) is not from an authorized vendor.  
-
::'''Recommended Action'''  Enter the '''show interface transceiver''' CLI command or similar Fabric Manager/Device Manager command to determine the transceiver being used. Please contact your customer support representative for a list of authorized transceiver vendors.
+
::'''Recommended Action'''  Enter the '''show interface transceiver''' CLI command or similar Fabric Manager or Device Manager command to determine the transceiver being used. Contact your customer support representative for a list of authorized transceiver vendors.
===Syslog Server Implementation===
===Syslog Server Implementation===
-
The syslog facility allows the Cisco MDS 9000 Family platform to send a copy of the message log to a host for more permanent storage. This can be useful if the logs need to be examined over a long period of time or when the Cisco MDS switch is not accessible.  
+
The syslog facility allows the Cisco MDS 9000 Family platform to send a copy of the message log to a host for more permanent storage. This function is useful if the logs need to be examined over a long period of time or when the Cisco MDS switch is not accessible.  
-
This example will demonstrate how to configure a Cisco MDS switch to utilize the syslog facility on a Solaris platform. Although a Solaris host is being used, syslog configuration on all UNIX and Linux systems is very similar.  
+
This example demonstrates how to configure a Cisco MDS switch to use the syslog facility on a Solaris platform. Although a Solaris host is being used, the syslog configuration on all UNIX and Linux systems is very similar.  
Syslog uses the concept of a facility to determine how it should be handled on the syslog server (the Solaris system in this example), and the message severity. Therefore, different message severities can be handled differently by the syslog server. They could be logged to different files or e-mailed to a particular user. Specifying a severity determines that all messages of that level and greater severity (lower number) will be acted upon.  
Syslog uses the concept of a facility to determine how it should be handled on the syslog server (the Solaris system in this example), and the message severity. Therefore, different message severities can be handled differently by the syslog server. They could be logged to different files or e-mailed to a particular user. Specifying a severity determines that all messages of that level and greater severity (lower number) will be acted upon.  
----
----
-
{{Note| The Cisco MDS messages should be logged to a different file from the standard syslog file so that they cannot be confused with other non-Cisco syslog messages. The logfile should not be located on the / file system, to prevent log messages from filling up the / file system.
+
{{Note| The Cisco MDS messages should be logged to a different file from the standard syslog file so that they cannot be confused with other non-Cisco syslog messages. The logfile should not be located on the / file system to prevent log messages from filling up the / file system.
<br>
<br>
Syslog Client: switch1<br>
Syslog Client: switch1<br>
Line 259: Line 268:
To configure system message logging servers, follow these steps:  
To configure system message logging servers, follow these steps:  
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
-
# In Fabric Manager, choose Switches > Events > Syslog in the Physical pane, and then click the Servers tab in the Information pane. In Device Manager, choose Logs > Syslog > Setup and click the Servers tab in the Syslog dialog box.  
+
# In Fabric Manager, choose '''Switches''' > '''Events''' > '''Syslog''' in the '''Physical''' pane, and then click the '''Servers''' tab in the '''Information''' pane. In Device Manager, choose '''Logs''' > '''Syslog''' > '''Setup''' and click the '''Servers''' tab in the Syslog dialog box.  
-
# Click the Create Row icon in Fabric Manager or Create in Device Manager to add a new syslog server.  
+
# Click the '''Create Row''' icon in Fabric Manager or '''Create''' in Device Manager to add a new syslog server.  
-
# Enter the name or IP address in dotted decimal notation (for example, 192.168.2.12) of the syslog server in the Name or IP Address field.  
+
# Enter the name or IP address in dotted decimal notation (for example, 192.168.2.12) of the syslog server in the '''Name''' or '''IP Address''' field.  
-
# Set the message severity threshold by clicking the MsgSeverity radio button, and set the facility by clicking the Facility radio button.  
+
# Set the message severity threshold by clicking the '''MsgSeverity''' radio button, and set the facility by clicking the '''Facility''' radio button.  
-
# Click Apply Changes in Fabric Manager or click Create in Device Manager to save and apply your changes.  
+
# Click '''Apply Changes''' in Fabric Manager or click '''Create''' in Device Manager to save and apply your changes.  
-
# If CFS is enabled in Fabric Manager for the syslog feature, click CFS to commit these changes to propagate the configuration through the fabric.  
+
# If CFS is enabled in Fabric Manager for the syslog feature, click '''CFS''' to commit these changes to propagate the configuration through the fabric.  
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
-
Device Manager allows you to view event logs on your local PC as well as those on the switch. For a permanent record of all events that occur on the switch, you should store these messages off the switch. To do this the Cisco MDS switch must be configured to send syslog messages to your local PC and a syslog server must be running on that PC to receive those messages. These messages can be categorized into four classes:  
+
Device Manager allows you to view event logs on your local PC as well as those on the switch. For a permanent record of all events that occur on the switch, you should store these messages off the switch. The Cisco MDS switch must be configured to send syslog messages to your local PC and a syslog server must be running on that PC to receive those messages. These messages can be categorized into four classes:  
-
* Hardware—Line card or power supply problems  
+
* Hardware — Line card or power supply problems  
-
* Link incidents—FICON port condition changes  
+
* Link incidents — FICON port condition changes  
-
* Accounting—User change events  
+
* Accounting — User change events  
-
* Events—All other events  
+
* Events — All other events  
----
----
-
{{Note| You should avoid using PCs that have IP addresses randomly assigned to them by DHCP. The switch continues to use the old IP address unless you manually change it; however the Device Manager prompts you if it does detect this situation. UNIX workstations have a built-in syslog server. You must have root access (or run the Cisco syslog server as setuid to root) to stop the built-in syslog daemon and start the Cisco syslog server.}}
+
{{Note| You should avoid using PCs that have IP addresses randomly assigned to them by DHCP. The switch continues to use the old IP address unless you manually change it; however, the Device Manager prompts you if it does detect this situation. UNIX workstations have a built-in syslog server. You must have root access (or run the Cisco syslog server as setuid to root) to stop the built-in syslog daemon and start the Cisco syslog server.}}
----
----
Line 315: Line 324:
     root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd
     root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd
</pre>
</pre>
-
3. Test the syslog server by creating an event on the Cisco MDS switch . In this case, port fc1/2 was bounced and the following was listed on the syslog server. Notice that the IP address of the switch is listed in brackets.  
+
3. Test the syslog server by creating an event on the Cisco MDS switch. In this case, port fc1/2 was bounced and the following was listed on the syslog server. Notice that the IP address of the switch is listed in brackets.  
<pre>
<pre>
# tail -f /var/adm/MDS_logs
# tail -f /var/adm/MDS_logs
Line 332: Line 341:
===Viewing Logs with Fabric Manager===
===Viewing Logs with Fabric Manager===
Fabric Manager and Device Manager present concise views of the generated system messages and other logged events:  
Fabric Manager and Device Manager present concise views of the generated system messages and other logged events:  
-
* In Device Manager, click Logs to set up and view logs.  
+
* In Device Manager, click '''Logs''' to set up and view logs.  
-
* In Fabric Manager, select the Logs tab at the bottom of the fabric pane to view log information.  
+
* In Fabric Manager, click the '''Logs''' tab at the bottom of the fabric pane to view log information.  
* Learn to use Threshold Manager to alert you that critical statistics have exceeded a set threshold.  
* Learn to use Threshold Manager to alert you that critical statistics have exceeded a set threshold.  
-
---Viewing Logs with the CLI===
+
===Viewing Logs with the CLI===
The following CLI commands are available to access and view logs on a switch:  
The following CLI commands are available to access and view logs on a switch:  
<pre>
<pre>
Line 352: Line 361:
</pre>
</pre>
-
The following is an example of the show logging CLI command output.
+
The following is an example of the '''show logging''' CLI command output:
<pre>
<pre>
Musky-9506# show logging server
Musky-9506# show logging server
Line 364: Line 373:
You can view system messages from Device Manager if Device Manager is running from the same workstation as the Fabric Manager Server. Choose '''Logs''' > '''Events''' > '''current''' to view the system messages on Device Manager. The columns in the events table are sortable. In addition, you can use the Find button to locate text within the table.  
You can view system messages from Device Manager if Device Manager is running from the same workstation as the Fabric Manager Server. Choose '''Logs''' > '''Events''' > '''current''' to view the system messages on Device Manager. The columns in the events table are sortable. In addition, you can use the Find button to locate text within the table.  
-
You can view switch-resident logs even if you have not set up your local syslog server or your local PC is not in the switch's syslog server list. Because of memory constraints, these logs will wrap when they reach a certain size. The switch syslog has two logs: an NVRAM log that holds a limited number of critical and greater messages and a nonpersistent log that contains notice or greater severity messages. Hardware messages are part of these logs.  
+
You can view switch-resident logs even if you have not set up your local syslog server or your local PC is not in the switch's syslog server list. Because of memory constraints, these logs wrap when they reach a certain size. The switch syslog has two logs: an NVRAM log that holds a limited number of critical and greater messages and a nonpersistent log that contains notice or greater severity messages. Hardware messages are part of these logs.  
Use the '''show logging''' CLI command to view the logs on the supervisor.  
Use the '''show logging''' CLI command to view the logs on the supervisor.  
'''Viewing NVRAM logs''' <br>
'''Viewing NVRAM logs''' <br>
-
System messages that are priority 0, 1, or 2 are logged into NVRAM on the supervisor module. After a switch reboots, you can display these syslog messages in NVRAM using the show logging nvram CLI command.
+
System messages that are priority 0, 1, or 2 are logged into NVRAM on the supervisor module. After a switch reboots, you can display these syslog messages in NVRAM using the '''show logging nvram''' CLI command.
<pre>
<pre>
switch# show logging nvram  
switch# show logging nvram  
Line 406: Line 415:
* Brief explanation of the steps you have already taken to isolate and resolve the problem  
* Brief explanation of the steps you have already taken to isolate and resolve the problem  
-
After you have collected this information, see the "'''Obtaining Documentation, Obtaining Support, and Security Guidelines'''".  
+
After you have collected this information, see "Obtaining Documentation, Obtaining Support, and Security Guidelines".  
-
For more information on steps to take before calling Technical Support, see the "'''Before Contacting Technical Support'''".  
+
For more information on steps to take before calling Technical Support, see "Before Contacting Technical Support".  
 +
-----
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
<center>'''Back to Main Page''': [[Cisco MDS SAN-OS Troubleshooting Guide]]
<center>'''Back to Main Page''': [[Cisco MDS SAN-OS Troubleshooting Guide]]
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

Latest revision as of 10:05, 30 November 2010

This section introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when configuring and using the Cisco MDS 9000 Family of multilayer directors and fabric switches.

Contents



Generating a PDF of this section

Cisco MDS SAN-OS Troubleshooting Guide -- Overview PDF



Overview of the Troubleshooting Process

To troubleshoot your fabric environment, follow these general steps:


  1. Gather information that defines the specific symptoms.
  2. Identify all potential problems that could be causing the symptoms.
  3. Systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear.

To identify the possible problems, you need to use a variety of tools and understand the overall storage environment. For this reason, this guide describes a number of general troubleshooting tools in Appendix B, "Troubleshooting Tools and Methodology", including those that are specific to the Cisco MDS 9000 Family. This chapter also provides a plan for investigating storage issues. See other chapters in this book for detailed explanations of specific issues.

Best Practices

Best practices are the recommended steps you should take to ensure the proper operation of your fabric. We recommend the following general best practices for most SAN fabrics:

  • Maintain a consistent Cisco SAN-OS release across all your Cisco MDS switches.
  • Refer to the release notes for your Cisco SAN-OS release for the latest features, limitations, and caveats.
  • Enable system message logging. See the "System Messages" section.
  • Troubleshoot any new configuration changes after implementing the change.
  • Use Fabric Manager and Device Manager to proactively manage your fabric and detect possible problems before they become critical.

Troubleshooting Basics

This section provides a series of questions that may be useful when troubleshooting a problem with a Cisco MDS 9000 Family switch or connected devices. Use the answers to these questions to plan a course of action and to determine the scope of the problem. For example, if a host can only access some, but not all, of the logical unit numbers (LUNs) on an existing subsystem, then fabric-specific issues (such as FSPF, ISLs, or FCNS) do not need to be investigated. The fabric components can therefore be eliminated from possible causes of the problem.

General Steps

The two most common symptoms of problems in a storage network are:

  • A host cannot access its allocated storage
  • An application does not respond after attempting to access the allocated storage

By answering the questions in the following subsections, you can determine the paths you need to follow and the components that you should investigate further. These questions are independent of host, switch, or subsystem vendor.

Answer the following questions to determine the status of your installation:

  • Is this a newly installed system or an existing installation? (It could be a new SAN, host, or subsystem, or new LUNs exported to an existing host.)
  • Has the host ever been able to see its storage?
  • Does the host recognize any LUNs in the subsystem?
  • Are you trying to solve an existing application problem (too slow, too high latency, excessively long response time) or did the problem show up recently?
  • What changed in the configuration or in the overall infrastructure immediately before the applications started to have problems?

To discover a SAN problem, use the following general SAN troubleshooting steps:


  1. Gather information on problems in your fabric. See the "Gathering Information Using Common Fabric Manager Tools and CLI Commands" section.
  2. Verify physical connectivity between your switches and end devices. See the "Verifying Basic Connectivity" section.
  3. Verify registration to your fabric for all SAN elements. See the "Verifying SAN Element Registration" section.
  4. Verify the configuration for your end devices (storage subsystems and servers).
  5. Verify end-to-end connectivity and fabric configuration. See the "Fibre Channel End-to-End Connectivity" section.

Gathering Information Using Common Fabric Manager Tools and CLI Commands

This section highlights the Fabric Manager tools and CLI commands that are commonly used to troubleshoot problems within your fabric. These tools and commands are a subset of what you might use to troubleshoot your specific problem.

Each section in this guide may include additional tools and commands specific to the symptoms and possible problems covered in that chapter.

Common Fabric Manager Tools
Use the following navigation paths in Fabric Manager or Device Manager to access common troubleshooting information:

  • Overview of switch status — In Fabric Manager, click the Switch Health Analysis icon.
  • End-to-end connectivity — In Fabric Manager, click the End-to-End Connectivity Analysis icon.
  • Fabric configuration — In Fabric Manager, click the Fabric Configuration Analysis icon.
  • Module status — In Device Manager, choose Physical > Modules.
  • Cisco SAN-OS version — In Device Manager, choose Physical > System.
  • View logs — In Device Manager, choose Logs > FM Server or Logs > Switch Resident.
  • View Fabric Manager events — In Fabric Manager, click the Events tab at the bottom of the fabric pane.
  • Interface status — In Fabric Manager, choose Switches > Interfaces and select the port type you are interested in.
  • View name server information — In Device Manager, choose FC > Name Server.
  • View FLOGI information — In Fabric Manager, choose Switches > Interfaces > FC Physical > FLOGI.
  • Analyze the results of merging zones — In Fabric Manager, choose Zone > Merge Analysis.
  • View traffic statistics — In Device Manager, click the Device tab, select one or more ports, right-click the selected port, and choose Monitoring from the pop-up menu.

Fabric Manager and Device Manager also provide the following tools to proactively monitor your fabric:

  • ISL performance — In Fabric Manager, click the ISL Performance icon.
  • Network monitoring — In Device Manager, click the Summary tab.
  • Performance monitoring — In Fabric Manager, choose Performance > Start Collection.

Common CLI Commands
Enter the following commands and examine the outputs:

  • show module
  • show version
  • show running-config
  • show logging log
  • show interfaces brief
  • show fcns
  • show flogi
  • show hardware internal errors
  • show zoneset active
  • show accounting log

Note Note: Use the show running interface CLI command to view the interface configuration in Cisco SAN-OS Release 3.0(1) or later. The interface configuration as seen in the show running-config CLI command is no longer consolidated.


Note Note: To use commands with the internal keyword, you must have an account that is a member of the network-admin group.

Verifying Basic Connectivity

Answer the following questions to verify basic connectivity between your end devices:

  • Are you using the correct fiber (SM or MM)?
  • Did you check for a broken fiber?
  • Is the Fibre Channel port LED on the connected module green, and do the LEDs on any host bus adapter (HBA) or storage subsystem ports indicate normal functionality?
  • Is there a LUN masking policy applied on the storage subsystem? If yes, is the server allowed to see the LUNs exported by the storage array?
  • Is there a LUN masking policy configured on the host? Did you enable the server to see all the LUNs it can access?
  • If LUN masking software is used, is the host's pWWN listed in the LUN masking database?
  • Is the subsystem configured for an N port?

Examine the FLOGI database on the two switches that are directly connected to the host HBA and subsystem ports. Also, verify that both ports (attached to MDS-A and MDS-B) are members of the same VSAN. If both devices are listed in the FCNS database, then ISLs are not an issue.

In Fabric Manager, choose Tools > Ping or Tools > Traceroute (or use the fcping or fctrace CLI commands) to verify connectivity. See the "FC Ping and FC Traceroute" section.

Verifying SAN Element Registration

Answer the following questions to verify that your end devices are registered to the fabric:

  • Are the HBAs and subsystem ports successfully registered with the fabric name server?
    • In Device Manager, choose FC > Name Server.
    • In the CLI, use the show fcns command.
  • Does the correct pWWN for the HBAs and the storage subsystem ports show up on the correct port in the FLOGI database?
    • In Fabric Manager, choose Switches > Interfaces > FC Physical > FLOGI.
    • In the CLI, use the show flogi command.
  • Are the HBA and storage subsystem on the same VSAN?
    • In Fabric Manager, choose End Devices and verify that the VSAN IDs are identical.
    • From the CLI, use the show vsan membership command.
  • Does any single zone contain both devices?
    • In Fabric Manager, choose Zone > Edit Full Zone Database and select the active zone set (in bold) for the VSAN that contains the end devices. Verify that both devices are members of the same zone.
    • From the CLI, use the show zoneset active command.

Fibre Channel End-to-End Connectivity

Answering the following questions will help to determine if end-to-end Fibre Channel connectivity exists from a host or subsystem perspective:

  • Does the host list the subsystem's port WWN (pWWN) or FC ID in its logs?
  • Does the subsystem list the host's pWWN or FC ID in its logs or LUN masking database?
  • Can the host complete a port login (PLOGI) to the storage subsystem?
  • Is there any SCSI exchange that takes place between the server and the disk array?
  • Is the HBA configured for the N port?

To validate whether FSPF is working correctly, you can use the HBA configuration utilities or the host system logs to determine if the subsystem pWWN or FC ID is listed as a device.

Fabric Issues
Answering the following questions will help to determine the status of the fabric configuration:

  • Are both the HBA and the subsystem port successfully registered with the fabric name server?
  • Does the correct pWWN for the server HBA and the storage subsystem port show up on the correct port in the FLOGI database? In other words, is the device plugged into the correct port?
  • Does any single zone contain both devices? The zone members can be WWNs or FC IDs.
  • Is the zone correctly configured and part of the active configuration or zone set within the same VSAN?
  • Do the ISLs show any VSAN isolation?
  • Do the host and storage belong to the same VSAN?
  • Are any parameters, such as FSPF, static domain assignment, VSAN, or zoning, mismatched in the configuration of the different switches in the fabric?

Port Issues
Answer these questions when investigating port connectivity issues:

  • Are you using the correct media: copper or optical; single-mode (SM) or multimode (MM)?
  • Is the media broken or damaged?
  • Is the LED on the switch green?
  • Is the active LED on the HBA for the connected device on?

Basic port monitoring using Device Manager begins with the visual display in the Device View. (See Figure 1-1.) Port display descriptions include:

  • Green box: A successful fabric login has occurred; the connection is active.
  • Red X: A small form-factor pluggable (SFP) transceiver is present but there is no connection. This could indicate a disconnected or faulty cable, or no active device connection.
  • Red box: An SFP is present but fabric login (FLOGI) has failed. Typically there is a mismatch in port or fabric parameters with the neighboring device. For example, a port parameter mismatch would occur if a node device were connected to a port configured as an E port. An example of a fabric parameter mismatch would be differing timeout values.
  • Yellow box: In Device Manager, a port has been selected.
  • Gray box: The port is administratively disabled.
  • Black box: An SFP is not present.

Figure 1-1 Device Manager: Device View

1-1.jpg

Device Manager: Summary View
In Device Manager, selecting the Summary View expands the information available for port monitoring. (See Figure 1-2.) The display includes the following:

  • VSAN assignment
  • For N ports, the port World Wide Name (pWWN) and Fibre Channel ID (FC ID) of the connected device
  • For ISLs, the IP address of the connected switch
  • Speed
  • Frames transmitted and received
  • Percentage utilization for the CPU, dynamic memory, and flash memory

Figure 1-2 Device Manager: Summary View

1-2.jpg

Device Manager: Port Selection
To drill down for additional port information, use the Device View or Summary View. Select and double-click any port. The initial display shows administrative settings for Mode, Speed, and Status, plus current operational status, failure cause, and date of the last configuration change.

Additional tabs include the following:

  • Rx BB Credit — Configure and view buffer-to-buffer credits (BB_credits).
  • Other — View PortChannel ID, WWN, and maximum transmission unit (MTU), and configure maximum receive buffer size.
  • FLOGI — View FC ID, pWWN, nWWN, BB_credits, and class of service for N port connections.
  • ELP — View pWWN, nWWN, BB_credits, and supported classes of service for ISLs.
  • Trunk Config — View and configure trunk mode and allowed VSANs.
  • Trunk Failure — View the failure cause for ISLs.
  • Physical — Configure beaconing; view SFP information.
  • Capability — View current port capability for hold-down timers, BB credits, maximum receive buffer size.

Device Manager: Port Monitoring
To display additional details about port traffic, use the Device View or Summary View. In Device View, choose one or more ports, right-click and choose Monitoring from the pop-up menu. In Summary View, choose one or more interfaces, and click the Monitor tool. The initial display shows traffic statistics for the selected interval, including bytes and frames transmitted and received.

Additional tabs include the following:

  • Protocol — View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to-buffer flow control.
  • Discards — View the number of frames discarded by the port, including Class 2, Class 3, and Class F frames, EISL frames, and totals.
  • Link Errors — View the number of link errors, including link failures, signal losses, synchronization failures, invalid transmission words, and delimiter and address identifier errors.
  • Frame Errors — View frame error statistics, including the number of frames with invalid CRC, Class 3 frames that were discarded upon reception, FBSY returns for selected situations, and FRJT returns resulting from frame rejection by fabric.
  • Class 2 Traffic — View the amount of Class 2 traffic for the selected interval.
  • Class 2 Errors — View error statistics for Class 2 traffic, including busy frame responses and port rejects.
  • FICON — View FICON error statistics, including pacing, disparity, EOF, OOF, and order sets errors.

Device Manager: Oversubscription Information
Device Manager provides oversubscription information for supported switching modules and line cards. Use the Device View to view an individual module, and then right-click to select from the following tasks:

  • Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness)
  • Check port group oversubscription (display includes group and ports, port speed, and send and receive speed and includes possible oversubscription indicators)
  • Show port resources
  • Reset the module

Note Note: Port group oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching modules, the 32-port 2-Gbps Fibre Channel switching module, the Cisco MDS 9120 20-port 1/2-Gbps Fibre Channel module, and the Cisco MDS 9140 40-port 1/2-Gbps Fibre Channel module.

Primary Troubleshooting Flowchart

The flowchart in the following figure shows the overall troubleshooting process. Begin any troubleshooting investigation by checking one of the following four areas:

  • Physical port issues
  • Physical switch issues
  • Fx port issues
  • Fabric services

Figure 1-3 Troubleshooting Process Flowchart


1-3.jpg

System Messages

The system software sends these syslog (system) messages to the console (and, optionally, to a logging server on another system) during operation. Not all messages indicate a problem with your system. Some messages are only informational, while others might help diagnose problems with links, internal hardware, or the system software.

System Message Text

Message text is a text string that describes the condition. This portion of the message might contain detailed information about the event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory address space. Because the information in these variable fields changes from message to message, it is represented here by short strings enclosed in square brackets ([ ]). A decimal number, for example, is represented as [dec].

PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.

Use this string to find the matching system message in the Cisco MDS 9000 Family System Messages Reference.

Each system message is followed by an explanation and recommended action. The action might be as simple as "No action required." The message might involve a fix or a recommendation to contact technical support as shown in the following example:

Error Message PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.
Explanation Transceiver (SFP) is not from an authorized vendor.
Recommended Action Enter the show interface transceiver CLI command or similar Fabric Manager or Device Manager command to determine the transceiver being used. Contact your customer support representative for a list of authorized transceiver vendors.

Syslog Server Implementation

The syslog facility allows the Cisco MDS 9000 Family platform to send a copy of the message log to a host for more permanent storage. This function is useful if the logs need to be examined over a long period of time or when the Cisco MDS switch is not accessible.

This example demonstrates how to configure a Cisco MDS switch to use the syslog facility on a Solaris platform. Although a Solaris host is being used, the syslog configuration on all UNIX and Linux systems is very similar.

Syslog uses the concept of a facility to determine how it should be handled on the syslog server (the Solaris system in this example), and the message severity. Therefore, different message severities can be handled differently by the syslog server. They could be logged to different files or e-mailed to a particular user. Specifying a severity determines that all messages of that level and greater severity (lower number) will be acted upon.


Note Note: The Cisco MDS messages should be logged to a different file from the standard syslog file so that they cannot be confused with other non-Cisco syslog messages. The logfile should not be located on the / file system to prevent log messages from filling up the / file system.


Syslog Client: switch1
Syslog Server: 172.22.36.211 (Solaris)
Syslog facility: local1
Syslog severity: notifications (level 5, the default)

File to log MDS messages to: /var/adm/MDS_logs

Implementing Syslog with Fabric Manager

To configure system message logging servers, follow these steps:


  1. In Fabric Manager, choose Switches > Events > Syslog in the Physical pane, and then click the Servers tab in the Information pane. In Device Manager, choose Logs > Syslog > Setup and click the Servers tab in the Syslog dialog box.
  2. Click the Create Row icon in Fabric Manager or Create in Device Manager to add a new syslog server.
  3. Enter the name or IP address in dotted decimal notation (for example, 192.168.2.12) of the syslog server in the Name or IP Address field.
  4. Set the message severity threshold by clicking the MsgSeverity radio button, and set the facility by clicking the Facility radio button.
  5. Click Apply Changes in Fabric Manager or click Create in Device Manager to save and apply your changes.
  6. If CFS is enabled in Fabric Manager for the syslog feature, click CFS to commit these changes to propagate the configuration through the fabric.

Device Manager allows you to view event logs on your local PC as well as those on the switch. For a permanent record of all events that occur on the switch, you should store these messages off the switch. The Cisco MDS switch must be configured to send syslog messages to your local PC and a syslog server must be running on that PC to receive those messages. These messages can be categorized into four classes:

  • Hardware — Line card or power supply problems
  • Link incidents — FICON port condition changes
  • Accounting — User change events
  • Events — All other events

Note Note: You should avoid using PCs that have IP addresses randomly assigned to them by DHCP. The switch continues to use the old IP address unless you manually change it; however, the Device Manager prompts you if it does detect this situation. UNIX workstations have a built-in syslog server. You must have root access (or run the Cisco syslog server as setuid to root) to stop the built-in syslog daemon and start the Cisco syslog server.

Implementing Syslog with the CLI

To configure a syslog server using the CLI, follow these steps:


1. Configure the Cisco MDS switch:

switch1# config terminal
Enter configuration commands, one per line. End with CNTL/Z. 
switch1(config)# logging server 172.22.36.211 6 facility local1 
To display the configuration:
switch1# show logging server
Logging server: enabled
{172.22.36.211}
     server severity: notifications
     server facility: local1

2. Configure the syslog server:

a. Modify /etc/syslog.conf to handle local1 messages. For Solaris, there needs to be at least one tab between the facility.severity and the action (/var/adm/MDS_logs).
     #Below is for the MDS 9000 logging
     local1.notice /var/adm/MDS_logs
b. Create the log file.
     #touch /var/adm/MDS_logs
c. Restart syslog.
     # /etc/init.d/syslog stop
     # /etc/init.d/syslog start
     syslog service starting.
d. Verify syslog started.
     # ps -ef |grep syslogd
     root 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd

3. Test the syslog server by creating an event on the Cisco MDS switch. In this case, port fc1/2 was bounced and the following was listed on the syslog server. Notice that the IP address of the switch is listed in brackets.

# tail -f /var/adm/MDS_logs
Sep 17 11:07:41 [172.22.36.142.2.2] : 2004 Sep 17 11:17:29 pacific:
%PORT-5-IF_DOWN_INITIALIZING: %$VSAN 1%$ Interface fc1/2 is down (Initializing)
Sep 17 11:07:49 [172.22.36.142.2.2] : 2004 Sep 17 11:17:36 pacific: %PORT-5-IF_UP:
%$VSAN 1%$ Interface fc1/2 is up in mode TE
Sep 17 11:07:51 [172.22.36.142.2.2] : 2004 Sep 17 11:17:39 pacific:
%VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring console from pts/0
(dhcp-171-71-49-125.cisco.com)

Troubleshooting with Logs

Cisco SAN-OS generates many types of system messages on the switch and sends them to a syslog server. These messages can be viewed using Fabric Manager or the CLI to determine what events may have led up to the current problem condition you are facing.

Viewing Logs with Fabric Manager

Fabric Manager and Device Manager present concise views of the generated system messages and other logged events:

  • In Device Manager, click Logs to set up and view logs.
  • In Fabric Manager, click the Logs tab at the bottom of the fabric pane to view log information.
  • Learn to use Threshold Manager to alert you that critical statistics have exceeded a set threshold.

Viewing Logs with the CLI

The following CLI commands are available to access and view logs on a switch:

Musky-9506# show logging ?
console Show console logging configuration
info Show logging configuration
last Show last few lines of logfile
level Show facility logging configuration
logfile Show contents of logfile
module Show module logging configuration
monitor Show monitor logging configuration
nvram Show NVRAM log
server Show server logging configuration
<cr> Carriage Return

The following is an example of the show logging CLI command output:

Musky-9506# show logging server
Logging server: enabled
{10.91.51.204}
server severity: critical
server facility: user

Viewing the Log from the Supervisor

You can view system messages from Device Manager if Device Manager is running from the same workstation as the Fabric Manager Server. Choose Logs > Events > current to view the system messages on Device Manager. The columns in the events table are sortable. In addition, you can use the Find button to locate text within the table.

You can view switch-resident logs even if you have not set up your local syslog server or your local PC is not in the switch's syslog server list. Because of memory constraints, these logs wrap when they reach a certain size. The switch syslog has two logs: an NVRAM log that holds a limited number of critical and greater messages and a nonpersistent log that contains notice or greater severity messages. Hardware messages are part of these logs.

Use the show logging CLI command to view the logs on the supervisor.

Viewing NVRAM logs
System messages that are priority 0, 1, or 2 are logged into NVRAM on the supervisor module. After a switch reboots, you can display these syslog messages in NVRAM using the show logging nvram CLI command.

switch# show logging nvram 
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial 
number )
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-FANMOD_FAN_OK: Fan module 1 (Front fan) ok
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-FANMOD_FAN_OK: Fan module 2 (Rear fan) ok
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-CHASSIS_CLKMODOK: Chassis clock module A ok
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-CHASSIS_CLKMODOK: Chassis clock module B ok
2005 Sep 16 13:19:20 172.20.150.82 %PLATFORM-2-CHASSIS_CLKSRC: Current chassis clock 
source is clock-A
2005 Sep 16 13:19:36 172.20.150.82 %PLATFORM-2-PFM_STDBY_BIOS_STUCK: standby supervisor 
bios failed
2005 Sep 16 13:20:19 172.20.150.82 %IMAGE_DNLD-SLOT13-2-IMG_DNLD_STARTED:  Module image 
download process. Please wait until completion...
2005 Sep 16 13:20:32 172.20.150.82 %IMAGE_DNLD-SLOT13-IMG_DNLD_COMPLETE:  Module image 
download process. Download successful.
2005 Sep 16 15:44:46 172.20.150.82 %PLATFORM-2-PFM_STDBY_BIOS_STUCK: standby supervisor 
bios failed
2005 Sep 16 15:44:53 172.20.150.82 %PLATFORM-2-MOD_ALL_PWRDN_NOXBAR: All modules powered 
down due to non-availability of xbar modules
2005 Sep 16 15:45:41 172.20.150.82 %PLATFORM-2-MOD_PWRUP_XBAR: Modules powered up due to 
xbar availability
2005 Sep 18 15:12:07 172.20.150.82 %MODULE-2-MOD_FAIL: Initialization of module 14 
(serial: JAB092501FC) failed

Contacting Customer Support

If you are unable to solve a problem after using the troubleshooting suggestions in this guide, contact a customer service representative for assistance and further instructions. Before you call, have the following information ready to help your service provider assist you as quickly as possible:

  • Date you received the switch
  • Chassis serial number (located on a label on the right side of the rear panel of the chassis)
  • Type of software and release number
  • Maintenance agreement or warranty information
  • Brief description of the problem
  • Brief explanation of the steps you have already taken to isolate and resolve the problem

After you have collected this information, see "Obtaining Documentation, Obtaining Support, and Security Guidelines".

For more information on steps to take before calling Technical Support, see "Before Contacting Technical Support".



Back to Main Page: Cisco MDS SAN-OS Troubleshooting Guide

Rating: 0.0/5 (0 votes cast)

Personal tools