Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Troubleshooting Unicast Traffic

From DocWiki

Revision as of 15:08, 7 October 2010 by Sdurham (Talk | contribs)
Jump to: navigation, search

This article provides information on how to troubleshoot unicast packet flow traffic issues for the M1 Series modules.

Guide Contents
Troubleshooting Overview
Troubleshooting Installs, Upgrades, and Reboots
Troubleshooting Licensing'
Troubleshooting VDCs
Troubleshooting CFS
Troubleshooting Ports
Troubleshooting vPCs
Troubleshooting VLANs
Troubleshooting STP
Troubleshooting Routing
Troubleshooting Licensing (this section)
Troubleshooting WCCP
Troubleshooting Memory
Troubleshooting Packet Flow Issues
Before Contacting Technical Support
Troubleshooting Tools and Methodology

Contents


Step 1: Packet is Received into Interface from Wire

During this step, the packet is received into the Nexus 7000 port. When troubleshooting this step, we want to look to ensure there is transceiver interoperability, and validate whether we are seeing any errors on the interface. We do this via using the following commands

  • show interface interface
  • show interface interface transceiver

PHX2-N7K-1# show interface e1/1

 Ethernet1/1 is up
 Hardware: 10000 Ethernet, address: 0024.986c.00b0 (bia 0024.986c.00b0)
 Description: N7K-vdc-1 connecting to core 6506
 MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 1/255, rxload 1/255
 Encapsulation ARPA
 Port mode is trunk
 full-duplex, 10 Gb/s, media type is 10g
 Beacon is turned off
 Auto-Negotiation is turned off
 Input flow-control is off, output flow-control is off
 Rate mode is shared
 Switchport monitor is off 
 Last link flapped 7week(s) 4day(s)
 Last clearing of "show interface" counters never
 1 minute input rate 13056 bits/sec, 9 packets/sec
 1 minute output rate 4608 bits/sec, 0 packets/sec
 Rx
   341190251 input packets 276211313 unicast packets 52112947 multicast packets
   12865991 broadcast packets 0 jumbo packets 0 storm suppression packets
   94295027129 bytes
 Tx
   462437316 output packets 85121 multicast packets
   188251 broadcast packets 0 jumbo packets
   648159081064 bytes
   0 input error 0 short frame 0 watchdog
   0 no buffer 0 runt 0 CRC 0 ecc
   0 overrun  0 underrun 0 ignored 0 bad etype drop
   0 bad proto drop 0 if down drop 0 input with dribble
   0 input discard
   0 output error 0 collision 0 deferred
   0 late collision 0 lost carrier 0 no carrier
   0 babble
   0 Rx pause 0 Tx pause
 1 interface resets

PHX2-N7K-1# show interface e1/1 transceiver details

 Ethernet1/1
   sfp is present
   name is CISCO-AVAGO     <<< If this says type is (unknown), it is not supported.
   part number is SFBR-7700SDZ    
   revision is B4  
   serial number is AGD12434116     
   nominal bitrate is 10300 MBits/sec
   Link length supported for 50/125um fiber is 82 m(s)
   Link length supported for 62.5/125um fiber is 26 m(s)
   cisco id is --
   cisco extended id number is 4
          SFP Detail Diagnostics Information (internal calibration)
 ----------------------------------------------------------------------------
                                    Alarms                  Warnings
                               High        Low         High          Low
 ----------------------------------------------------------------------------
 Temperature   45.46 C        75.00 C     -5.00 C     70.00 C        0.00 C
 Voltage        3.28 V         3.63 V      2.97 V      3.46 V        3.13 V
 Current        6.92 mA       10.50 mA     2.50 mA    10.50 mA       2.50 mA
 Tx Power       -2.75 dBm       1.69 dBm  -11.30 dBm   -1.30 dBm     -7.30 dBm
 Rx Power       -2.43 dBm       1.99 dBm  -13.97 dBm   -1.00 dBm     -9.91 dBm
 Transmit Fault Count = 0
 ----------------------------------------------------------------------------
 Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning

Step 2: Linksec Decryption Occurs, 1st stage Port QoS

In step 2, Linksec decryption occurs as well as receive side stage 1 QoS.

It is important to step back and evaluate the difference between stage 1 and stage 2 QoS. The difference is that some ports can be configured in shared mode, whereas some can be configured in dedicated mode, on the 10G modules. What this means, is that there is 10g of bandwidth that can be dedicated to a port or shared amongst ports (4 ports, on the m132 module).

When running in shared mode, there exists a chance for contention accessing the 10g bandwidth through the 4:1 Mux. To alleviate this, some QoS intelligence was passed down to the 4:1 Mux which aggregates the ports.

In dedicated mode, there is no QoS applied at the Mux, instead, all traffic is processed in phase 2 QoS. To summarize, in shared mode, 1st stage QoS ensures fair access to the 10g of port bandwidth. In both shared and dedicated mode, 2nd stage QoS occurs to provide ingress queuing to the system.

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.

Use the show policy-map command to see per queue dropped packets.

The commands to troubleshoot Linksec and Port QoS are as follows:

  • show cts interface [all | interface]
  • show queuing interface interface
  • show policy-map interface (for per queue drop)


switch# show cts interface all Working Example

CTS Information for Interface Ethernet2/24:
   CTS is enabled, mode:   CTS_MODE_DOT1X
   IFC state:              CTS_IFC_ST_CTS_OPEN_STATE
   Authentication Status:  CTS_AUTHC_SUCCESS
     Peer Identity:        india1
     Peer is:              CTS Capable
     802.1X role:          CTS_ROLE_AUTH
     Last Re-Authentication:
   Authorization Status:   CTS_AUTHZ_SUCCESS
     PEER SGT:             2
     Peer SGT assignment:  Trusted
     Global policy fallback access list:
   SAP Status:             CTS_SAP_SUCCESS
     Configured pairwise ciphers: GCM_ENCRYPT
     Replay protection: Enabled
     Replay protection mode: Strict
     Selected cipher: GCM_ENCRYPT
     Current receive SPI: sci:1b54c1fbff0000 an:0
     Current transmit SPI: sci:1b54c1fc000000 an:0


PHX2-N7K-1# show cts interface eth 1/8 Broken Example

CTS Information for Interface Ethernet1/8:
   CTS is enabled, mode:   CTS_MODE_MANUAL
   IFC state:              Unknown
   Authentication Status:  CTS_AUTHC_INIT
     Peer Identity:        
     Peer is:              Not CTS Capable
     802.1X role:          CTS_ROLE_UNKNOWN
     Last Re-Authentication: 
   Authorization Status:   CTS_AUTHZ_INIT
     PEER SGT:             0
     Peer SGT assignment:  Not Trusted
   SAP Status:             CTS_SAP_INIT
     Configured pairwise ciphers: 
     Replay protection: 
     Replay protection mode: 
     Selected cipher: 
     Current receive SPI: 
     Current transmit SPI: 


PHX2-N7K-1# show queuing int eth 1/1

 Interface Ethernet1/1 TX Queuing strategy: Weighted Round-Robin
 Port QoS is enabled
   Queuing Mode in TX direction: mode-cos
   Transmit queues [type = 1p7q4t]
   Queue Id                       Scheduling   Num of thresholds
   _____________________________________________________________
      1p7q4t-out-q-default        WRR             04
      1p7q4t-out-q2               WRR             04
      1p7q4t-out-q3               WRR             04
      1p7q4t-out-q4               WRR             04
      1p7q4t-out-q5               WRR             04
      1p7q4t-out-q6               WRR             04
      1p7q4t-out-q7               WRR             04
      1p7q4t-out-pq1              Priority        04

  Configured WRR
     WRR bandwidth ratios:   25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 12[1p7q4t-out-q3]
 12[1p7q4t-out-q4] 12[1p7q4t-out-q5] 12[1p7q4t-out-q6] 12[1p7q4t-out-q7]
  WRR configuration read from HW
     WRR bandwidth ratios:   25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 11[1p7q4t-out-q3]
 11[1p7q4t-out-q4] 11[1p7q4t-out-q5] 11[1p7q4t-out-q6] 11[1p7q4t-out-q7]

  Configured queue-limit ratios
     queue-limit ratios:     78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3]
 *1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1]
  * means unused queue with mandatory minimum queue-limit

  queue-limit ratios configuration read from HW
     queue-limit ratios:     78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3]
*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1]
  * means unused queue with mandatory minimum queue-limit

  Thresholds:
     COS    Queue                       Threshold Type      Min     Max
     __________________________________________________________________
      0     1p7q4t-out-q-default            DT               100     100
      1     1p7q4t-out-q-default            DT               100     100
      2     1p7q4t-out-q-default            DT               100     100
      3     1p7q4t-out-q-default            DT               100     100
      4     1p7q4t-out-q-default            DT               100     100
      5     1p7q4t-out-pq1                  DT               100     100
      6     1p7q4t-out-pq1                  DT               100     100
      7     1p7q4t-out-pq1                  DT               100     100
 
 Interface Ethernet1/1 RX Queuing strategy: Weighted Round-Robin
   Queuing Mode in RX direction: mode-cos
   Receive queues [type = 8q2t]
   Port Cos not configured
   Queue Id                      Scheduling   Num of thresholds
   ____________________________________________________________
      8q2t-in-q-default           WRR             02
      8q2t-in-q2                  WRR             02
      8q2t-in-q3                  WRR             02
      8q2t-in-q4                  WRR             02
      8q2t-in-q5                  WRR             02
      8q2t-in-q6                  WRR             02
      8q2t-in-q7                  WRR             02
      8q2t-in-q1                  WRR             02

  Configured WRR
     WRR bandwidth ratios:   20[8q2t-in-q-default] 0[8q2t-in-q2] 0[8q2t-in-q3]
0[8q2tin-q4] 0[8q2t-in-q5] 0[8q2t-in-q6] 0[8q2t-in-q7] 80[8q2t-in-q1]
  WRR configuration read from HW
     WRR bandwidth ratios:   20[8q2t-in-q-default] 0[8q2t-in-q2] 0[8q2t-in-q3] 
0[8q2t-in-q4] 0[8q2t-in-q5] 0[8q2t-in-q6] 0[8q2t-in-q7] 80[8q2t-in-q1]

  No queue-limit ratios user configuration
  ________________________________________

  queue-limit ratios configuration read from HW
     queue-limit ratios:     100[8q2t-in-q-default] 100[8q2t-in-q2] 100[8q2t-in-q3] 
100[8q2t-in-q4] 100[8q2t-in-q5] 100[8q2t-in-q6] 100[8q2t-in-q7] 100[8q2t-in-q1]

  Thresholds:
     COS    Queue                       Threshold Type      Min     Max
     __________________________________________________________________
      0     8q2t-in-q-default               DT               100     100
      1     8q2t-in-q-default               DT               100     100
      2     8q2t-in-q-default               DT               100     100
      3     8q2t-in-q-default               DT               100     100
      4     8q2t-in-q-default               DT               100     100
      5     8q2t-in-q1                      DT               100     100
      6     8q2t-in-q1                      DT               100     100
      7     8q2t-in-q1                      DT               100     100


PHX2-N7K-1# show policy-map interface eth 1/2

Global statistics status :   enabled
Ethernet1/2
 Service-policy (queuing) input:   default-in-policy 
   policy statistics status:   enabled

   Class-map (queuing):   in-q1 (match-any)
     queue-limit percent 50 
     bandwidth percent 80 
     queue dropped pkts : 0 

   Class-map (queuing):   in-q-default (match-any)
     queue-limit percent 50 
     bandwidth percent 20 
     queue dropped pkts : 0 

 Service-policy (queuing) output:   default-out-policy 
   policy statistics status:   enabled

   Class-map (queuing):   out-pq1 (match-any)
     priority level 1
     queue-limit percent 16 
     queue dropped pkts : 0

   Class-map (queuing):   out-q2 (match-any)
     queue-limit percent 1 
     queue dropped pkts : 0 

   Class-map (queuing):   out-q3 (match-any)
     queue-limit percent 1 
     queue dropped pkts : 0 

   Class-map (queuing):   out-q-default (match-any)
     queue-limit percent 82 
     bandwidth remaining percent 25
     queue dropped pkts : 0


Step 3: Second Stage Port QoS Occurs

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.

Use the show policy-map command to view queue drops .

The commands to troubleshoot Port QoS are

  • show queuing interface interface
  • show policy-map interface


Step 4: Layer 2 Source/Destination MAC Processing

In this step, the ASIC submits the packet headers to theLayer 2 engine for lookup, and the Layer 2 engine performs source/destination MAC processing.

To validate forwarding of the Layer 2 engine, we should first look at the centralized mac table aggregated on the supervisor to validate whether the mac addresses are correlated as we expect them, and assigned to the ports where we expect the Mac’s to reside.

Based off of this, we can then validate the hardware programming on the ingress linecard to validate that our mac address table is properly programmed into the hardware based Layer 2 engine on the linecard.

We first will look at the mac address table, then we can ensure programming is properly occurring in the hardware table.

The commands used to accomplish this are as follows:

  • show mac address-table
  • show hardware mac address-table module interface interface

To drill down on a specific MAC address, we can use the grep function with these commands to validate the mac is associated with a particular port, and that the hardware programming reflects that.

  • show mac address-table | grep macaddress
  • show hardware mac address-table module interface interface | grep macaddress

{note|When evaluating the Hardware mac table, if the Index is set to 0x00400, or the GM bit is set to “1”, that traffic will be routed. For example, you will see the index set to 0x00400 and GM bit set to 1 for traffic destined to the mac address local to the device}

PHX2-N7K-1# show mac address-table

Legend: 
       * - primary entry, G - Gateway MAC, (R) - Routed MAC
       age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure  NTFY   Ports
---------+-----------------+--------+---------+------+------+----------------
G     -    0023.ac67.dd41    static       -     False  False sup-eth1(R)
G 5        0023.ac67.dd41    static       -     False  False sup-eth1(R)
* 5        0000.0c07.ac01    dynamic   0        False  False Eth1/1
* 5        000c.2943.a67e    dynamic   180      False  False Eth1/1
* 5        000c.294b.c5ca    dynamic   0        False  False Eth1/1
* 5        000d.ece2.0640    dynamic   180      False  False Eth1/1
* 5        0013.5f32.aa80    dynamic   0        False  False Eth1/1
* 5        0018.8b45.41b7    dynamic   0        False  False Eth1/1
* 5        0019.bb2f.4871    dynamic   0        False  False Eth1/1
* 5        0019.bbe5.f3b8    dynamic   1230     False  False Eth1/1
* 5        001a.4b33.ccdc    dynamic   1080     False  False Eth1/1
* 5        001a.4ba8.6a9c    dynamic   1680     False  False Eth1/1
* 5        001b.210a.87f9    dynamic   600      False  False Eth1/1
* 5        001b.d46f.70e0    dynamic   60       False  False Eth1/1
* 5        001c.c4e5.ac9a    dynamic   150      False  False Eth1/1
* 5        0023.ac64.6f7c    dynamic   1230     False  False Eth1/1
* 5        0024.986d.21c8    dynamic   270      False  False Eth1/1


PHX2-N7K-1# sho hardware mac address-table 1 int eth1/1

Valid| PI |  BD   |      MAC      |  Index | Stat| SW | Modi| Age | Tmr | GM | Sec|
 TR | NT | RM | RMA | Cap | Fld | Always     |    |       |               | 
     |  ic |    | fied| Byte| Sel |    | ure| AP | FY |    |     | TURE|     |  Learn
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+----+--
--+----+----+----+-----+-----+-----+--------
 1     1    2     000c.294b.c5ca  0x00422    0     3    0     67    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     0050.567e.58e6  0x00422    0     3    0     68    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     0050.56aa.6067  0x00422    0     3    0     67    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     00c0.b72e.cfa0  0x00422    0     3    0     67    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     0018.8b45.41b7  0x00422    0     3    0     68    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     0013.5f32.aa80  0x00422    0     3    0     68    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     0050.56aa.75ca  0x00422    0     3    0     64    1     0    
0    0    0    0     0     0     0   0 
 1     1    2     00a0.9811.a233  0x00422    0     3    0     39    1     0    
0    0    0    0     0     0     0   0 


PHX2-N7K-1# show mac address-table | grep 000c.294b.c5ca

* 5        000c.294b.c5ca    dynamic   150      False  False Eth1/1


PHX2-N7K-1# show hardware mac address-table 1 int eth 1/1 | grep 000c.294b.c5ca

 1     1    2     000c.294b.c5ca  0x00422    0     3    0     67    1     0    
0    0    0    0     0     0     0   0

Rating: 4.0/5 (10 votes cast)

Personal tools