Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Troubleshooting Unicast Traffic
From DocWiki
This article provides information on how to troubleshoot unicast packet flow traffic issues for the M1 Series modules.
Contents |
Step 1: Packet is Received into Interface from Wire
During this step, the packet is received into the Nexus 7000 port. When troubleshooting this step, we want to look to ensure there is transceiver interoperability, and validate whether we are seeing any errors on the interface. We do this via using the following commands
- show interface interface
- show interface interface transceiver
PHX2-N7K-1# show interface e1/1
Ethernet1/1 is up
Hardware: 10000 Ethernet, address: 0024.986c.00b0 (bia 0024.986c.00b0)
Description: N7K-vdc-1 connecting to core 6506
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10g
Beacon is turned off
Auto-Negotiation is turned off
Input flow-control is off, output flow-control is off
Rate mode is shared
Switchport monitor is off
Last link flapped 7week(s) 4day(s)
Last clearing of "show interface" counters never
1 minute input rate 13056 bits/sec, 9 packets/sec
1 minute output rate 4608 bits/sec, 0 packets/sec
Rx
341190251 input packets 276211313 unicast packets 52112947 multicast packets
12865991 broadcast packets 0 jumbo packets 0 storm suppression packets
94295027129 bytes
Tx
462437316 output packets 85121 multicast packets
188251 broadcast packets 0 jumbo packets
648159081064 bytes
0 input error 0 short frame 0 watchdog
0 no buffer 0 runt 0 CRC 0 ecc
0 overrun 0 underrun 0 ignored 0 bad etype drop
0 bad proto drop 0 if down drop 0 input with dribble
0 input discard
0 output error 0 collision 0 deferred
0 late collision 0 lost carrier 0 no carrier
0 babble
0 Rx pause 0 Tx pause
1 interface resets
PHX2-N7K-1# show interface e1/1 transceiver details
Ethernet1/1 sfp is present name is CISCO-AVAGO <<< If this says type is (unknown), it is not supported. part number is SFBR-7700SDZ revision is B4 serial number is AGD12434116 nominal bitrate is 10300 MBits/sec Link length supported for 50/125um fiber is 82 m(s) Link length supported for 62.5/125um fiber is 26 m(s) cisco id is -- cisco extended id number is 4
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 45.46 C 75.00 C -5.00 C 70.00 C 0.00 C
Voltage 3.28 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 6.92 mA 10.50 mA 2.50 mA 10.50 mA 2.50 mA
Tx Power -2.75 dBm 1.69 dBm -11.30 dBm -1.30 dBm -7.30 dBm
Rx Power -2.43 dBm 1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Step 2: Linksec Decryption Occurs, 1st stage Port QoS
In step 2, Linksec decryption occurs as well as receive side stage 1 QoS.
It is important to step back and evaluate the difference between stage 1 and stage 2 QoS. The difference is that some ports can be configured in shared mode, whereas some can be configured in dedicated mode, on the 10G modules. What this means, is that there is 10g of bandwidth that can be dedicated to a port or shared amongst ports (4 ports, on the m132 module).
When running in shared mode, there exists a chance for contention accessing the 10g bandwidth through the 4:1 Mux. To alleviate this, some QoS intelligence was passed down to the 4:1 Mux which aggregates the ports.
In dedicated mode, there is no QoS applied at the Mux, instead, all traffic is processed in phase 2 QoS. To summarize, in shared mode, 1st stage QoS ensures fair access to the 10g of port bandwidth. In both shared and dedicated mode, 2nd stage QoS occurs to provide ingress queuing to the system.
For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.
Use the show policy-map command to see per queue dropped packets.
The commands to troubleshoot Linksec and Port QoS are as follows:
- show cts interface [all | interface]
- show queuing interface interface
- show policy-map interface (for per queue drop)
switch# show cts interface all Working Example
CTS Information for Interface Ethernet2/24:
CTS is enabled, mode: CTS_MODE_DOT1X
IFC state: CTS_IFC_ST_CTS_OPEN_STATE
Authentication Status: CTS_AUTHC_SUCCESS
Peer Identity: india1
Peer is: CTS Capable
802.1X role: CTS_ROLE_AUTH
Last Re-Authentication:
Authorization Status: CTS_AUTHZ_SUCCESS
PEER SGT: 2
Peer SGT assignment: Trusted
Global policy fallback access list:
SAP Status: CTS_SAP_SUCCESS
Configured pairwise ciphers: GCM_ENCRYPT
Replay protection: Enabled
Replay protection mode: Strict
Selected cipher: GCM_ENCRYPT
Current receive SPI: sci:1b54c1fbff0000 an:0
Current transmit SPI: sci:1b54c1fc000000 an:0
PHX2-N7K-1# show cts interface eth 1/8 Broken Example
CTS Information for Interface Ethernet1/8:
CTS is enabled, mode: CTS_MODE_MANUAL
IFC state: Unknown
Authentication Status: CTS_AUTHC_INIT
Peer Identity:
Peer is: Not CTS Capable
802.1X role: CTS_ROLE_UNKNOWN
Last Re-Authentication:
Authorization Status: CTS_AUTHZ_INIT
PEER SGT: 0
Peer SGT assignment: Not Trusted
SAP Status: CTS_SAP_INIT
Configured pairwise ciphers:
Replay protection:
Replay protection mode:
Selected cipher:
Current receive SPI:
Current transmit SPI:
PHX2-N7K-1# show queuing int eth 1/1
Interface Ethernet1/1 TX Queuing strategy: Weighted Round-Robin
Port QoS is enabled
Queuing Mode in TX direction: mode-cos
Transmit queues [type = 1p7q4t]
Queue Id Scheduling Num of thresholds
_____________________________________________________________
1p7q4t-out-q-default WRR 04
1p7q4t-out-q2 WRR 04
1p7q4t-out-q3 WRR 04
1p7q4t-out-q4 WRR 04
1p7q4t-out-q5 WRR 04
1p7q4t-out-q6 WRR 04
1p7q4t-out-q7 WRR 04
1p7q4t-out-pq1 Priority 04
Configured WRR
WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 12[1p7q4t-out-q3]
12[1p7q4t-out-q4] 12[1p7q4t-out-q5] 12[1p7q4t-out-q6] 12[1p7q4t-out-q7]
WRR configuration read from HW
WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 11[1p7q4t-out-q3]
11[1p7q4t-out-q4] 11[1p7q4t-out-q5] 11[1p7q4t-out-q6] 11[1p7q4t-out-q7]
Configured queue-limit ratios
queue-limit ratios: 78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3]
*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1]
* means unused queue with mandatory minimum queue-limit
queue-limit ratios configuration read from HW
queue-limit ratios: 78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3]
*1[1p7q4t-out-q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1]
* means unused queue with mandatory minimum queue-limit
Thresholds:
COS Queue Threshold Type Min Max
__________________________________________________________________
0 1p7q4t-out-q-default DT 100 100
1 1p7q4t-out-q-default DT 100 100
2 1p7q4t-out-q-default DT 100 100
3 1p7q4t-out-q-default DT 100 100
4 1p7q4t-out-q-default DT 100 100
5 1p7q4t-out-pq1 DT 100 100
6 1p7q4t-out-pq1 DT 100 100
7 1p7q4t-out-pq1 DT 100 100
Interface Ethernet1/1 RX Queuing strategy: Weighted Round-Robin
Queuing Mode in RX direction: mode-cos
Receive queues [type = 8q2t]
Port Cos not configured
Queue Id Scheduling Num of thresholds
____________________________________________________________
8q2t-in-q-default WRR 02
8q2t-in-q2 WRR 02
8q2t-in-q3 WRR 02
8q2t-in-q4 WRR 02
8q2t-in-q5 WRR 02
8q2t-in-q6 WRR 02
8q2t-in-q7 WRR 02
8q2t-in-q1 WRR 02
Configured WRR
WRR bandwidth ratios: 20[8q2t-in-q-default] 0[8q2t-in-q2] 0[8q2t-in-q3]
0[8q2tin-q4] 0[8q2t-in-q5] 0[8q2t-in-q6] 0[8q2t-in-q7] 80[8q2t-in-q1]
WRR configuration read from HW
WRR bandwidth ratios: 20[8q2t-in-q-default] 0[8q2t-in-q2] 0[8q2t-in-q3]
0[8q2t-in-q4] 0[8q2t-in-q5] 0[8q2t-in-q6] 0[8q2t-in-q7] 80[8q2t-in-q1]
No queue-limit ratios user configuration
________________________________________
queue-limit ratios configuration read from HW
queue-limit ratios: 100[8q2t-in-q-default] 100[8q2t-in-q2] 100[8q2t-in-q3]
100[8q2t-in-q4] 100[8q2t-in-q5] 100[8q2t-in-q6] 100[8q2t-in-q7] 100[8q2t-in-q1]
Thresholds:
COS Queue Threshold Type Min Max
__________________________________________________________________
0 8q2t-in-q-default DT 100 100
1 8q2t-in-q-default DT 100 100
2 8q2t-in-q-default DT 100 100
3 8q2t-in-q-default DT 100 100
4 8q2t-in-q-default DT 100 100
5 8q2t-in-q1 DT 100 100
6 8q2t-in-q1 DT 100 100
7 8q2t-in-q1 DT 100 100
PHX2-N7K-1# show policy-map interface eth 1/2
Global statistics status : enabled
Ethernet1/2
Service-policy (queuing) input: default-in-policy
policy statistics status: enabled
Class-map (queuing): in-q1 (match-any)
queue-limit percent 50
bandwidth percent 80
queue dropped pkts : 0
Class-map (queuing): in-q-default (match-any)
queue-limit percent 50
bandwidth percent 20
queue dropped pkts : 0
Service-policy (queuing) output: default-out-policy
policy statistics status: enabled
Class-map (queuing): out-pq1 (match-any)
priority level 1
queue-limit percent 16
queue dropped pkts : 0
Class-map (queuing): out-q2 (match-any)
queue-limit percent 1
queue dropped pkts : 0
Class-map (queuing): out-q3 (match-any)
queue-limit percent 1
queue dropped pkts : 0
Class-map (queuing): out-q-default (match-any)
queue-limit percent 82
bandwidth remaining percent 25
queue dropped pkts : 0
Step 3: Second Stage Port QoS Occurs
For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.
Use the show policy-map command to view queue drops .
The commands to troubleshoot Port QoS are
- show queuing interface interface
- show policy-map interface
Step 4: Layer 2 Source/Destination MAC Processing
In this step, the ASIC submits the packet headers to theLayer 2 engine for lookup, and the Layer 2 engine performs source/destination MAC processing.
To validate forwarding of the Layer 2 engine, we should first look at the centralized mac table aggregated on the supervisor to validate whether the mac addresses are correlated as we expect them, and assigned to the ports where we expect the Mac’s to reside.
Based off of this, we can then validate the hardware programming on the ingress linecard to validate that our mac address table is properly programmed into the hardware based Layer 2 engine on the linecard.
We first will look at the mac address table, then we can ensure programming is properly occurring in the hardware table.
The commands used to accomplish this are as follows:
- show mac address-table
- show hardware mac address-table module interface interface
To drill down on a specific MAC address, we can use the grep function with these commands to validate the mac is associated with a particular port, and that the hardware programming reflects that.
- show mac address-table | grep macaddress
- show hardware mac address-table module interface interface | grep macaddress
{note|When evaluating the Hardware mac table, if the Index is set to 0x00400, or the GM bit is set to “1”, that traffic will be routed. For example, you will see the index set to 0x00400 and GM bit set to 1 for traffic destined to the mac address local to the device}
PHX2-N7K-1# show mac address-table
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+------+----------------
G - 0023.ac67.dd41 static - False False sup-eth1(R)
G 5 0023.ac67.dd41 static - False False sup-eth1(R)
* 5 0000.0c07.ac01 dynamic 0 False False Eth1/1
* 5 000c.2943.a67e dynamic 180 False False Eth1/1
* 5 000c.294b.c5ca dynamic 0 False False Eth1/1
* 5 000d.ece2.0640 dynamic 180 False False Eth1/1
* 5 0013.5f32.aa80 dynamic 0 False False Eth1/1
* 5 0018.8b45.41b7 dynamic 0 False False Eth1/1
* 5 0019.bb2f.4871 dynamic 0 False False Eth1/1
* 5 0019.bbe5.f3b8 dynamic 1230 False False Eth1/1
* 5 001a.4b33.ccdc dynamic 1080 False False Eth1/1
* 5 001a.4ba8.6a9c dynamic 1680 False False Eth1/1
* 5 001b.210a.87f9 dynamic 600 False False Eth1/1
* 5 001b.d46f.70e0 dynamic 60 False False Eth1/1
* 5 001c.c4e5.ac9a dynamic 150 False False Eth1/1
* 5 0023.ac64.6f7c dynamic 1230 False False Eth1/1
* 5 0024.986d.21c8 dynamic 270 False False Eth1/1
PHX2-N7K-1# sho hardware mac address-table 1 int eth1/1
Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr | GM | Sec|
TR | NT | RM | RMA | Cap | Fld | Always | | | |
| ic | | fied| Byte| Sel | | ure| AP | FY | | | TURE| | Learn
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+----+--
--+----+----+----+-----+-----+-----+--------
1 1 2 000c.294b.c5ca 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
1 1 2 0050.567e.58e6 0x00422 0 3 0 68 1 0
0 0 0 0 0 0 0 0
1 1 2 0050.56aa.6067 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
1 1 2 00c0.b72e.cfa0 0x00422 0 3 0 67 1 0
0 0 0 0 0 0 0 0
1 1 2 0018.8b45.41b7 0x00422 0 3 0 68 1 0
0 0 0 0 0 0 0 0
1 1 2 0013.5f32.aa80 0x00422 0 3 0 68 1 0
0 0 0 0 0 0 0 0
1 1 2 0050.56aa.75ca 0x00422 0 3 0 64 1 0
0 0 0 0 0 0 0 0
1 1 2 00a0.9811.a233 0x00422 0 3 0 39 1 0
0 0 0 0 0 0 0 0
PHX2-N7K-1# show mac address-table | grep 000c.294b.c5ca
* 5 000c.294b.c5ca dynamic 150 False False Eth1/1
PHX2-N7K-1# show hardware mac address-table 1 int eth 1/1 | grep 000c.294b.c5ca
1 1 2 000c.294b.c5ca 0x00422 0 3 0 67 1 0 0 0 0 0 0 0 0 0
