Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Troubleshooting FCoE

From DocWiki

(Difference between revisions)
Jump to: navigation, search
Line 976: Line 976:
<br>
<br>
 +
<pre>sw8-gd76-sup2-storage-vdc# show system internal fcfwd idxmap interface-to-port
-
sw8-gd76-sup2-storage-vdc# show system internal fcfwd idxmap interface-to-port
+
Interface to Port Table:(All values in hex)
-
 
+
-
Interface to Port Table:(All values in hex)  
+
-
 
+
-
<nowiki>--------------------------------------------------------------------------------</nowiki>
+
-
 
+
-
                          |  |    |  |lcl|  | S | P |    |    |    |     
+
-
 
+
-
                          |no|    |  |idx|if | L | O |    |    |    |     
+
-
 
+
-
                          |of|    |  |  |ty-| O | R | mts | port| if  |     
+
-
 
+
-
        if_index        |GI|  GI |VL|LCI|pe | T | T | node| mode|flags|     
+
-
 
+
-
<nowiki>--------------------------|--|-----|--|---|---|---|---|-----|-----|-----|-------</nowiki>
+
-
 
+
-
16000108 Epo265          | 1|  406| 0| 00| 16| 00| 00| 0101| 0000| 0000|
+
-
 
+
-
06081000 sup-fc0        | 1| 10ea| 0| 01| 06| 00| 01| 0101| 0000| 0000|
+
-
 
+
-
160002bb Epo700          | 1|  402| 0| 00| 16| 00| 00| 0101| 0000| 0000|
+
-
 
+
-
1a200000 Ethernet5/1    | 1|    0| 0| 00| 1a| 04| 00| 0502| 0000| 0000|
+
-
 
+
-
1a201000 Ethernet5/2    | 1|    1| 0| 01| 1a| 04| 01| 0502| 0000| 0000|
+
-
 
+
-
1a202000 Ethernet5/3    | 1|    2| 0| 02| 1a| 04| 02| 0502| 0000| 0000|
+
-
 
+
-
1a203000 Ethernet5/4    | 1|    3| 0| 03| 1a| 04| 03| 0502| 0000| 0000|
+
 +
------------------------------------------------------------------------
 +
                        |  |    |  |lcl|  | S | P |    |    |    |
 +
                        |no|    |  |idx|if | L | O |    |    |    |     
 +
                        |of|    |  |  |ty-| O | R | mts | port| if  |
 +
        if_index        |GI|  GI |VL|LCI|pe | T | T | node| mode|flags|
 +
------------------------------------------------------------------------
 +
16000108 Epo265          | 1|  406| 0| 00| 16| 00| 00| 0101| 0000| 0000|
 +
06081000 sup-fc0        | 1| 10ea| 0| 01| 06| 00| 01| 0101| 0000| 0000|
 +
160002bb Epo700          | 1|  402| 0| 00| 16| 00| 00| 0101| 0000| 0000|
 +
1a200000 Ethernet5/1    | 1|    0| 0| 00| 1a| 04| 00| 0502| 0000| 0000|
 +
1a201000 Ethernet5/2    | 1|    1| 0| 01| 1a| 04| 01| 0502| 0000| 0000|
 +
1a202000 Ethernet5/3    | 1|    2| 0| 02| 1a| 04| 02| 0502| 0000| 0000|
 +
1a203000 Ethernet5/4    | 1|    3| 0| 03| 1a| 04| 03| 0502| 0000| 0000|
 +
</pre>
<br>
<br>

Revision as of 22:47, 8 March 2013

This article describes how to troubleshoot licensing on a Cisco NX-OS device.

Guide Contents
Troubleshooting Overview
Troubleshooting FCoE (this section)
Troubleshooting Licensing
Troubleshooting VDCs
Troubleshooting CFS
Troubleshooting Ports
Troubleshooting vPCs
Troubleshooting VLANs
Troubleshooting STP
Troubleshooting Routing
Troubleshooting Unicast Traffic
Troubleshooting WCCP
Troubleshooting Memory
Troubleshooting Packet Flow Issues
Before Contacting Technical Support
Troubleshooting Tools and Methodology

Contents



Information About Troubleshooting FCoE Issues


This article describes how to troubleshoot FCoE issues on a Cisco NX-OS device.



VFC in error disabled state because of missing F1 module license

Example:

sw2-gd73-storage-vdc# show interface vfc2/2

vfc2/2 is down (Error Disabled - Ethernet interface module is not licensed)
                      /*** If FCoE license is not enabled for F1 module           ***/

                      /*** then VFC interface will remain in error disabled state ***/
    Bound interface is Ethernet2/2

    Hardware is Ethernet
    Port WWN is 20:42:00:26:51:cf:55:00

    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled

    Port vsan is 256
    477 fcoe in packets

    48312 fcoe in octets
    1616 fcoe out packets

    2181244 fcoe out octets
    Interface last changed at Mon Feb  4 11:19:20 2012



Recommended Action:


1) Enable FCoE license for F1 module (from default VDC)

sw2-gd73# conf

Enter configuration commands, one per line.  End with CNTL/Z.
sw2-gd73(config)# show mod

Mod  Ports  Module-Type                         Model              Status
2    32     1/10 Gbps Ethernet Module           N7K-F132XP-15      ok /*** F1 module

5    0      Supervisor module-1X                N7K-SUP1           active *
6    0      Supervisor module-1X                N7K-SUP1           ha-standby

sw2-gd73(config)# license fcoe module 2
sw2-gd73(config)# show license fcoe

Module-Number    Package-Name
2          FCOE-N7K-F132XP       /*** FCoE license has successfully been ***/
                                 /*** checked out for F1 module 2 ***/
sw2-gd73(config)#

sw2-gd73# switchto vdc storage-vdc
sw2-gd73-storage-vdc# show license usage FCOE-N7K-F132XP   

FCoE Package: FCOE-N7K-F132XP
Installed count is: 0

License reserved for module: 2
Application              Vdc                      Module
----------------------------------------------------------------------------
fcoe                     storage-vdc               2
/*** License usage details shown in storage VDC ***/
----------------------------------------------------------------------------
sw2-gd73-storage-vdc#



Verify FLOGI was processed successfully for VFC interface

Recommended Action:


1) "show flogi database" should show an entry for VFC interface with host PWWN/NWWN

sw2-gd71# show flogi database
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME
vfc21            3839  0xaf0000  50:0a:09:85:89:2a:b3:f8 50:0a:09:80:89:2a:b3:f8

vfc22            3839  0xaf0001  21:00:00:c0:dd:10:28:b1 20:00:00:c0:dd:10:28:b1
Total number of flogi = 2.

sw2-gd71#


2) "Show fcoe database" should show an entry for VFC interface with host PWWN/MAC address

sw2-gd71# show fcoe database

INTERFACE       FCID            PORT NAME               MAC ADDRESS

vfc21           0xaf0000        50:0a:09:85:89:2a:b3:f8 00:c0:dd:10:e9:c5

vfc22           0xaf0001        21:00:00:c0:dd:10:28:b1 00:c0:dd:10:28:b1
sw2-gd71#


3) "Show interface vfc <no>" should declare VFC interface as up on VSAN where successful FLOGI happened

sw2-gd71# show interface vfc 21
vfc21 is trunking (Not all VSANs UP on the trunk)
    Bound interface is Ethernet2/1
    Hardware is Ethernet

    Port WWN is 20:00:00:24:f7:16:5d:02
    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled
    Port mode is TF

    Port vsan is 3839
    Speed is auto

    Trunk vsans (admin allowed and active) (1,3839)
    Trunk vsans (up)                       (3839)

    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             (1)

    5 minute average counters not available
    Current counters not available

    Interface last changed at Tue Sep 28 11:08:41 2010
sw2-gd71# show fcns database (should show an entry for each successful FLOGI)
    VSAN 3839:

    FCID        TYPE  PWWN                    (VENDOR)        FC4-TYPE:FEATURE

    0x020000    N     21:00:00:c0:dd:10:f3:a5 (Qlogic)        scsi-fcp:init

    Total number of entries = 1
    
    sw2-gd75# show fc2 plogi_pwwn  (should show an entry with host PWWN for each successful FLOGI)
    
    HIX  ADDRESS   VSAN   S_ID   D_ID  IFINDEX   FL  STATE                     PWWN

    672 dfc8e540   3839 fffffa 020000 1e000015 0000      2 21:00:00:c0:dd:10:f3:a5
    678 dfc8e1c0   3839 fffffc 020000 1e000015 0000      2 21:00:00:c0:dd:10:f3:a5
sw2-gd75#


Verify ELP was processed successfully for VFC interface

Recommended Action:


1) "Show fcoe database" should show an entry for VFC interface with peer FCF MAC and VSAN

sw1-gd74# show fcoe database

VE Ports:
INTERFACE       MAC ADDRESS             VSAN
vfc18           aa:b1:ba:58:03:fc       3839   

sw1-gd74#


2) "show system internal fcfwd mpmap vfcs" should show both local and peer FCF MAC details

sw1-gd74# show system internal fcfwd mpmap vfcs

FCoE VFC Information<span id="fck_dom_range_temp_1362765161072_218" /><span id="fck_dom_range_temp_1362765161073_764" />

ID | if-index |S|M|T| Members | MAC(s)
       {}-------------------------
vfc18 |0x1e209700|D|E||0x1a097000|Ethernet4/8 (U)|f8:66:f2:e4:c4:63 /*** *Local FCF MAC
      |          | | ||          |               |6c:9c:ed:43:db:cf /***  Peer FCF MAC
sw1-gd74#


3) "Show interface vfc <no>" should declare VFC interface as up on VSAN where successful ELP happened

sw1-gd74# show interface vfc 18

vfc18 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet4/8

    Hardware is Ethernet
    Port WWN is 27:02:00:0d:ec:fc:63:78

    Peer port WWN is 00:00:00:00:00:00:00:00
    Admin port mode is E, trunk mode is on

    snmp link state traps are enabled
    Port mode is TE

    Port vsan is 3839
    Speed is 10 Gbps

    Trunk vsans (admin allowed and active) (1,3839)
    Trunk vsans (up)                       (3839)   
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             (1)

    5 minute average counters not available
    Current counters not available

    Interface last changed at Tue Sep 28 12:26:47 2010
sw1-gd74#switch# show system internal fcfwd sfib unicast vsan 2 (sfib should have an unicast entry for VFC interface)
vsan 2 - # of routes 2
0x980020/0xffffff -> vfc45

0x98ffff/0xffffff -> sup-fc (high)


VFC (VF/VE) interface is not up on FCoE enabled VSAN

Recommended Action:

  1. Check LLDP/DCBX status (Verify that DCBX negotiation has successfully gone through on port where VFC is bound to, relevant for F port)
sw2-gd73# show lldp dcbx interface ethernet 2/1

Local DCBXP Control information:
Operation version: 00  Max version: 00  Seq no: 1  Ack no: 2 

Type/
Subtype    Version    En/Will/Adv Config

004/000     000        Y/N/Y      8906001b21 08
003/000     000        Y/N/Y      0808

002/000     000        Y/N/Y      1123200019 19191900 00000004
Peer's DCBXP Control information:

Operation version: 00  Max version: 00  Seq no: 2  Ack no: 1
Type/      Max/Oper

Subtype    Version    En/Will/Err Config
002/000     000/000    Y/Y/N      0001000032 32000000 00000002

003/000     000/000    Y/Y/N      0801
004/000     000/000    Y/Y/N      8906001b21 08891400 1b2108

sw2-gd73#


2. Check MPMap status (both on Sup and FCoE LC, bind information for VFC should be available in MPMap)

sw2-gd73# show system internal fcfwd mpmap vfcs

                           FCoE VFC Information

   ID   | if-index |S|M|T|          Members              |      MAC(s)
     }--------------

    1718|0x1e0006b5|D|E|-|0x16000018|eth-pc 25       (U) |aa:b1:9b:68:*03:fc
        |          | | | |          |                    |00:00:00:00:00:00
     }--------------

      21|0x1e000014|D|F|-|0x1a080000|Ethernet2/1     (U)*|aa:b1:9b:68:03:fc
     }--------------

      22|0x1e000015|D|F|-|0x160000ff|eth-pc 256      (U)*|aa:b1:9b:68:03:fc
     }--------------

      28|0x1e00001b|D|E|-|0x1a087000|Ethernet2/8     (U)*|aa:b1:9b:68:03:fc
        |          | | | |          |                    |6a:ef:2b:30:03:fc
     }--------------

sw2-gd73# 


3. VLAN-VSAN mapping status (FCoE enabled VLAN should be operational)

sw2-gd73# sh vlan fcoe

Original VLAN ID        Translated VSAN ID      Association State
---------

      3839                      3839             Operational
      4079                      299              Operational

sw2-gd73#


4. EthPM status (Check EthPM status for port where VFC is bound to)

sw2-gd73# show running-config interface ethernet 2/1

!Command: show running-config interface Ethernet2/1
!Time: Sun Sep 26 10:36:07 2010

version 5.2(1)
interface Ethernet2/1

  switchport mode trunk
  switchport trunk allowed vlan 1,3839,4079

 /*** VFC's allowed VSAN those are FCoE enabled should be in Ethernet ports allowed
 /*** VLAN list, native vlan should be configured and operationally up to allow vlan 
 /*** discovery packet
  speed 10000

  medium p2p
  no shutdown

sw2-gd73#
sw2-gd73# show int br | inc 2/1

Eth2/1        1      eth  trunk  up      none                        10G(D) --
sw2-gd74# show interface ethernet 2/1 trunk

Port          Native  Status        Port

              Vlan                  Channel
Eth2/1        1       trunking      --
Port          Vlans Allowed on Trunk
Eth2/1        1-3967,4048-4093
Port          Vlans Err-disabled on Trunk
Eth2/1        none
Port          STP Forwarding
Eth2/1        1,3839

/*** Native VLAN and FCoE enabled VLAN should be up and STP forwarding ***/

sw2-gd74#


5. IDXmap status (Check if Ethernet interface where VFC is bound to is part of idxmap list)

sw2-gd73# show system internal fcfwd idxmap port-to-interface


6. Check VFC and VFC's VSAN operation status (VFC and VFC's VSAN should not be error disabled, VFC's VSAN valid status Waiting for FLOGI/Initializing/UP)

switch# show interface vfc 310 trunk vsan

vfc310 is trunking
    Vsan 1 is down (Error Disabled - VLAN not FCoE Enabled)
    Vsan 2 is up (None) -->
    Vsan 3 is down (Error Disabled - VLAN not FCoE Enabled)

switch#


7. Check if VLAN discovery has gone through (For F port only)

switch# debug fcoe_mgr trace (Check for following trace)

"Sending VLAN <VLAN> for FCF <FCF>"

Also look for "show platform software fcoe_mgr event-histroy interface vfc <number>" 
and look for "FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY" in state machine details


8. Check if FCF discovery has gone through

switch# show platform software fcoe_mgr info interface vfc <number>

 FSM current state: FCOE_MGR_VFC_ST_PHY_UP
  PSS Runtime Config:-

      Type: 3
      Bound IF: Eth2/1

      FCF Priority: 128 (Global)
  PSS Runtime Data:-

      IOD: 0x00000000, WWN: 20:02:00:26:98:08:61:02
      Created at: Thu Oct 21 01:09:18 2010

      FC Admin State: up
      Oper State: up, Reason: down

      Eth IF Index: Eth2/1
      Port Vsan: 299  /*** Check port VSAN is fcoe enabled ***/

      Port Mode: F port
      Config Vsan: 1,299,3839

      Oper Vsan: 299/*** Oper VSAN is FCoE enabled ***/
                    /*** Expect FLOGI/ELP on these are operationally UP VSANs only ***/
  Solicits on vsan: 299
                    /*** Solicitation happened on these set of VSANs ***/
                    /*** (relevant for F port)                       ***/
      FIP Capable ? : TRUE

      UP using DCBX ? : FALSE
  PSS VN Port data:- /*** VN port information ***/

     FC ID 131072 -
     vfc  index 503316500 vfc name vfc21

     vsan id 299
     enode_mac 00:c0:dd:14:58:c7

     vfc wwn 50:0a:09:84:89:fb:6c:4f
 (For E port)

"show fcoe database" should show an entry for VFC on VSAN if FCF discovery goes through successfully.


9. Check if FLOGI/ELP has gone through

"show flogi databsae"

"show flogi internal error"
"show port internal event-history error"

"show fc2 internal event-histroy error"



Packet does not reach SUP or FCoE data packet gets dropped

Recommended Action:

1. Check Packet Drop statistics (Check relevant drop field on asic instance where VFC is bound)

"Show hardware internal statistics pktflow dropped"


2. Check CBL status (If CBL drop happens on port/VLAN which is FCoE enabled and bound to VFC interface)

module-2# show ha internal forwarding 0 table cbl-vlan port 0

                             (INST 0, PORT 0)
|                                  INGRESS                                     |

| Disable State | 0,2-4031,4036-4078,4080-4095
| Forward State | 1, 3839, 2, 4032-4035,4079

/***FCoE enabled VLAN must be forwarding on operationally UP vlan for a given 
/*** ethernet interface on N7K, FCoE enabled mapped VSAN should be operationally 
/*** up on MDS, VLAN should be in BLOCKED state
| Blocked State |

| Learn State   |
|                                   EGRESS                                     |

| Disable State | 0,2-4031,4036-4078,4080-4095
| Forward State | 1, 2, 3839, 4032-4035,4079''','''

/*** same as defined on ingress ***/
| Blocked State |

| Learn State   |
module-2#


3. FCF MAC programming

module-2# show ha internal forwarding 0 registers | inc FCF_MAC

    1c68  ORI_DI_CFG_FCF_MAC_P0__1                                      0000aab1
    1c6c  ORI_DI_CFG_FCF_MAC_P0__0                                      9b6803fc

/*** P0_0 and P0_1 Should match with MAC address programmed
/*** under MAC(s) column of "show system internal fcfwd mpmap vfcs" 

/*** for a given vfc interface
    1c90  ORI_DI_CFG_FCF_MAC_P1__1                                      0000aab1

    1c94  ORI_DI_CFG_FCF_MAC_P1__0                                      9b6803fcmodule-2#

module-2# show ha internal mac asic <instance> dump reg eb | inc FCOE
[0x874]ORI_EB_CFG_FCOE_ETHERTYPE:        0x00008906

[0x875]ORI_EB_CFG_FCOE_VERSION:          0x00000000
[0x879]ORI_EB_CFG_V2_AT_FCOE:            0x00000009

[0x88f]ORI_EB_CFG_FC_TO_FCOE_EOF__0:     0x41464200
          _FCOE_FC_CRC_EN[10]            0x1

          _FCOE_DA_OVERRIDE_EN[30]       0x1
/*** DA override register must set for Orion-CR and beyond 

/*** version LC for VE link which is up on ethernet interface
          _FCOE_SA_OVERRIDE_EN[31]       0x1 

/*** SA override register must set for Orion-CR and beyond
/*** version LC for VE link which is up on ethernet

interface[0x8a6]ORI_EB_CNT_PORT0_TOTAL_FCOE:              0x00000000
[0x8ab]ORI_EB_CNT_PORT0_FCOE_CRC:                         0x00000000

          _FCOE_FC_CRC_EN[10]                             0x1
          _FCOE_DA_OVERRIDE_EN[30]                        0x1

         _FCOE_SA_OVERRIDE_EN[31]                         0x1
[0x93e]ORI_EB_CFG_PORT0_FCOE_DA:                          0x00000000
[0x93e]ORI_EB_CFG_PORT0_FCOE_DA__1:                       0x00000000

[0x93f]ORI_EB_CFG_PORT0_FCOE_DA__0:                       0x00000000
[0x940]ORI_EB_CFG_PORT0_FCOE_SA:                          0x0000aab1

/***DA/SA overrride MAC register information must match with MAC address
/*** defined in "show system internal fcfwd mpmap vfcs"

/*** for a given VFC interface
/*** Under MAC(s) column first MAC should be set in

/***Port 0/1, SA_0 and SA_1 register,
/*** 2nd MAC (only applicable for VE) should be set in

/*** Port0/1, DA_0 and DA_1 register

 [0x940]ORI_EB_CFG_PORT0_FCOE_SA__1: 0x0000aab1
[0x941]ORI_EB_CFG_PORT0_FCOE_SA__0: 0x9b6803fc
[0x942]ORI_EB_CFG_PORT1_FCOE_DA:    0x00000000

[0x942]ORI_EB_CFG_PORT1_FCOE_DA__1: 0x00000000
[0x943]ORI_EB_CFG_PORT1_FCOE_DA__0: 0x00000000

[0x944]ORI_EB_CFG_PORT1_FCOE_SA:    0x0000aab1
[0x944]ORI_EB_CFG_PORT1_FCOE_SA__1: 0x0000aab1

[0x945]ORI_EB_CFG_PORT1_FCOE_SA__0: 0x9b6803fc
module-2#                       

4. FCoE enabled on VLAN (If FCoE packets don't get forwarded then check following)

 module-2# show ha internal dftm bd_table fe <asic inst> vlan <vlan>

 /*** Check VLAN which is operational according to "show vlan fcoe"
 Detailed Vlan BD Table info for FE : <asic inst>

 Port: 0 VLAN: <vlan> BDDB_INDEX: <BD> ACL_LABEL: 0
 Detailed Vlan BD Table info for FE : <asic inst>

 Port: 1 VLAN: <vlan> BDDB_INDEX: < BD> ACL_LABEL: 0
 module-2#

 module-2# show ha internal forwarding <asic inst> table bddb start_addr <BD> end_addr <BD>
              
                    BDDB TABLE SRAM ---------------------------

                                    (INST# 0)
Note: any fields marked with * are only valid in OrionCR

Note: any fields marked with * are only valid in OrionCR
[ 3]| mc_ftag_base     : 0 mc_ftag_num              : 1 ftag             : fff
[ 3]| ftag_m_null      : n ftag_u_null              : n ce_vlan          : dis 
/*** dis for MDS, enable on N7K ***/
[ 3]| def_gw_en        : dis def_gw                 : 0 age_unit         : 0
[ 3]| age_grp_sel      : 0 ipv6_pim_snp             : dis ipv6_mld_snp   : dis
[ 3]| ipv4_pim_snp     : dis ipv4_igmp_snp          : en pt_cam_en       : en 
[ 3]| drp_sa_miss      : n drp_da_miss              : y ntf_sa_miss_in   : y 
[ 3]| ntf_da_miss_in   : n fcoe_en                                       : y 
/*** Must be FCoE enabled ***/                        mc_miss_cp_sup     : n 
[ 3]| uc_miss_cp_sup   : n mc2_gen_en               : y use_ox_id        : y 
[ 3]| cp_mc3_to_sup    : n dl_pi                    : n dl_np            : n 
[ 3]| bd               : 8 en_proxy_learn           : n l3_mc_fwd_en     : n 
[ 3]| svi_en_ipv4      : n svi_en_ipv6              : n ntf_sa_miss_eg   : n 
[ 3]| dont_ff          : y hash_fcoe_lxr            : y in_if_chk_en     : y 
[ 3]| opt_mc_fwd_en    : y rbacl_en                 : y otv_en*          : n 
-----------------------------------------------------------------------
module-2#


5. FIB route table programming (If any fclid cam related drop found then check FIB entry)

http://wikicentral.cisco.com/confluence/display/GROUP/RIB_FIB_FOR_VFC


6. Zone ACL programming (If any ACL drop found then check zoning acls following way)

sw2-gd75# sh fcoe database

INTERFACE       FCID            PORT NAME               MAC ADDRESS

vfc21           0x020000        50:0a:09:84:89:fb:6c:4f 00:c0:dd:14:58:c7

vfc22           0x020001        21:00:00:c0:dd:10:f3:a5 00:c0:dd:10:f3:a5

sw2-gd75#

sw2-gd75# sh zoneset active

zoneset name zoneset-1 vsan 3839

zone name zone-12 vsan 3839

* fcid 0x020000 [pwwn 50:0a:09:84:89:fb:6c:4f]
* fcid 0x020001 [pwwn 21:00:00:c0:dd:10:f3:a5 ]

switch#attach mod <fcoe mod>

module# show ha internal forwarding asic <instance> tcam acl ingress start_addr
0x0 end_addr 0x3ff valid | inc 0x020000 (should show zoning acl entry)


7. Verify PAUSE enable/disable status (debugging congestion issue)

module-10# sh hardware internal mac port 1 qos configuration

QOS State for port 1 (Asic 0 Internal port 0)
GD

TX PAUSE:
VL#    ENABLE  RESUME  REFRESH  REF_PERIOD   QUANTA

     0     OFF     OFF     OFF        0x0        0x0
     1     OFF     OFF     OFF        0x0        0x0    

     2     OFF     OFF     OFF        0x0        0x0
     3     ON     ON    ON        0x0        0x0
     /*** PAUSE should be enabled on no drop VL ***/
     /*** (here VL 3 according to 7e template)  ***/
     4     OFF     OFF     OFF        0x0        0x0     

     5     OFF     OFF     OFF        0x0        0x0
     6     OFF     OFF     OFF        0x0        0x0    

     7     OFF     OFF     OFF        0x0        0x0
    LFC    OFF      ON      ON        0x1000     0xffff 


8. QoS MTU size checking (verify MTU size for no drop VL in case packet gets dropped due to MTU size)

module-2# show ha internal mac port <ethernet port number where VFC is bound> 
qos configuration | inc MTU

    VL 0-7 MTU: 1522 1522 1522 2180
    /*** should be set to 2180 for no drop VL ***/ 1522 1522 1522 1522
Port MTU (No dropping action): 9472

module-2#


9. Elam Capture (For other kind of drop verify packet captured through ELAM, check drop packet having 8906/8914)

switch# attach mod <number>

module# test ha internal forwarding <asic instance> elam trigger di/de drop (for ingress/egress packet drop)
module# show ha inforwarding <asic instance> elam capture (show look for packet drop with packet ether type 8906/8914 if there are any)


Packets should match inband packets:

Recommended Action:

You can verify that the packets match inband packets as shown the following example:

module-4# show hardware internal statistics device l2lu all

|------------------------------------------------------------------------|
| Device:Orion Fwding Driver      Role:L2                                |

| Last cleared @ Wed Jul 15 20:40:08 2009
|------------------------------------------------------------------------|

Instance:0
ID   Name                                          Value              Ports

-   ---                                          ---              ---
28   smallcnt ACL redirected pkts                  0000000000000703   1-2 -

2008 Ingress packets that don't have MIM           0000000000000703   1 -
2009 Ingress good packets sent to IB               0000000000000703   1 -

module-4# sh ha in statistics device mac all
|------------------------------------------------------------------------|

| Device:Orion MAC Driver         Role:MAC                               |
| Last cleared @ Wed Jul 15 20:40:09 2009

|------------------------------------------------------------------------|
Instance:0

ID   Name                                          Value              Ports
-   ---                                          ---              ---

0    IB num packets received from DI0              0000000000000703   1 -
13   IB num credited packets requested by VQ       0000000000000703   1-2 -

15   IB num supervisor packets requested by VQ     0000000000000703   1-2 -
18   IB num credited packets sent to FT            0000000000000703   1-2 -

22   IB num credited superframes sent to FT        0000000000000703   1-2 -
100  PL0 Ingress RTOT pkts VL0                     0000000000000703   1 -

108  PL0 Ingress RFCOE VL0                         0000000000000703   1 -
116  PL0 Ingress ROCT octets VL0                   0000000000126440   1 -

124  PL0 Ingress RFCOE octets VL0                  0000000000126440   1 -
156  PL0 Ingress RTOT pkts unicast VL0             0000000000000703   1 -

790  GD Total received frames                      0000000000000703   1 -
874  GD Shim rx count                              0000000000000703   1 -

881  GD Post decryption, accept uncontrolled       0000000000000703   1 -
883  GD Received frames w/ len of (64, 127]        0000000000000001   1 -

884  GD Received frames w/ len of (127, 255]       0000000000000702   1 -
954  GD GMAC transition from nosync to sync int    0000000000000001   1 -


ACL:

module-4# show system internal access-list input entries detail
/*** this should have hardware address programmed by fcoe mgr ***/
Flags: F - Fragment entry  E - Port Expansion

       D - DSCP Expansion  M - ACL Expansion

take the hardware address from the above and use it in the next show command, example used is 4B

module-4# show hardware internal forwarding 0 tcam acl ingress start_addr 0x4B end_addr 0x4B

--- ACL TCAM/RAM{}- (INST#  0, ingress) ---
[4b]| acl type                  :            3 (          l2)

[4b]| key_type                  :                0/               3
[4b]| key_sub_type              :                3/               3
[4b]| vlan                      :                1/             fff /*** verify this
[4b]| port_index                :                0/               1 /*** verify this
[4b]| ethertype                 :             8914/            ffff /*** verify this
[4b]| lkup_cnt                  :                1/               3
[4b]| acl_result_type           :            0
[4b]| priority                  :            2
[4b]| permit_vld                :            0
[4b]| vl_vld                    :            0
[4b]| vlan_ftag_vld             :            0
[4b]| ftag_vld                  :            0
[4b]| reason_code_vld           :            1 /*** verify this should be 1
[4b]| redirect_vld              :            1 /*** verify this should be 1
[4b]| fc_d_id_vld               :            0
[4b]| fc_s_id_vld               :            0
[4b]| fc_lun_vld                :            0
[4b]| oda_ooo_vld               :            0
[4b]| osa_dl_vld                :            0
[4b]| dscp_vld                  :            0
[4b]| cap_eth                   :            0
[4b]| cap_fc                    :            0
[4b]| permit                    :            0
[4b]| dont_cnt_rpkt             :            0
[4b]| vl                        :            0
[4b]| add_bcnai                 :            0
[4b]| vlan                      :            0
[4b]| ftag                      :            0
[4b]| reason_code               :           41 /*** hex, in dec - 65 - FIP snooping code
[4b]| mp_table_prt              :          402 /*** hex, ptr - this has the add to sup
[4b]| mp_table_num              :            1 /*** multi-path table number
[4b]| dest_fcf                  :            0
[4b]| dc3vxhdr_rdt              :            0
[4b]| fc_d_id                   :            0
[4b]| fc_s_id                   :            0
[4b]| fc_lun                    :            0
[4b]| oda_ooo                   :            0
[4b]| osa_dl                    :            0
[4b]| dscp                      :            0

module-4# show hardware internal forwarding 0 table mp start_addr 0x402 end_addr 0x402

------------ MULTIPATH TABLE SRAM -----------
----------------- (INST# 0) -----------------
|  Index  | cp_to_sup1 | cp_to_sup2 |  lid_idx  |
|-----------------------------------------------|
|   [402] |        0   |        0   |        0  |
the lid_idx is the value to check, if this entry is wrong it won't go to sup
module-4# show hardware internal forwarding 0 acl-statistics ingress  /*** this command should 
have an entry for hardware address programmed by fcoe mgr, then pktmgr will receive the frames
Instance: 0, direction: 1 


------------------ ACL Statistics ------------------
(Inst #0, Direction Ingress)
----------------------------------------------------------
 Index Frames Bytes
[0002] 000000000000169e 00000000000fe5ec

INBAND - check the atype=8 value, should match linecard value:
switch# show hardware internal sup-fc0 atype-pkt-count
ATYPE:            Pkt Count:
0                 0x12
1                 0x534
2                 0x0
3                 0x0
4                 0x0
5                 0x0
6                 0x0
7                 0x0
8                 0x2bf
9                 0x0
10                0x0
11                0x0
12                0x0
13                0x0
14                0x0
15                0x0<span id="fck_dom_range_temp_1362780923029_253" />
switch# sh ha in sup-fc0 dicounts 0 /*** this is for ethernet 4/1
Hi Prio Voq Enabled:
Hi Prio Voq Status:0x1
Lo Prio Voq Enabled:
Lo Prio Voq Status:0x1
Enabled: Yes
tx di dma frame cnt            0
rx di dma frame cnt            356
rx di frame cnt                356
Packets removed when port went down: 0
Packets removed when hi voq down: 0
Packets removed when lo voq down: 0
Packets dropped as voq is large: 0
Hi Packets dropped because port was down: 0
Lo Packets dropped because port was down: 0
Packets dropped when lo voq was down: 0
Packets dropped when hi voq was down: 0
Total packets successfully sent for this di:0
Total Hi Priority Pkts queued for SW Tx for this di: 0
Total Hi Priority Pkts queued for HW Tx for this di: 0
Total Low Priority Pkts queued for SW Tx for this di: 0
Total Low Priority Pkts queued for HW Tx for this di: 0
Total packets received from this si:703
Total packets received by this di:0
Packets waited for long time in Tx queue:0
Old packets dropped after Rx:0
RX Bad DI:0
TxHi0: Head:cf8c0020 (0x1f8c0020) HW:cf8c0020 (0x1f8c0020) Tail:cf8c0020 (0x1f8c
0020) HW:cf8c0020 (0x1f8c0020)
cf8c0020 (0x1f8c0020) next:00000000(0x0) buf:0x0 tl:0 st:0x0 bs:0 pkt:0x0
TxLo0: Head:cf8c8020 (0x1f8c8020) HW:cf8c8020 (0x1f8c8020) Tail:cf8c8020 (0x1f8c
8020) HW:cf8c8020 (0x1f8c8020)
cf8c8020 (0x1f8c8020) next:00000000(0x0) buf:0x0 tl:0 st:0x0 bs:0 pkt:0x0
 
NETSTACK/PKTMGR - check the rx_recv in the driver and then for client 8914 check the transmit:
switch# sh system internal pktmgr driver
 
--------------------------------------------
Driver: bc-fcoe
--------------------------------------------
    
   State:               Up
   Filter:              0xeeee
   fd:                  15
   Enabled:             Yes
   Discard In:          0
   Discard Out:         0
   No of Ifs:           1
   Tx lot:              128
   Reader:              1
   Writer:              1
   Hdr len:             96
   tx_sent:             0
   tx_rate:             0
   tx_max_rate:         0
   rx_recv:             702
   rx_rate:             0
   rx_max_rate:         0
switch# sh system internal pktmgr client
Client uuid: 124, 1 filters
  Filter 0: EthType 0x2000, Snap 8192, Dmac 0100.0ccc.cccc
  Options: TO 0, Flags 0x1, AppId 0, Epid 0
  Ctrl SAP: 946, Data SAP 363 (1)
Counters:
   rx:                               0
   rx_rate:                          9187201950444158976
   tx:                               64
   tx_rate:                          9187201948296708096
Client uuid: 601, 1 filters
  Filter 0: EthType 0x88cc,  Dmac 0180.c200.000e
  Options: TO 0, Flags 0x1, AppId 0, Epid 0
  Ctrl SAP: 456, Data SAP 456 (1)
Counters:
   rx:                               0
   rx_rate:                          9187201950444158976
   tx:                               128
   tx_rate:                          9187201948296675456
Client uuid: 699, 1 filters
  Filter 0: EthType 0x8914,
  Options: TO 0, Flags 0x1, AppId 0, Epid 0
  Ctrl SAP: 473, Data SAP 473 (1)
Counters:
   rx:                               0
   rx_rate:                          9187201950444158976
   tx:                               0
   tx_rate:                          9187201948305063936
Number of clients: 3
debugs of help - debug pktmgr frame
 
FCoE MGR:
 
FC2 - make sure there are no drops in FC2. If there are drops check what these drops are:
switch# sh fc2 port brief
       0 fc2_very_bad_frames_rx
 IX ST MODE EMUL   TXPKTS   TXDROP    TXERR   RXPKTS   RXDROP
  0  D    1    0        0        0        0        1        1
 10  U    1    0      678        0        0        1        0
 11  U    1    0        1        0        0        1        0
 12  U    1    0        1        0        0        1        0
 13  U    1    0        1        0        0        1        0
 14  U    1    0        1        0        0        1        0
 15  U    1    0        1        0        0        1        0
 16  U    1    0        1        0        0        1        0
 17  U    1    0        1        0        0        1        0
 18  U    1    0        1        0        0        1        0
 19  U    1    0        1        0        0        1        0
 1a  U    1    0        1        0        0        1        0
 1b  U    1    0        1        0        0        1        0
 1c  U    1    0        1        0        0        1        0
 1d  U    1    0        1        0        0        1        0
 1e  U    1    0        1        0        0        1        0
 1f  U    1    0        1        0        0        1        0
 20  U    1    0        1        0        0        1        0
 21  U    1    0        1        0        0      678        0
 fe  U    1    0      352        0        0        0        0
 ff  U    1    0      351        0        0      703        0
 
PIXM related show commands -
sh system internal pixm info interface ethernet 4/1
sh system internal pixm info piod-table
show sys int pixm info ltl-type phy
ETHPM related show commands -
show system internal ethpm info interface ethernet 4/1
 show system internal ethpm event-history errors

Troubleshooting – Hardware Errors

Recommended Action:

The following helps to troubleshoot hardware errors:

  • “show hardware internal errors module <module-number>” will print any errors which may cause the packet to be dropped
  • CLI can be run from the sup or alternately run on the module after attaching to the module


module-6# show hardware internal errors

|------------------------------------------------------------------------|
| Device:Clipper MAC Role:MAC Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:34 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
|------------------------------------------------------------------------|
| Device:Clipper XBAR Role:QUE Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:35 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|

|------------------------------------------------------------------------|
| Device:Clipper FWD Role:L2 Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:34 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
|------------------------------------------------------------------------|
| Device:Clipper L3 Driver Role:L3 Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:35 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|

|------------------------------------------------------------------------|
| Device:Sacramento Xbar ASIC Role:FABRIC Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:35 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
|------------------------------------------------------------------------|
| Device:Clipper MAC Role:MAC Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:34 2012
| Device Statistics Category :: CONGESTION
|------------------------------------------------------------------------|

|------------------------------------------------------------------------|
| Device:Clipper XBAR Role:QUE Mod: 6 |
| Last cleared @ Tue Apr 24 04:13:35 2012
| Device Statistics Category :: CONGESTION
|------------------------------------------------------------------------|
module-6#


Troubleshooting – IdxMap verification

Recommended Action:

The following helps to troubleshoot idxmap verification:

  • IDXMap is a kernel table used by FC-components to look up all FC and ethernet interfaces and the corresponding Dis
  • “show system internal fcfwd idxmap interface-to-port” should have
    • Sup-fc0 interface details
    • All ethernet interfaces allocated to the storage VDC and the corresponding DI
    • Ethernet port-channel interfaces and corresponding Dis


sw8-gd76-sup2-storage-vdc# show system internal fcfwd idxmap interface-to-port

Interface to Port Table:(All values in hex)

------------------------------------------------------------------------
                         |  |     |  |lcl|   | S | P |     |     |     |
                         |no|     |  |idx|if | L | O |     |     |     |       
                         |of|     |  |   |ty-| O | R | mts | port| if  |
        if_index         |GI|  GI |VL|LCI|pe | T | T | node| mode|flags|
------------------------------------------------------------------------
16000108 Epo265          | 1|  406| 0| 00| 16| 00| 00| 0101| 0000| 0000|
06081000 sup-fc0         | 1| 10ea| 0| 01| 06| 00| 01| 0101| 0000| 0000|
160002bb Epo700          | 1|  402| 0| 00| 16| 00| 00| 0101| 0000| 0000|
1a200000 Ethernet5/1     | 1|    0| 0| 00| 1a| 04| 00| 0502| 0000| 0000|
1a201000 Ethernet5/2     | 1|    1| 0| 01| 1a| 04| 01| 0502| 0000| 0000|
1a202000 Ethernet5/3     | 1|    2| 0| 02| 1a| 04| 02| 0502| 0000| 0000|
1a203000 Ethernet5/4     | 1|    3| 0| 03| 1a| 04| 03| 0502| 0000| 0000|


Troubleshooting – MPMap verification

Recommended Action:

The following helps to troubleshoot mpmap verification:

  • IDXMap is a kernel table which stores the VFC to Ethernet binding information
  • “show system internal fcfwd mpmap vfcs” should have all the VFCs which have a Ethernet interface binding


sw8-gd76-sup2-storage-vdc# show system internal fcfwd mpmap vfcs

                          FCoE VFC Information

----------------------------------------------------------------------------

    ID     | if-index |S|M|T|          Members              |      MAC(s)

------------+----------+-+-+-+-------------------------------+------------------

vfc-po700 |0x1e1002bb|D|E|-|0x160002bb|eth-pc 700 (U)*|64:a0:e7:44:8e:81

           |          | | | |          |                    |00:26:98:04:62:82

------------+----------+-+-+-+-------------------------------+------------------

vfc6/16 |0x1e228f00|D|E|-|0x1a28f000|Ethernet6/16 (U)*|6c:9c:ed:43:35:a7

           |          | | | |          |                    |58:8d:09:7c:d8:e7

------------+----------+-+-+-+-------------------------------+------------------

vfc-po70 |0x1e100045|D|E|-|0x16000045|eth-pc 70 (U)*|64:a0:e7:44:8e:82

           |          | | | |          |                    |00:26:98:04:62:83

------------+----------+-+-+-+-------------------------------+------------------

vfc5/10 |0x1e220900|D|F|-|0x1a209000|Ethernet5/10 (U)*|cc:ef:48:eb:62:21

------------+----------+-+-+-+-------------------------------+------------------

vfc526 |0x1e00020d|D|E|-|0x1a219000|Ethernet5/26 (U)*|cc:ef:48:eb:62:31

           |          | | | |          |                    |68:ef:bd:a8:31:f3

------------+----------+-+-+-+-------------------------------+------------------




Troubleshooting – FIP BPDU Classification

Recommended Action:

To verify whether the PL block BPDU classification for the FIP is enabled use the following CLI (in this case I use port 6/16 as an example)

 module-6# show hardware internal clm bpdu port 16 action

in parse

token = 285212929

token = 130

#-Type | MAC - (mask)| EthType - (mask)|

-------|------------------- - ---- |------------ -----|

 0 F  | 01:80:c2:00:00:c0  (0x0)  |  0x0        (0x1)|
 1 F  | 01:00:0c:cc:cc:c0  (0x0)  |  0x0        (0x1)|
 2 F  | 01:00:0c:cd:cd:d0  (0x0)  |  0x0        (0x1)|
 3 F  | 00:00:00:00:00:00  (0x1)  |  0x22f4     (0x0)|
 4 F  | 00:00:00:00:00:00  (0x1)  |  0x8914     (0x0)|
 5 F  | 00:00:00:00:00:00  (0x0)  |  0x0        (0x1)|
 6 F  | 00:00:00:00:00:00  (0x0)  |  0x0        (0x1)|
 0 R  | 00:18:0c:20:00:00  (0x0)  |  0x0        (0x1)|
 1 R  | 00:18:0c:20:00:02  (0x0)  |  0x0        (0x1)|
 0 U  | 00:00:00:00:00:00  (0x0)  |  0x0        (0x1)|

-------|------------------- - ---- |------------ -----|



==

Troubleshooting – SMM FIP Redirect Criteria

Recommended Action:

To check the SMM table for the FIP entries first check the match criteria and based on the index where you find the match for 0x8914 lookup the equivalent action match

module-6# show hardware internal forwarding inst 3 table smm-mcr index 1

 

-------------------------------

         SMM MCR ENTRY
           (INST# 3)

-------------------------------

trill_encap (trill packet encap)  : 0x0

trill_encap_en (enable trill encap match) : 0x0

mim (mim packet)  : 0x0

mim_en (enable mim packet match)  : 0x0

pg (port group value)  : 0x0

pg_en (enable port group match)  : 0x0

prot_type (ip protocol type)  : 0x0

prot_type_en (enable ip protocol match)  : 0x0

src_prot (source protocol)  : 0x0

src_prot_en (enable source protocol match): 0x0

m_src (1=match src mac, 0=match dst mac)  : 0x0

m_src_dst_en (enable match src or dst mac): 0x1

mac_mask (mask off dst mac bits)  : 0x0

mac_neg (trigger if no mac match)  : 0x0

mac_addr_W11 (upper 16 bits of mac addr)  : 0x0

mac_addr_W10 (middle 16 bits of mac addr) : 0x0

mac_addr_W0 (lower 16 bits of mac addr)  : 0x0

use_outer_mac (if set match outer dst mac): 0x0

use_mac (use mac addr matching)  : 0x0

vdc_mask (mask for vdc profile)  : 0x0

vdc (vdc value to match)  : 0x0

vdc_en (enable vdc match)  : 0x0

bd (bridge domain value)  : 0x0

bd_en (enable bridge domain match)  : 0x0

bpdu (bpdu bit to match)  : 0x0

bpdu_en (enable bpdu match)  : 0x0

sm_en (match sm_en field in lbd entry)  : 0x7

etype (match with ether type)  : 0x8914

etype_en (enable ether type match  : 0x1

port_mask (match on port mask)  : 0x1f



==

Troubleshooting – SMM FIP Redirect –Action

Recommended Action:

Verify SMM Action matching the criteria register for the 0x8914 ether-type frames

module-6# show hardware internal forwarding inst 3 table smm-mar start 0x1 end 0x1

-----------------------------------------------------------

------------- Static MAC Match Action Table --------------

                      (INST# 3)                    

-----------------------------------------------------------

[1]| flood_pkt : n no_mac_sa_lrn : y

[1]| redir_pkt : y redir_pkt_if_idx : 3ff

[1]| mim_vl : 3 repl_mim_vl : y

[1]| pkt_is_bpdu : y

[1]| copy_to_sup1 : n copy_to_sup2 : n

[1]| new_acos : 5 use_new_acos : y

[1]| l3_disable : n l3_bypass_en : n

 

flood_pkt = Flood to BD, redirect and DI will be ignored

no_mac_sa_lrn = Don't learn if static match

redir_pkt = Redirect

redir_pkt_if_idx = Redirect DI or final DI if redirect not set

mim_vl = New COS

repl_mim_vl = If set use new COS, else preserve header COS

pkt_is_bpdu = BPDU packet

copy_to_sup1 = CAP1

copy_to_sup2 = CAP2

new_acos = New ACOS

use_new_acos = Use new ACOS to replace existing ACOS

l3_disable = L3 disable

l3_bypass_en = Enable L3 bypass when static match redirect

 

The SMM redirect entry is programmed for 0x8914 with with the redirect DI being 0x3ff. Look up the idxmap to make sure this is the sup-fc0 DI

 

module-6# show system internal fcfwd idxmap interface-to-port | inc 3ff

06081000 sup-fc0         | 1|  3ff| 0| 01| 06| 00| 01| 0101| 0000| 0000|



Troubleshooting – FMM (FCoE MAC Match)

Recommended Action:

To check the FMM registers for port 6/16 the ASIC instance is 3

(ports 13-16) and the register instance is 3 (port13=0, 14=1, 15=2, 16=3)

module-6# show hardware internal clp_l3 inst 3 registers name CLP_LBD_FMM_COND_REG_3

 

+-------------------------------------------------------------------------------

| Instance Registers for Clipper L3 Driver

| Inst 3; port(s) 13-16

|

 

ADDR(0x) REG NAME VAL(0x)

[POS] FLD_NAME

________ ________ ________

   1a0a  CLP_LBD_FMM_COND_REG_3__2                                     00000010
   1a0b  CLP_LBD_FMM_COND_REG_3__1                                     154e493e
   1a0c  CLP_LBD_FMM_COND_REG_3__0                                     2ac00011

[00:00] MATCH_EN 1

[05:01] PORT_MASK 8

[06:06] BD_EN 0

[20:07] BD 0

[21:21] USE_ODA 0

[69:22] MAC_DA 4055

                                                                       3924f8ab

[74:70] MAC_DA_MASK 0

[75:75] PG_EN 0

[85:76] PG 0

[86:86] MCX_EN 0

 

 

The port mask should be 8 (0x1000). In case of the port 6/13 the port-mask would have been 1(0x0001) for 6/14 the port-mask value should be 2(0x0010) and so on.

 

MAC DA value here should be the same as the FCF-MAC. This should match the FCF-MAC value in the mpmap tables for the VE ports and FPMA address for the VF ports.



Troubleshooting – Port-Type (ILM Table)

Recommended Action:


  1. To get the index in the ILM table (in this case we are looking for the vfc 6/16 and vlan 3 as an example )

 

sw8-gd71-sup2-storage-vdc# show vlan fcoe

 

Original VLAN ID Translated VSAN ID Association State

---------------- ------------------ -----------------

 

     3                         3                Operational

module-6# show system internal iftmc hardware interface ethernet 6/16 vdc 2

 

Interface    L2/L3 GBL  Part CPC 

------------- ----- ---- ---- ---

Eth6/16       2    0    1    0  

 

 Inst In Use     : 0xfad (1111 1010 1101) 
 Inst To Del     : 0xfad (1111 1010 1101) 
 Intf Inst       : 0x008 (0000 0000 1000) 

 

 Act VLANs :   3
 Del VLANs :   3

  

Vlan : 3 mem : 2 flag : 0x0000


Inst   PG    TCAM    LDB    ILM    ELM    Ref   Flag 

------ ------ ------ ------ ------ ------ ------ -----

0     0x2    0x10   0xf    0x40   0xe     16     0x1   
2     0x2    0x10   0xf    0x40   0xe     16     0x1   
3     0x7f   0x12   0x800f 0x42   0x800e  1      0x1   
5     0x2    0x10   0xf    0x40   0xe     16     0x1   
7     0x2    0x10   0xf    0x40   0xe     16     0x1   
8     0x2    0x10   0xf    0x40   0xe     16     0x1   
9     0x2    0x10   0xf    0x40   0xe     16     0x1   
10    0x2    0x10   0xf    0x40   0xe     16     0x1   
11    0x2    0x10   0xf    0x40   0xe     16     0x1   

--------------------------------------------------------



  1. For port 6/16 we need to look at inst 3 in the list above and get the ILM index.


module-6# show hardware internal forwarding inst 3 table ilm index 0x42

 -------------------------------

          ILM ENTRY
          (INST# 3)

-------------------------------

[ 66] outer_de_vld  : 0x0

[ 66] otv_p2p  : 0x0

[ 66] fcoe_en  : 0x1

[ 66] mac_ip_chk_arp  : 0x0

[ 66] v6_rpf_mode  : 0x0

[ 66] v6_vpn  : 0x1

[ 66] v6_rpfv3_en  : 0x0

[ 66] v6_sgt_prio  : 0x0

[ 66] v6_dgt_prio  : 0x0

[ 66] ids_mask_sel  : 0x0

[ 66] per_pkt_ls_en  : 0x0

[ 66] no_stats  : 0x0

[ 66] v4_sgt_prio  : 0x3

[ 66] v4_dgt_prio  : 0x1

[ 66] diag  : 0x0

[ 66] df_mask  : 0x0

[ 66] non_ipv6_vpn  : 0x1

[ 66] v4_rpfv3_en  : 0x0

[ 66] ipv4_en  : 0x0

[ 66] ipv4_mcast_en  : 0x0

[ 66] v4_rpf_mode  : 0x0

[ 66] ipv6_mcast_en  : 0x0

[ 66] mac_ip_chk_ipv4  : 0x0

[ 66] mac_ip_chk_ipv6  : 0x0

[ 66] ipv6_en  : 0x0

[ 66] cpp_en  : 0x1

[ 66] acl_en  : 0x3

[ 66] qos_en  : 0x0

[ 66] acos_sel  : 0x1

[ 66] label_b  : 0x7f1

[ 66] label_a  : 0x7f1

[ 66] ls_hash_sel  : 0x0

[ 66] base_pol_id  : 0x0

[ 66] mut_map_index  : 0x0

[ 66] ilm_trig  : 0x0


There are 8 reserved labels ranging from 0x7f0 to 0x7f7, each one corresponding to a FC port-type.

 

Here’s how to interpret the port-type (the label value)

 

PORT_FC_PORT_TYPE_TE 0x7f1 // FC Port mode TE (VE port)

PORT_FC_PORT_TYPE_F 0x7f2 // FC Port mode F (VF port)


FIB verification – Verifying routes

Recommended Action:

To verify routes:

module-4# sh fib unicast

 

IOD Mode: in order mode Mod  : #adjacency

All fields in hex except VSAN

 

Route IOD Fwd | Adj Met Adj Adj

Type Mode VSAN D ID Idx | Mod Idx Idx Mark St VDC FE

---- ---- ---- ------ ---- +--- ----- ----- ---- --- -- ---

wka idle 0 fffff5 b7ea | 1 00000 00059 0009 000 03 00

wka idle 0 fffff5 b7ea | 1 00000 00059 0009 000 03 0b

wka idle 0 fffff6 b7ec | 1 00001 0005a 0009 000 03 00

wka idle 0 fffff6 b7ec | 1 00001 0005a 0009 000 03 0b

wka idle 0 fffff7 b7ee | 1 00002 0005b 0009 000 03 00

wka idle 0 fffff7 b7ee | 1 00002 0005b 0009 000 03 0b

wka idle 0 fffff8 b7f0 | 1 00003 0005c 0009 000 03 00

wka idle 0 fffff8 b7f0 | 1 00003 0005c 0009 000 03 0b

wka idle 0 fffff9 b7f2 | 1 00004 0005d 0009 000 03 00

wka idle 0 fffff9 b7f2 | 1 00004 0005d 0009 000 03 0b

wka idle 0 fffffa b7f4 | 1 00005 0005e 0009 000 03 00

wka idle 0 fffffa b7f4 | 1 00005 0005e 0009 000 03 0b

wka idle 0 fffffb b7f6 | 1 00006 0005f 0009 000 03 00

wka idle 0 fffffb b7f6 | 1 00006 0005f 0009 000 03 0b

wka idle 0 fffffc b7f8 | 1 00007 00060 0009 000 03 00

wka idle 0 fffffc b7f8 | 1 00007 00060 0009 000 03 0b

wka idle 0 fffffd b7fa | 1 00008 00061 0009 000 03 00

wka idle 0 fffffd b7fa | 1 00008 00061 0009 000 03 0b

wka idle 0 fffffe b7fc | 1 00009 00062 0009 000 03 00

wka idle 0 fffffe b7fc | 1 00009 00062 0009 000 03 0b

local idle 1 ceffff .... | 0 00000 deadbeef 0009 000 03

local idle 2 112233 b666 | 1 0000e 00064 0000 000 03 00

local idle 2 112233 b666 | 1 0000e 00064 0000 000 03 0b

local idle 2 c1ffff b5fc | 1 0000a 00063 0009 000 03 00

local idle 2 c1ffff b5fc | 1 0000a 00063 0009 000 03 0b

ucast idle 2 eb0000 b3d6 | 1 0000f 00065 0000 000 03 00

ucast idle 2 eb0000 b3d6 | 1 0000f 00065 0000 000 03 0b



From the given command, the following information can be obtained:

  • The Route Type indicates the type as before - “wka/local/Remote etc”
  • The presence of routes in each clipper (FE) as indicated in the FE field. If the FE field is blank, as in the case of FCID 0xceffff with VSAN 1, it is because the route is not programmed in h/w (also note Fwd Idx is “....” and Adj Idx is “deadbeef”
  • Fwd Idx – Specifies the FIB TCAM address as well as the FIB UC Result DRAM address. The RDRAM stores the LS-MET base address and num paths (-1 actually)
  • Adj Mod – Specifies the number of paths (VE links / ISLs). This will be 1 for local routes.
  • Met Idx – Specifies the Multipath base address (similar to Mp table base addr in Orion). One difference with Orion is that there is an additional table which has the “bundle of Adj pointers” called the LS-MET. When there are multiple paths to a destinations, the RDRAM points to the LS-MET base address and a load balance of the link take place at LS-MET level. The LS-MET entry has the “adj_ptr” which tells where the ADJ is in the ADJ Table as well as where the Rewrite Entry is in the RIT table.
  • Adj Idx – Specifies the Adjacency Index when there is only ONE path to the destination. When there is only one path to destination, the RDRAM directly encodes the “adj_ptr” instead of the LS-MET base address. VDC Id is obtained from VDC field.

Rating: 5.0/5 (3 votes cast)

Personal tools