Finding Packet Drops - ASR9K

From DocWiki

Jump to: navigation, search

Since the ASR9K is a distributed platform with many modular parts that provide output to other modular parts, there are many places for packets to be potentially dropped.

To first confirm the ASR9K is dropping packets use Hardware ACLs to count packets on ingress and egress.

Contents

Strategy

Find the packet rate

Using this guide, Hardware Packet Counting - ASR9K, it should be possible to know the packet rate. That makes finding the incrementing counter much easier.

  • Is the rate constant or bursty?
  • For a given interval, 30 seconds, 5 minutes, 10 minutes, what is the minimum, average, and maximum rate

Can you stop and start the flow

This will help you find the counter, since it likely only increments when traffic is flowing.

The Easy Way

Requires IOS-XR 4.3.2+

Transit: Does not check the PHY.

Transit: Does not check for ASIC errors.

For-US: Does not check LPTS.

For-US: Does not check Netio.

For-US: Does not check SPP.

For-US: Does not check the punt switch.

RP/0/RSP0/CPU0:ASR9000# show drops 

Use the ongoing keyword to show what changed since the command was last run.

RP/0/RSP0/CPU0:ASR9000# show drops all ongoing location 0/0/CPU0
Found grammer file packet_drops.list at /disk0a:/usr/. Using this file instead of default

=====================================
Checking for ongoing drops on 0/0/CPU0
=====================================

show arp traffic:
[arp:ARP] IP Packet drop count for node 0/0/CPU0: +52

show cef drops:
[cef:0/0/CPU0] No route drops       packets : +1352

show netio drops:
[netio:Interface: FINT0/0/CPU0] /pkg/lib/libl2_adj_fint_netio.dll: +1351

show controller np counters:
[np:NP0] PARSE_ING_DISCARD: +10824
[np:NP0] PARSE_ING_L3_CFM_DROP: +5412
[np:NP0] UNKNOWN_L2_ON_L3_DISCARD: +5412
[np:NP0] RSV_DROP_IN_L3_NOT_MYMAC: +5413
[np:NP0] ARP_EXCD: +108238
[np:NP0] MDF_PUNT_POLICE_DROP: +108237

Show drops checks these components.

RP/0/RSP0/CPU0:ASR9000# show drops all commands 
Module                    CLI            
[arp]                     show arp traffic
[cef]                     show cef drops 
[fabric]                  show controllers fabric fia drops egress
[fabric]                  show controllers fabric fia drops ingress
[lpts]                    show lpts pifib hardware entry statistics
[lpts]                    show lpts pifib hardware police
[lpts]                    show lpts pifib hardware static-police
[netio]                   show netio drops
[netio]                   fwd_netio_debug
[niantic-driver]          show controllers dmac client punt statistics
[np]                      show controller np counters
[np]                      show controllers np tm counters all
[spp]                     show spp node-counters
[spp]                     show spp client detail
[spp]                     show spp ioctrl

How to Trace Drops through the Router - Dataplane

Architecture

ASR9K Architecture.jpg

ASIC Errors

RP/0/RSP0/CPU0:ASR9000# show asic-errors all location 0/0/CPU0
************************************************************
*                  Fia ASIC Error Summary                  *
************************************************************
Instance            : 0
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 1
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
Instance            : 2
Number of nodes     : 0
SBE error count     : 0
MBE error count     : 0
Parity error count  : 0
CRC error count     : 0
Generic error count : 0
Reset error count   : 0
--------------------
<snip>

CEF

RP/0/RSP0/CPU0:ASR9006-B# show cef drops 
 CEF Drop Statistics
 Node: 0/RSP0/CPU0
  Unresolved drops     packets :               0
  Unsupported drops    packets :               0
  Null0 drops          packets :             333
  No route drops       packets :             105
  No Adjacency drops   packets :               0
  Checksum error drops packets :               0
  RPF drops            packets :               0
  RPF suppressed drops packets :               0
  RP destined drops    packets :               0
  Discard drops        packets :               0
  GRE lookup drops     packets :               0
  GRE processing drops packets :               0
  LISP punt drops      packets :               0
  LISP encap err drops packets :               0
  LISP decap err drops packets :               0
 Node: 0/RSP1/CPU0
  Unresolved drops     packets :               0
  Unsupported drops    packets :               0
  Null0 drops          packets :               0
  No route drops       packets :               0
  No Adjacency drops   packets :               0
  Checksum error drops packets :               0
  RPF drops            packets :               0
  RPF suppressed drops packets :               0
  RP destined drops    packets :               0
  Discard drops        packets :               0
  GRE lookup drops     packets :               0
  GRE processing drops packets :               0
  LISP punt drops      packets :               0
  LISP encap err drops packets :               0
  LISP decap err drops packets :               0
 Node: 0/0/CPU0
  Unresolved drops     packets :               0
  Unsupported drops    packets :               0
  Null0 drops          packets :               0
  No route drops       packets :               0
  No Adjacency drops   packets :               0
  Checksum error drops packets :               0
  RPF drops            packets :               0
  RPF suppressed drops packets :               0
  RP destined drops    packets :               0
  Discard drops        packets :              19
  GRE lookup drops     packets :               0
  GRE processing drops packets :               0
  LISP punt drops      packets :               0
  LISP encap err drops packets :               0
  LISP decap err drops packets :               0

PHY

RP/0/RSP0/CPU0:ASR9000# show controllers tenGigE 0/0/0/0 stats | e pkts   
Statistics for interface TenGigE0/0/0/0 (cached values):

Ingress: 
    Input total bytes           = 58408547
    Input good bytes            = 58408547

    Input total packets         = 283327
    Input 802.1Q frames         = 0
    Input pause frames          = 0


    Input drop overrun          = 0
    Input drop abort            = 0
    Input drop invalid VLAN     = 0
    Input drop invalid DMAC     = 0
    Input drop invalid encap    = 0
    Input drop other            = 0

    Input error giant           = 0
    Input error runt            = 0
    Input error jabbers         = 0
    Input error fragments       = 0
    Input error CRC             = 0
    Input error collisions      = 0
    Input error symbol          = 0
    Input error other           = 0

    Input MIB giant             = 0
    Input MIB jabber            = 0
    Input MIB CRC               = 0

Egress:
    Output total bytes          = 298287727
    Output good bytes           = 298287727

    Output total packets        = 692526
    Output 802.1Q frames        = 0
    Output pause frames         = 0


    Output drop underrun        = 0
    Output drop abort           = 0
    Output drop other           = 0

    Output error other          = 0

NPU - Network Processing Unit

See which NPU is being used

RP/0/RSP0/CPU0:ASR-9000# show controller np ports all loc 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/9
1 -- 1 GigabitEthernet0/0/0/10 - GigabitEthernet0/0/0/19
2 -- 2 TenGigE0/0/1/0
3 -- 3 TenGigE0/0/1/1

Check the NP counters

The NP counts all kinds of packets, only some of the counters are drops.

RP/0/RSP0/CPU0:ASR9000# show controllers np counters np0 location 0/0/CPU0

                Node: 0/0/CPU0:
----------------------------------------------------------------

Show global stats counters for NP0, revision v2

Read 43 non-zero NP counters:
Offset  Counter                                         FrameValue   Rate (pps)
-------------------------------------------------------------------------------
  16  MDF_TX_LC_CPU                                        1967821           5
  17  MDF_TX_WIRE                                           194180           1
  21  MDF_TX_FABRIC                                         107563           0
  29  PARSE_FAB_RECEIVE_CNT                                 107563           0
  33  PARSE_INTR_RECEIVE_CNT                               5296977          15
  37  PARSE_INJ_RECEIVE_CNT                                 205892           1
  41  PARSE_ENET_RECEIVE_CNT                                326160           1
  45  PARSE_TM_LOOP_RECEIVE_CNT                            1768343           5
  57  PARSE_ING_DISCARD                                        105           0
  63  PRS_HEALTH_MON                                       1768343           5
  72  INTR_FRAME_TYPE_7                                    1647061           5
  98  PARSE_ING_IPV6_LINK_LOCAL                               1914           0
 102  PARSE_RSP_INJ_FAB_CNT                                  94946           0
 105  PARSE_EGR_INJ_PKT_TYP_UNKNOWN                          18250           0
 106  PARSE_EGR_INJ_PKT_TYP_IPV4                             26544           0
 110  PARSE_EGR_INJ_PKT_TYP_IPV4_PREROUTE                    50152           0
 116  PARSE_LC_INJ_FAB_CNT                                   94948           0
 117  PARSE_LC_INJ_PORT_CNT                                 104871           0
 118  PARSE_LC_INJ_DIAGS_CNT                                  6073           0
 129  PARSE_DROP_IN_UIDB_TCAM_MISS                               3           0
 132  PARSE_DROP_IN_UIDB_DOWN                                   31           0
 201  PARSE_LLDP                                             34285           0
 203  UNKNOWN_L2_ON_L3_DISCARD                                  74           0
 220  PARSE_UDLD                                            133797           0
 260  DBG_RSV_EP_L_RSV_ING_L3_IFIB                          128274           0
 261  DBG_RSV_EP_L_RSV_ING_L3_IFIB_MATCH                    128274           0
 262  DBG_RSV_EP_L_RSV_ING_L3_IFIB_PUNT_LOCAL               128274           0
 264  DBG_RSV_EP_L_RSV_ING_L3_RSLTS_MATCH                   128274           0
 265  DBG_RSV_EP_L_RSV_ING_PUNT                            1949566           5
 404  RSV_ING_VPWS_ERR_DROP                                  17002           0
 774  CDP                                                   133797           0
 776  ARP                                                       45           0
 790  DIAGS                                                   6073           0
 832  PUNT_NO_MATCH                                             25           0
 850  PUNT_ADJ                                                   4           0
 862  IPV6_HOP_BY_HOP                                            7           0
 902  PUNT_STATISTICS                                      1647059           5
 908  NETIO_RP_TO_LC_CPU_PUNT                                18250           0
 958  PUNT_LLDP                                              34285           0

Description of NP counters can be found here.

To see descriptions on the router.

RP/0/RSP0/CPU0:ASR9000# show controllers np descriptions location 0/0/CPU0 

Counter                                              Drop Feature  Description
-------------------------------------------------------------------------------
MDF_TX_LC_CPU                                             Infra    Frames sent (punted) to linecard CPU. Includes packets and some messages between NPs.
PARSE_ENET_SAT_RECEIVE_CNT                                Infra    Frames received from Ethernet Satellite ports by this NP
PARSE_INTER_NP_RECEIVE_CNT                                Infra    Frames received on the special connection between paired NPs of a 100GE port.

<snip>

FIA

Fabric Interface ASIC.

Aggregate counters.

RP/0/RSP0/CPU0:ASR9000# show controllers fabric fia stats location 0/0/CPU0 | i "FIA|count|drop"
 ********** FIA-0 **********
Category: count-0
                                Ingress drop:                       13
                                 Egress drop:                        0
                                 Total drop:                        13
 ********** FIA-1 **********
Category: count-1
                                Ingress drop:                        7
                                 Egress drop:                        0
                                 Total drop:                         7

Why the drops happened. Remember to check both directions.

RP/0/RSP0/CPU0:ASR9000# show controllers fabric fia drops egress location 0/0/CPU0
 ********** FIA-0 **********
Category: eg_drop-0
                           From Xbar Uc Crc-0                        0
                           From Xbar Uc Crc-1                        0
                           From Xbar Uc Crc-2                        0
                           From Xbar Uc Crc-3                        0
                           From Xbar Uc Drp-0                        0
                           From Xbar Uc Drp-1                        0
                           From Xbar Uc Drp-2                        0
                           From Xbar Uc Drp-3                        0
                           From Xbar Mc Crc-0                        0
                           From Xbar Mc Crc-1                        0
                           From Xbar Mc Crc-2                        0
                           From Xbar Mc Crc-3                        0
                           From Xbar Mc Drp-0                        0
                           From Xbar Mc Drp-1                        0
                           From Xbar Mc Drp-2                        0
                           From Xbar Mc Drp-3                        0
       Uc dq pkt-len-crc/RO-seq/len error drp                        0
                                Mc rf crc drp                        0
                  Mc vl0 src0 buffer full drp                        0
                  Mc vl1 src0 buffer full drp                        0
                  Mc vl2 src0 buffer full drp                        0
                  Mc vl3 src0 buffer full drp                        0
                  Mc vl0 src1 buffer full drp                        0
                  Mc vl1 src1 buffer full drp                        0
                  Mc vl2 src1 buffer full drp                        0
                  Mc vl3 src1 buffer full drp                        0

<snip>

Fabric

  • Remember to check instance 0 and instance 1
RP/0/RSP0/CPU0:ASR9000# show controllers fabric crossbar statistics instance 0 location 0/0/cpu0 | inc "Port|CRC"
Port statistics for xbar:0 port:0
Port statistics for xbar:0 port:1
Port statistics for xbar:0 port:2
Port statistics for xbar:0 port:3
Port statistics for xbar:0 port:4
Port statistics for xbar:0 port:5
Port statistics for xbar:0 port:7
Port statistics for xbar:0 port:8
Port statistics for xbar:0 port:9
Port statistics for xbar:0 port:11
    Packet CRC Error Count                     : 492435
Port statistics for xbar:0 port:12
    Packet CRC Error Count                     : 491046
Port statistics for xbar:0 port:14
    Packet CRC Error Count                     : 295744
Port statistics for xbar:0 port:15
    Packet CRC Error Count                     : 295007
Port statistics for xbar:0 port:16
    Packet CRC Error Count                     : 295367
Port statistics for xbar:0 port:17
Port statistics for xbar:0 port:24

Checking for drops to the router - Control Plane

Architecture

ASR9K punt inject.jpg

ARP

RP/0/RSP0/CPU0:ASR9000# show arp traffic location all | i "CPU|drop|Error"
0/0/CPU0
  Resolve requests dropped: 0
  Errors: 0 out of memory, 0 no buffers, 0 out of sunbet
  IP Packet drop count for node 0/0/CPU0: 0
0/RSP1/CPU0
  Resolve requests dropped: 0
  Errors: 0 out of memory, 0 no buffers, 0 out of sunbet
  IP Packet drop count for node 0/RSP1/CPU0: 1
0/RSP0/CPU0
  Resolve requests dropped: 0
  Errors: 0 out of memory, 0 no buffers, 0 out of sunbet
  IP Packet drop count for node 0/RSP0/CPU0: 26

LPTS

LPTS is an always on packet policer for any traffic destined to the router. It prevents resource exhaustion from a distributed denial of service attack.

RP/0/RSP0/CPU0:ASR9000# show lpts pifib hardware static-police location 0/0/CPU0
-------------------------------------------------------------
                Node 0/0/CPU0:
-------------------------------------------------------------
 Burst = 100ms for all flow types
-------------------------------------------------------------
Punt Reason             SID                     Flow Rate  Burst Rate Accepted             Dropped              Destination
----------------------- ----------------------- ---------  ---------  -------------------- -------------------- -----------
PUNT_INVALID            NETIO_LOW               100        20         0                    0                    Local
PUNT_ALL                NETIO_HI                1000       200        0                    0                    Local
CDP                     NETIO_CRUCIAL           1000       200        0                    0                    Local
PUNT_CDP_RSP            CDP                     1000       200        0                    0                    0x0030 (0/RSP1/CPU0)
ARP                     ARP                     1000       200        478074551            1910829386           Local

<snip> 

Second spot to check.

RP/0/RSP0/CPU0:ASR9000# show lpts pifib hardware entry statistics location 0/0/CPU0 | e /0

----------------------------------------
L3 - L3 Protocol;L4 - Layer4 Protocol; Intf - Interface;
Dest - Destination Node;
LU - Local chassis fabric unicast;
LM - Local chassis fabric multicast;
RU - Multi chassis fabric unicast;
RM - Multi chassis fabric multicast;
na - Not Applicable or Not Available

Offset L3   VRD id       L4     Intf            Dest   Pkts/Drops             laddr,Port raddr,Port
------ ---- ------------ ------ --------------- ------ ---------------------  -----------------------------------

------------------------
statistics:

Type            Num. Entries         Pkts
------          ------------         ----
Packets accepted by deleted entries: 1720
Packets dropped by deleted entries: 0
Run out of statistics counter errors: 0

Punt Switch

RP/0/RSP0/CPU0:ASR9000# show controllers punt-switch switch-stats location 0/0/CPU0 | i drops
Global Tail drops: 0

SPP - Software Packet Path

RP/0/RSP0/CPU0:ASR9000# show spp client location 0/0/CPU0
Clients
=======
Software Offload EA, JID 335 (pid 544897)
----------------------------------------------------------
Reconnect Pending: F, Exited: F, Keep Queues: F, Pakman Client: F
Quota Current: 0, Limit: 0, Drops 0
Queues:
Key         Cur  Max      Enqueues   High Watermark (Timestamp)        Drops
0xffffffff    0   10             0    0 (22:00:26.872 Mar 01 16 UTC)       0

ASR9K SPIO client stream ID 40, JID 137 (pid 540844)
----------------------------------------------------------
Reconnect Pending: F, Exited: F, Keep Queues: F, Pakman Client: F
Quota Current: 0, Limit: 20000, Drops 0
Queues:
Key         Cur  Max      Enqueues   High Watermark (Timestamp)        Drops
0xffffffff    0   10             0    0 (21:08:11.809 Mar 01 16 UTC)       0
0x02000028    0 4096             0    0 (21:08:11.820 Mar 01 16 UTC)       0

<snip>

Netio

RP/0/RSP0/CPU0:ASR9000# show netio drops location 0/0/CPU0

Drops for interfaces on node 0/0/CPU0

--------------------------------------------------------------------------------
Interface: Bundle-Ether10150 (0x00001a20)
--------------------------------------------------------------------------------

Interface input drops:         0 pkts
Interface input errors:        0 pkts
Interface input unknown proto: 0 pkts
Interface output drops:        0 pkts
Interface output errors:       0 pkts

Node drop counts:

Caps <num>  (Protocol <num> Chain):
-----------------------------------
    DLL/function name                          Address    Drop pkts   Drop bytes
    ----------------------------------------  ----------  ----------  ----------

ipv4 <26>  (ipv4 <12> encaps):
    /pkg/lib/libipv4_fib_switch.dll           0x4e0ea400          12         624


--------------------------------------------------------------------------------
Interface: TenGigE0/0/0/0 (0x040000c0)
--------------------------------------------------------------------------------

Interface input drops:         0 pkts
Interface input errors:        0 pkts
Interface input unknown proto: 3014 pkts
Interface output drops:        0 pkts
Interface output errors:       0 pkts

Node drop counts:

No node drops

References

Cisco Live Slides - BRKSPG-2904 - ASR9000/IOS-XR Hardware Architecture, QoS, EVC, IOS-XR Configuration and Troubleshooting

ASR9K Platform Troubleshooting Manual

ASR 9000 Series Punt Fabric Data Path Failures Troubleshoot Guide

Xander's Guide - Support Forums

Community Tech-Talk:ASR9k Life of a Packet and Troubleshooting Packet Loss - Youtube

Rating: 5.0/5 (6 votes cast)

Personal tools