Cisco Application Control Engine (ACE) Troubleshooting Guide -- Preliminary ACE Troubleshooting

From DocWiki

Revision as of 23:41, 9 March 2011 by Dakelley (Talk | contribs)
Jump to: navigation, search

This article describes some basic troubleshooting steps that you can perform to rule out some of the simpler issues before delving deeper into the troubleshooting process.


Guide Contents
Main Article
Overview of ACE Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting ACE Appliance Ethernet Ports
Troubleshooting Remote Access
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Layer 7 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Performance Issues
ACE Resource Limits
Managing ACE Resources

Contents






Preliminary ACE Troubleshooting Steps

  1. Check the status of the ACE module from the Catalyst 6500 series switch or Cisco 7600 series router. See the "Checking the ACE Status from the Supervisor Engine" section.
  2. Verify that you have allocated the correct VLANs to the ACE in the Multilayer Switch Feature Card (MSFC) VLAN configuration. See the "Verifying the MSFC VLAN Configuration" section.
  3. Verify that you can establish a session to the ACE from the supervisor engine. See the "Establishing a Session with the ACE from the Supervisor Engine" section.
  4. Verify that the ACE is receiving VLAN allocations from the MSFC. See the Verifying the ACE is Receiving VLAN Allocations from the MSFC" section.
  5. Verify your ACE bandwidth, SSL, and virtualization licenses. See the "Verifying Your ACE Licenses" section.
  6. Verify that you have configured an access control list (ACL) to permit traffic on the interfaces on which you wish the ACE to receive traffic. If you do not configure an ACL to permit traffic on an interface, all traffic destined to that interface will be blocked by the ACE. See the "Configuring an ACL to Permit Input Traffic to the ACE" section.
  7. Verify that the ACE is sending and receiving traffic. See the "Verifying that the ACE is Sending and Receiving Traffic" section.
  8. Verify the management traffic to the control plane. See the "Verifying to-the-ACE Traffic"section.

Checking the ACE Status from the Supervisor Engine

Before you begin to troubleshoot your ACE, Telnet to the Catalyst 6500 series switch or Cisco 7600 series router supervisor engine, log in, and check the status of the ACE.

telnet 10.1.1.2

User Access Verification

Password:
cat6k> enable
Password:
cat6k# show module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
 .   
 .
 5    1  Application Control Engine Module      ACE20-6500-K9      SAD1031044S
 .
 .
 
Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
 .
 .  
 5  0018.b9a6.9114 to 0018.b9a6.911b   1.1   8.7(0.22)ACE A2(2.0)      Ok <------- ACE status
 .
 .
 
Mod  Sub-Module                  Model              Serial       Hw     Status
--- --------------------------- ------------------ ----------- ------- -------
 6  Policy Feature Card 3       WS-F6K-PFC3A       SAL09094NUB  2.5    Ok
 6  MSFC3 Daughterboard         WS-SUP720          SAL09094N33  2.5    Ok
 
Mod  Online Diag Status
--- -------------------
 .  
 .  
 5  Pass
 .  
 .

Verifying the MSFC VLAN Configuration

To verify that the VLANs that you intend to use in your ACE have been configured and allocated to the ACE in the MSFC, follow these steps:

1. Check the VLANs configured and allocated to the ACE by entering the following command from the supervisor engine:
cat6k# show run | include svclc

svclc module 5 vlan-group 123,130,133


2. Ensure that the VLAN groups that you intend to use for your ACE are allocated properly in the MSFC configuration by entering the following commands:
cat6k# show svclc module 5 vlan-group
Module Vlan-groups
------ -----------
 05   123,130,133
cat6k# show svclc vlan-group
Display vlan-groups created by both ACE module and FWSM commands

Group    Created by      vlans
-----    ----------      -----
 123           ACE      103,105,107,111-112,119,134,160,171,200,203,205,207,211-212,226,253,260
 130           ACE      130
 133           ACE      100,194,221,256-257


3. Verify that the VLANs you intend to use in your ACE are configured in the MSFC by entering the following command:
cat6k# show interface te5/1 trunk

Port          Mode         Encapsulation  Status        Native vlan
Te5/1         on           802.1q         trunking      1

Port          Vlans allowed on trunk
Te5/1         100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257,260

Port          Vlans allowed and active in management domain
Te5/1         100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257

Port          Vlans in spanning tree forwarding state and not pruned
Te5/1         100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257


4. Ensure that traffic is routed to two ACEs in the same chassis when both client- and server-side VLANs are configured as switched virtual interfaces (SVIs) on the MSFC in routed mode by entering the following command:
cat6k# show svclc multiple-vlan-interfaces
Multiple ACE vlan interfaces feature is enabled

Establishing a Session with the ACE from the Supervisor Engine

To verify that you can establish a session with the ACE from the supervisor engine in the Catalyst 65000 series switch or the Cisco 7600 series router, enter the following command:

cat6k# session slot 5 processor 0
The default escape character is Ctrl-^, then x.
You can also type 'exit' at the remote prompt to end the session
Trying 127.0.0.50 ... Open
 
ACE_module5 login:

Verifying the ACE is Receiving VLAN Allocations from the MSFC

Ensure that the VLAN that you intend to use on ACE are allocated properly in the MSFC configuration by entering the following command:

ACE-1/Admin# show vlans
Vlans configured on SUP for this module
 vlan123  vlan130  vlan133


If interface VLANs are already assigned on the ACE you can use the show interface vlan <num> command to verify the interface is properly assigned on the MSFC and up on the MSFC:

ACE-1/Admin# show interface vlan 123 

vlan10 is up
  Hardware type is VLAN
  MAC address is 00:18:b9:a6:89:0d
  Mode : routed
  IP address is 10.10.10.1 netmask is 255.255.255.0
  FT status is non-redundant
  Description:not set
  MTU: 1500 bytes
  Last cleared: never
  Alias IP address not set
  Peer IP address not set
  Assigned from the Supervisor, up on Supervisor
     7101679 unicast packets input, 878043707 bytes
     0 multicast, 0 broadcast
     0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
     6387914 unicast packets output, 1541924399 bytes
     0 multicast, 22826 broadcast
     0 output errors, 0 ignored

Verifying the ACE Image

To display the version of the software image and the image filename that is currently running in your ACE, enter the following command:

ACE_module5/Admin# show version
Cisco Application Control Software (ACSW)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
  loader:    Version 12.2[121]
  system:    Version A2(2.0) [build 3.0(0)A2(2.0)]  <--------
  system image file: [LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin  <--------
  installed license: no feature license is installed
 
Hardware
  Cisco ACE (slot: 5)
  cpu info:
    number of cpu(s): 2
    cpu type: SiByte
    cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz
    cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz
  memory info:
    total: 955396 kB, free: 289704 kB
    shared: 0 kB, buffers: 2336 kB, cached 0 kB
  cf info:
    filesystem: /dev/cf
    total: 1000000 kB, used: 494912 kB, available: 505088 kB
 
last boot reason:  NP 1 Failed : NP ME Hung
 
configuration register:  0x1
ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)


This command provides other useful information, for example:

  • Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)
  • Available memory
  • Last boot reason
  • Configuration register (confreg) value
  • ACE uptime

Verifying Your ACE Licenses

Log in to your ACE and enter the following command to display the SSL, virtualization, and bandwidth licenses that are currently installed and in use in your ACE:

ACE_module5/Admin# show license usage
License                      Ins   Lic    Status   Expiry Date   Comments
                                  Count
--------------------------------------------------------------------------------
ACE-08G-LIC                   No    -    Unused                  -
ACE-16G-LIC                   No    -    Unused                  -
ACE-UPG1-LIC                  No    -    Unused                  -
ACE-UPG2-LIC                  No    -    Unused                  -
ACE-VIRT-020                  No    -    Unused                  -
ACE-VIRT-050                  No    -    Unused                  -
ACE-VIRT-100                  No    -    Unused                  -
ACE-VIRT-250                 Yes    1    In use      never       -
ACE-VIRT-UP1                  No    -    Unused                  -
ACE-VIRT-UP2                  No    -    Unused                  -
ACE-VIRT-UP3                  No    -    Unused                  -
ACE10-16G-LIC                 No    -    Unused                  -
ACE-SEC-LIC-K9                No    -    Unused                  -
ACE-SSL-05K-K9                No    -    Unused                  -
ACE-SSL-10K-K9                No    -    Unused                  -
ACE-SSL-15K-K9                No    -    Unused                  -
ACE-SSL-20K-K9                No    -    Unused                  -
ACE-SSL-UP1-K9                No    -    Unused                  -
ACE-SSL-UP2-K9                No    -    Unused                  -
ACE-SSL-UP3-K9                No    -    Unused                  -


ACE_module5/Admin# show license status
Licensed Feature                  Count
------------------------------    -----
SSL transactions per second       1000
Virtualized contexts              250
Module bandwidth in Gbps          4


You can also see the licenses that reside on the Flash disk by entering the following command:

ACE_module5/Admin# dir disk0:
    236  Oct 17 09:18:26 2006 ACE-SSL-05K-K9.lic  <--------
    235  Oct 17 09:16:58 2006 ACE-VIRT-250.lic  <--------
   1024  Sep 28 19:11:11 2006 cv/
1654606  Oct 26 12:56:16 2006 dplug

         Usage for disk0: filesystem
                  2759552 bytes total used
                  8405120 bytes free
                 11164672 total bytes


In the above example, there is an SSL 5K TPS license on the Flash disk that has not yet been installed in the ACE.

To install the license, enter the following command:

ACE_module5/Admin# license install disk0:ACE-SSL-05K-K9.lic
Installing license... done
ACE_module5/Admin#

Configuring an ACL to Permit Input Traffic to the ACE

You must configure an ACL to allow the ACE to receive traffic. All traffic to the ACE is blocked until you do so. For example, to configure an ACL that permits all IP trafffic except from the 10.1.1.0 network, enter the following commands:

ACE_module5/Admin(config)# access-list ACL1 extended deny ip 10.1.1.0 255.255.255.0 any
ACE_module5/Admin(config)# access-list ACL1 extended permit ip any any
ACE_module5/Admin(config)# interface vlan 100
ACE_module5/Admin(config-if)# access-group input ACL1

Verifying that the ACE is Sending and Receiving Traffic

You can tell if traffic is reaching the ACE by using the show svclc module number traffic command on the Catalyst 6500 series switch or Cisco 7600 series router. This command displays counters (packets input and packets output) that increase when the switch or router sends packets to or receives packets from the ACE.

cat6k# show svclc module 5 traffic
ACE module 5:

Specified interface is up line protocol is up (connected)
  Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  input flow-control is on, output flow-control is unsupported
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 4d02h
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 2 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     528888 packets input, 41329093 bytes, 0 no buffer <-------
     Received 469945 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     7776 packets output, 746361 bytes, 0 underruns <-------
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out


A trace of the Te?/1 (10-Gbps switch fabric interface, where ? = the module number) interface will show you whether packets are arriving at the switch fabric interface (SFI).

Another useful command is the show cde health command on the ACE. This command shows the current state of the Classification Distribution Engine (CDE). The network processors (NP1 and NP2) are represented by IXP0 and IXP1, respectively. You should not observe any drops, errors, or flow control issues in the output of this command. If the Packets Received or the Packets Transmitted counters of the CDE Hyperion Interface are not increasing, then packets are not coming into or going out of the ACE.

ACE_module5/Admin# show cde health

CDE BRCM INTERFACE
======================
Packets received                                             4933
Packets transmitted                                       2437922
Broadcom interface CRC error count                              0
BRCM VOQ status                           [empty]      [not full]
BRCM pull status                                        [pulling] 

CDE HYPERION INTERFACE
======================
Packets received                                         29913371 <-------
Packets transmitted                                          8034 <-------
Short packets drop count                                        0
Fifo Full drop count                                            0
Protocol error drop count                                       0
FCS error drop count                                            0
CRC error drop count                                            0
Num times flow control triggered on hyp interface               0
Num self generated multicast packets filtered                1880
HYP IXP0 VOQ status                       [empty]      [not full]
HYP IXP1 VOQ status                       [empty]      [not full]
HYP SLOW VOQ status                       [empty]      [not full]
HYP tx pull status                                      [pulling]

CDE IXP0 INTERFACE
======================
Packets received                                           784985
Packets transmitted                                      27827116
Num bad pkts recvd on fast spi channel0                         0
Num bad pkts recvd on slow spi channel8                         0
Num bad pkts recvd on fast spi channel2                         0
Num bad pkts recvd on slow spi channel4                         0
IXP0 Fast VOQ status                      [empty]      [not full]
IXP0 BRCM VOQ status                      [empty]      [not full]
IXP0 pull status                                        [pulling]
IXP0 spi src status                                     [healthy]
IXP0 spi snk status                                     [healthy]

CDE1 SWITCH1 INTERFACE
======================
Packets received (hyp, ixp0)                                 4415
Packets received (bcm)                                    1656608
Packets received (daughter card 0)                              0
Packets received (daughter card 1)                              0
Packets Errors received (hyp, ixp0)                             0
Packets Errors received (bcm)                                   0
Packets Errors received (daughter card 0)                       0
Packets Errors received (daughter card 1)                       0
Packets transmitted (ixp1)                                2089360
Packets transmitted (nitrox)                                    0
Packets Errors transmitted (ixp1)                               0
Packets Errors transmitted (nitrox)                             0

CDE2 SWITCH2 INTERFACE
======================
Packets received (ixp1)                                   2089360
Packets received (nitrox)                                       0
Packets Errors received (ixp1)                                  0
Packets Errors received (nitrox)                                0
Packets transmitted (hyp, ixp0)                              4415
Packets transmitted (broadcom)                            1656608
Packets transmitted (daughter card 0)                           0
Packets transmitted (daughter card 1)                           0
Packets Errors transmitted (ixp1)                               0
Packets Errors transmitted (nitrox)                             0
Packets Errors transmitted (daughter card 0)                    0
Packets Errors transmitted (daughter card 1)                    0

CDE IXP1 INTERFACE
======================
Packets received                                          1661023
Packets transmitted                                       2089360
Num bad pkts recvd on fast spi channel0                         0
Num bad pkts recvd on slow spi channel8                         0
Num bad pkts recvd on fast spi channel2                         0
Num bad pkts recvd on slow spi channel4                         0
IXP1 Fast VOQ status                      [empty]      [not full]
IXP1 BRCM VOQ status                      [empty]      [not full]
IXP1 pull status                                        [pulling]
IXP1 spi src status                                     [healthy]
IXP1 spi snk status                                     [healthy]

CDE NITROX INTERFACE
======================
Packets received                                                0
Packets transmitted                                             0
Num bad pkts recvd on fast spi channel0                         0
Num bad pkts recvd on slow spi channel8                         0
Num bad pkts recvd on fast spi channel2                         0
Num bad pkts recvd on slow spi channel4                         0
NTX Fast VOQ status                       [empty]      [not full]
NTX BRCM VOQ status                       [empty]      [not full]
NTX pull status                                         [pulling]
NTX spi src status                                      [healthy]
NTX spi snk status                                      [healthy]
== Backplane ==
ITASCA_SYS_CNTL1 0x300  data 0x61f0000
ITASCA_SYS_CNTL2 0x304  data 0x80c30000


You can also use the show interface command on the ACE to display traffic that is sent and received on the interface for each VLAN that is configured on the ACE.

ACE_module5/Admin# show interface

bvi2 is administratively down
  Hardware type is BVI
  MAC address is 00:18:b9:a6:91:15
  Mode : unknown
  FT status is non-redundant
  Description:not set
  MTU: 1500 bytes
  Last cleared: never
  Alias IP address not set
  Peer IP address not set
     0 unicast packets input, 0 bytes
     0 multicast, 0 broadcast
     0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
     0 unicast packets output, 0 bytes
     0 multicast, 0 broadcast
     0 output errors, 0 ignored

vlan100 is administratively down
  Hardware type is VLAN
  MAC address is 00:18:b9:a6:91:15
  Mode : unknown
  FT status is non-redundant
  Description:not set
  MTU: 1500 bytes
  Last cleared: never
  Alias IP address not set
  Peer IP address not set
  Assigned from the Supervisor, up on Supervisor
     0 unicast packets input, 0 bytes
     0 multicast, 0 broadcast
     0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
     0 unicast packets output, 0 bytes
     0 multicast, 0 broadcast
     0 output errors, 0 ignored

vlan130 is up
  Hardware type is VLAN
  MAC address is 00:18:b9:a6:91:15
  Mode : routed
  IP address is 10.86.215.134 netmask is 255.255.255.0
  FT status is non-redundant
  Description:not set
  MTU: 1500 bytes
  Last cleared: never
  Alias IP address not set
  Peer IP address not set
  Assigned from the Supervisor, up on Supervisor
     59858 unicast packets input, 41711169 bytes <-------
     193118 multicast, 280789 broadcast <-------
     0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops <-------
     6260 unicast packets output, 785167 bytes <-------
     0 multicast, 1892 broadcast <-------
     0 output errors, 0 ignored <-------

Verifying to-the-ACE Traffic

Traffic that is destined to the ACE itself arrives at the control plane in one of the following ways:

  • Directly from the console connection
  • Directly from the supervisor engine connection
  • Traffic from the SFI that is forwarded by the CDE in the data plane

Use the following commands to verify that traffic is going to and coming from the control plane.

ACE_module5/Admin# show netio stats

High Priority (Control)              Normal Priority (Data)
-----------------------              ----------------------
Net Rx Packets        : 224          Net Rx Packets        : 2521794
Net Rx Bytes          : 17528        Net Rx Bytes          : 196704169
Net Rx Unsupported L2 : 0            Net Rx Unsupported L2 : 0
Net Rx Lock Errors    : 0            Net Rx Lock Errors    : 0
Net Rx Interface Miss : 0            Net Rx Interface Miss : 2326290
Net Rx No Arp Client  : 0            Net Rx No Arp Client  : 0
Net Rx Alias Drops    : 0            Net Rx Alias Drops    : 0
Net Rx Repl. Errors   : 0            Net Rx Repl. Errors   : 0
Net Rx Repl. If Err   : 0            Net Rx Repl. If Errs  : 0
Net Rx Internal Errs  : 0            Net Rx Internal Errs  : 0

Net Tx Packets        : 0            Net Tx Packets        : 5213
Net Tx Bytes          : 0            Net Tx Bytes          : 414073
Net Tx Lock Errors    : 0            Net Tx Lock Errors    : 0
Net Tx Bad Context ID : 0            Net Tx Bad Context ID : 0
Net Tx No Route Found : 0            Net Tx No Route Found : 0
Net Tx No Adjacency   : 0            Net Tx No Adjacency   : 1
Net Tx Invalid If ID  : 0            Net Tx Invalid If ID  : 0
Net Tx If Down        : 0            Net Tx If Down        : 0
Net Tx No Src Addr    : 0            Net Tx No Src Addr    : 0
Net Tx No Encap       : 0            Net Tx No Encap       : 0
Net Tx FIFO Errors    : 0            Net Tx Fifo Errors    : 0
Net Tx No VMAC Errors : 0            Net Tx No VMAC Errors : 0 

IPC Tx Packets        : 76           IPC Tx Packets        : 0
IPC Tx Bytes          : 17638        IPC Tx Bytes          : 0
IPC Tx Fifo Errors    : 0            IPC Tx Fifo Errors    : 0 

Client Rx Queue Full  : 0            Client Rx Queue Full  : 0

Pseudo Rx Queue Full  : 0            Pseudo Rx Queue Full  : 0


ACE_module5/Admin# show fifo stats

High Priority (Control)          Normal Priority (Data)
-----------------------          ----------------------
Rx Packets        : 224          Rx Packets        : 2524886
Rx Bytes          : 17528        Rx Bytes          : 196952927
Rx DMA Errors     : 0            Rx DMA Errors     : 0
Rx Drop Events    : 0            Rx Drop Events    : 0
Rx Descr Errors   : 0            Rx Descr Errors   : 0
Rx Bad Descrs     : 0            Rx Bad Descrs     : 0
Rx Length Errors  : 0            Rx Length Errors  : 0

Tx Packets        : 76           Tx Packets        : 5241
Tx Bytes          : 17682        Tx Bytes          : 464991
Tx Drops          : 0            Tx Drops          : 0
Tx DMA Errors     : 0            Tx DMA Errors     : 0
Tx SOP Errors     : 0            Tx SOP Errors     : 0

Global Errors
-------------
Rx Underflows     : 0
Rx Overflows      : 0
Tx Underflows     : 0
Tx Overflows      : 0
Resets            : 0
Zbuff alloc fail  : 0 

Interrupt Stats
---------------
Total Interrupt count            : 2529603
Rx Interrupt count               : 2524302       
Tx interrupt count               : 5310

Rating: 3.8/5 (9 votes cast)

Personal tools