Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Performance Issues

From DocWiki

(Difference between revisions)
Jump to: navigation, search
m (1 revision)
(Overview of Troubleshooting Performance Issues)
Line 53: Line 53:
7. Be familiar with your application setup.
7. Be familiar with your application setup.
 +
 +
8. Note on the use of "tcp wan-optimization rtt 0" for slow connections.
==Troubleshooting Performance Issues==
==Troubleshooting Performance Issues==

Revision as of 16:16, 15 May 2009

This article describes how to troubleshoot performance issues with your ACE.

Guide Contents
Main Article
Overview of ACE Module Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Module Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting Remote Access
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Layer 7 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Performance Issues
ACE Module Resource Limits
Managing ACE Resources

Contents













Overview of Troubleshooting Performance Issues

Before you begin to troubleshoot ACE performance issues, check and record the following items:

1. Be sure that the correct licenses are installed in your ACE.

2. Record the number of flows that you are sending to the ACE.

3. Record the performance of a single flow.

4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)

5. Identify the ACE context that is receiving the traffic.

6. Enter the following Exec mode commands and save the output to a file:

  • clear stats all
  • show clock
  • show tech-support
  • show clock

7. Be familiar with your application setup.

8. Note on the use of "tcp wan-optimization rtt 0" for slow connections.

Troubleshooting Performance Issues

To troubleshoot performance issues with your ACE, follow these steps:

1. Display the resources allocated to each resource class in the ACE by entering the following command:

ACE_module5/Admin# show resource allocation
---------------------------------------------------------------------------
Parameter                 Min      Max         Class
---------------------------------------------------------------------------

acl-memory                0.00%    100.00%    default
                          0.00%    100.00%    RC1

syslog buffer             0.00%    100.00%    default
                          0.00%    100.00%    RC1

conc-connections          0.00%    100.00%    default
                          0.00%    100.00%    RC1

mgmt-connections          0.00%    100.00%    default
                          0.00%    100.00%    RC1

proxy-connections         0.00%    100.00%    default
                          0.00%    100.00%    RC1

bandwidth                 0.00%    100.00%    default
                          0.00%    100.00%    RC1

connection rate           0.00%    100.00%    default
                          0.00%    100.00%    RC1

inspect-conn rate         0.00%    100.00%    default
                          0.00%    100.00%    RC1

syslog rate               0.00%    100.00%    default
                          0.00%    100.00%    RC1

regexp                    0.00%    100.00%    default
                          0.00%    100.00%    RC1

sticky                    0.00%    100.00%    default
                          5.00%      5.00%    RC1

xlates                    0.00%    100.00%    default
                          0.00%    100.00%    RC1

ssl-connections rate      0.00%    100.00%    default
                          0.00%    100.00%    RC1

mgmt-traffic rate         0.00%    100.00%    default
                          0.00%    100.00%    RC1

mac-miss rate             0.00%    100.00%    default
                          0.00%    100.00%    RC1

throughput                0.00%    100.00%    default
                          0.00%    100.00%    RC1

2. Display the resources allocated to the context in question by entering the following command:

ACE_module5/Admin# show resource usage context C1
                                                    Allocation
        Resource         Current       Peak        Min        Max       Denied
-------------------------------------------------------------------------------
Context: C1
  conc-connections              0          0          0    8000000          0
  mgmt-connections              0          0          0     100000          0
  proxy-connections             0          0          0    1048574          0
  xlates                        0          0          0    1048574          0
  bandwidth                     0          0          0  625000000          0
    throughput                  0          0          0  500000000          0
    mgmt-traffic rate           0          0          0  125000000          0 <------- 1 GBps bandwidth reserved for management traffic
  connection rate               0          0          0    1000000          0
  ssl-connections rate          0          0          0       5000          0
  mac-miss rate                 0          0          0       2000          0
  inspect-conn rate             0          0          0       6000          0
  acl-memory                    0          0          0   78610432          0
  sticky                        0          0     209714          0          0
  regexp                        0          0          0    1048576          0
  syslog buffer                 0          0          0    4194304          0
  syslog rate                   0          0          0     100000          0
Note Note: All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply the displayed bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management (to-the-ACE) traffic.

3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:

cat6k# show fabric status
 slot    channel      speed    module               fabric
                               status               status
    2          0         8G        OK                   OK
    3          0         8G        OK                   OK
    4          0         8G        OK                   OK
    5          0         8G        OK                   OK <-------Shows 8 Gbps connectivity to the chassis back plane
    6          0        20G        OK                   OK
    8          0         8G        OK                   OK

4. Check the fabric utilization by entering the following command:

cat6k# show fabric utilization
  slot    channel      speed    Ingress %     Egress %
    2          0         8G            3            2
    3          0         8G            0            0
    4          0         8G            0            0
    5          0         8G            0            0
    6          0        20G            0            0
    8          0         8G            2            3


5. Display the load of the network processors (NPs) in terms of packets and connection processing for each microengine (ME) by entering the following command:

ACE_module5/Admin# show np 1 me-stats -cpu
0 proxies open.
ME Utilization Statistics
--------------
RECEIVE:                                          7
FASTPATH:                                        44
SLOWTX:                                           0
TCP_RX:                                           0
HTTP:                                             0
IH_RX                                             0
SSL_ME:                                           0
CM_CLOSE:                                        36
X_TO_ME:                                          0
FIXUP:                                            0
REASSEMBLY:                                       0
OCM:                                              0
TCP_TX:                                           0
ICM:                                             39
ACE/Admin# show np 2 me-stats -cpu
0 proxies open.
ME Utilization Statistics
--------------
RECEIVE:                                          9
FASTPATH:                                        46
SLOWTX:                                           2
TCP_RX:                                           0
HTTP:                                             0
IH_RX                                             0
SSL_ME:                                           0
CM_CLOSE:                                        43
X_TO_ME:                                          0
FIXUP:                                            0
REASSEMBLY:                                       0
OCM:                                              0
TCP_TX:                                           0
ICM:                                             46
Note Note: All show np commands must be entered for both NP1 and NP2 to obtain the total combined results. NPs operate safely at any percentage of utilization. As ME functions within the NPs approach 100 percent, the traffic load is stressing the system close to its architectural limits. Any ME function that reaches 100 percent utilization can cause back pressure and lead to dropped packets or dropped connections.

6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering the following command:

ACE_module5/Admin# show cde health | include Fifo
Fifo Full drop count                              0

Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally. Queues that can be affected and create backpressure are as follows:

  • FIFOs for the CDE, NPs, and the Crypto Module
  • Internal queues for each ME

It is possible that some packets that are received by the ACE could be dropped internally if backpressure is applied.

7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQ Transmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are not incrementing by entering the following command:

ACE_module5/Admin# show np 1 me-stats "-s fp" | include Backpressure
FastQ Transmit Backpressure:                      0
SlowQ Transmit Backpressure:                      0
Drop: Transmit Backpressure:                      0
ACE/Admin# show np 1 me-stats "-s fp" | include queue
Drop: Next-Hop queue full:                        0

8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpath queue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Drops due to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, it can drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.

ACE/Admin#  show np 1 me-stats "-s tcp" | include queue
Drop reproxy msg queue full:                      0
Drops due to FastTX queue full:                   0
Drops due to Fastpath queue full:                 0
Drops due to HTTP queue full:                     0
Drops due to SSL queue full:                      0
Drops due to AI queue full:                       0
Drops due to Fixup queue full:                    0

The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP, probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforce configuration dependencies and requirements before pushing the configuration to the data plane.

9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by entering the following command:

ACE_module5/Admin# show processes cpu | inc util
CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%

The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections (such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing or inspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectional concurrent connection limit regardless of the features enabled.

Table 1. Concurrent Connection Support

Connection Type ACE Module Limit
Layer 4 4,000,000
Layer 7 512,000





The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained with distinct connection objects.

10. Display the connection table by entering the following command:

ACE_module5/Admin# show conn

total current connections : 6

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
1          1  in  TCP   130  161.44.67.242:2856    10.86.215.134:23      ESTAB
2          1  out TCP   130  10.86.215.134:23      161.44.67.242:2856    ESTAB
4          1  in  TCP   130  161.44.67.242:2837    10.86.215.134:23      ESTAB
3          1  out TCP   130  10.86.215.134:23      161.44.67.242:2837    ESTAB
4          2  in  TCP   130  161.44.67.242:2857    10.86.215.134:23      ESTAB
3          2  out TCP   130  10.86.215.134:23      161.44.67.242:2857    ESTAB
Note Note: You can add the detail command option to provide the following additional fields: connection idle time, elapsed time of the connection, byte count, and packet count for each connection object.

The total current connections counter is also maintained in the output of the following command:

switch/Admin# show stats connection

+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
 Total Connections Created  : 124
 Total Connections Current  : 6
 Total Connections Destroyed: 62
 Total Connections Timed-out: 58
 Total Connections Failed   : 0
Note Note: The Total Connections Current counter counts the number of used connection objects, not the number of TCP flows. The number of TCP flows can be roughly determined as half the number of connection objects minus any UDP connections. The Total Connections Current counter is always up to date and the maximum value can be 8,000,000.

Because of the Cisco ACE Module’s architecture, with distinct paths for new and established connections, the number of existing concurrent connections does not heavily impact the rate at which new connections can be set up. Nevertheless, a very large number of concurrent connections will eventually affect the performance of the system in setting up new connections.

Rating: 4.2/5 (6 votes cast)

Personal tools