Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Performance Issues

From DocWiki

Latest revision as of 21:35, 11 March 2011

This article describes how to troubleshoot performance issues with your ACE.

Overview of Troubleshooting Performance Issues

Before you begin to troubleshoot ACE performance issues, check and record the following items:

1. Be sure that the correct licenses are installed in your ACE.

2. Record the number of flows that you are sending to the ACE.

3. Record the performance of a single flow.

4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)

5. Identify the ACE context that is receiving the traffic.

6. Enter the following Exec mode commands and save the output to a file:

• clear stats all
• show clock
• show tech-support
• show clock

7. Be familiar with your application setup.

Troubleshooting Performance Issues

To troubleshoot performance issues with your ACE, follow these steps:

1. Display the resources allocated to each resource class in the ACE by entering the following command:

ACE_module5/Admin# show resource allocation
---------------------------------------------------------------------------
Parameter                 Min      Max         Class
---------------------------------------------------------------------------

acl-memory                0.00%    100.00%    default
                          0.00%    100.00%    RC1

syslog buffer             0.00%    100.00%    default
                          0.00%    100.00%    RC1

conc-connections          0.00%    100.00%    default
                          0.00%    100.00%    RC1

mgmt-connections          0.00%    100.00%    default
                          0.00%    100.00%    RC1

proxy-connections         0.00%    100.00%    default
                          0.00%    100.00%    RC1

bandwidth                 0.00%    100.00%    default
                          0.00%    100.00%    RC1

connection rate           0.00%    100.00%    default
                          0.00%    100.00%    RC1

inspect-conn rate         0.00%    100.00%    default
                          0.00%    100.00%    RC1

syslog rate               0.00%    100.00%    default
                          0.00%    100.00%    RC1

regexp                    0.00%    100.00%    default
                          0.00%    100.00%    RC1

sticky                    0.00%    100.00%    default
                          5.00%      5.00%    RC1

xlates                    0.00%    100.00%    default
                          0.00%    100.00%    RC1

ssl-connections rate      0.00%    100.00%    default
                          0.00%    100.00%    RC1

mgmt-traffic rate         0.00%    100.00%    default
                          0.00%    100.00%    RC1

mac-miss rate             0.00%    100.00%    default
                          0.00%    100.00%    RC1

throughput                0.00%    100.00%    default
                          0.00%    100.00%    RC1

2. Display the resources allocated to the context in question by entering the following command:

ACE_module5/Admin# show resource usage context C1
                                                    Allocation
        Resource         Current       Peak        Min        Max       Denied
-------------------------------------------------------------------------------
Context: C1
  conc-connections              0          0          0    8000000          0
  mgmt-connections              0          0          0     100000          0
  proxy-connections             0          0          0    1048574          0
  xlates                        0          0          0    1048574          0
  bandwidth                     0          0          0  625000000          0
    throughput                  0          0          0  500000000          0
    mgmt-traffic rate           0          0          0  125000000          0 <------- 1 GBps bandwidth reserved for management traffic
  connection rate               0          0          0    1000000          0
  ssl-connections rate          0          0          0       5000          0
  mac-miss rate                 0          0          0       2000          0
  inspect-conn rate             0          0          0       6000          0
  acl-memory                    0          0          0   78610432          0
  sticky                        0          0     209714          0          0
  regexp                        0          0          0    1048576          0
  syslog buffer                 0          0          0    4194304          0
  syslog rate                   0          0          0     100000          0

Note:

All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply the displayed bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management (to-the-ACE) traffic.

3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:

cat6k# show fabric status
 slot    channel      speed    module               fabric
                               status               status
    2          0         8G        OK                   OK
    3          0         8G        OK                   OK
    4          0         8G        OK                   OK
    5          0         8G        OK                   OK <-------Shows 8 Gbps connectivity to the chassis back plane
    6          0        20G        OK                   OK
    8          0         8G        OK                   OK

4. Check the fabric utilization by entering the following command:

cat6k# show fabric utilization
  slot    channel      speed    Ingress %     Egress %
    2          0         8G            3            2
    3          0         8G            0            0
    4          0         8G            0            0
    5          0         8G            0            0
    6          0        20G            0            0
    8          0         8G            2            3

5. Display the load of the network processors (NPs) in terms of packets and connection processing for each microengine (ME) by entering the following command:

ACE_module5/Admin# show np 1 me-stats -cpu
0 proxies open.
ME Utilization Statistics
--------------
RECEIVE:                                          7
FASTPATH:                                        44
SLOWTX:                                           0
TCP_RX:                                           0
HTTP:                                             0
IH_RX                                             0
SSL_ME:                                           0
CM_CLOSE:                                        36
X_TO_ME:                                          0
FIXUP:                                            0
REASSEMBLY:                                       0
OCM:                                              0
TCP_TX:                                           0
ICM:                                             39

ACE/Admin# show np 2 me-stats -cpu
0 proxies open.
ME Utilization Statistics
--------------
RECEIVE:                                          9
FASTPATH:                                        46
SLOWTX:                                           2
TCP_RX:                                           0
HTTP:                                             0
IH_RX                                             0
SSL_ME:                                           0
CM_CLOSE:                                        43
X_TO_ME:                                          0
FIXUP:                                            0
REASSEMBLY:                                       0
OCM:                                              0
TCP_TX:                                           0
ICM:                                             46

Note:

All show np commands must be entered for both NP1 and NP2 to obtain the total combined results. NPs operate safely at any percentage of utilization. As ME functions within the NPs approach 100 percent, the traffic load is stressing the system close to its architectural limits. Any ME function that reaches 100 percent utilization can cause back pressure and lead to dropped packets or dropped connections.

6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering the following command:

ACE_module5/Admin# show cde health | include Fifo
Fifo Full drop count                              0

Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally. Queues that can be affected and create backpressure are as follows:

• FIFOs for the CDE, NPs, and the Crypto Module
• Internal queues for each ME

It is possible that some packets that are received by the ACE could be dropped internally if backpressure is applied.

7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQ Transmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are not incrementing by entering the following command:

ACE_module5/Admin# show np 1 me-stats "-s fp" | include Backpressure
FastQ Transmit Backpressure:                      0
SlowQ Transmit Backpressure:                      0
Drop: Transmit Backpressure:                      0

ACE/Admin# show np 1 me-stats "-s fp" | include queue
Drop: Next-Hop queue full:                        0

8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpath queue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Drops due to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, it can drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.

ACE/Admin#  show np 1 me-stats "-s tcp" | include queue
Drop reproxy msg queue full:                      0
Drops due to FastTX queue full:                   0
Drops due to Fastpath queue full:                 0
Drops due to HTTP queue full:                     0
Drops due to SSL queue full:                      0
Drops due to AI queue full:                       0
Drops due to Fixup queue full:                    0

The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP, probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforce configuration dependencies and requirements before pushing the configuration to the data plane.

9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by entering the following command:

ACE_module5/Admin# show processes cpu | inc util
CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%

The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections (such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing or inspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectional concurrent connection limit regardless of the features enabled.

Table 1. Concurrent Connection Support

The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained with distinct connection objects.

10. Display the connection table by entering the following command:

ACE_module5/Admin# show conn

total current connections : 6

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
1          1  in  TCP   130  161.44.67.242:2856    10.86.215.134:23      ESTAB
2          1  out TCP   130  10.86.215.134:23      161.44.67.242:2856    ESTAB
4          1  in  TCP   130  161.44.67.242:2837    10.86.215.134:23      ESTAB
3          1  out TCP   130  10.86.215.134:23      161.44.67.242:2837    ESTAB
4          2  in  TCP   130  161.44.67.242:2857    10.86.215.134:23      ESTAB
3          2  out TCP   130  10.86.215.134:23      161.44.67.242:2857    ESTAB

Note:

You can add the detail command option to provide the following additional fields: connection idle time, elapsed time of the connection, byte count, and packet count for each connection object.

The total current connections counter is also maintained in the output of the following command:

switch/Admin# show stats connection

+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
 Total Connections Created  : 124
 Total Connections Current  : 6
 Total Connections Destroyed: 62
 Total Connections Timed-out: 58
 Total Connections Failed   : 0

Note:

The Total Connections Current counter counts the number of used connection objects, not the number of TCP flows. The number of TCP flows can be roughly determined as half the number of connection objects minus any UDP connections. The Total Connections Current counter is always up to date and the maximum value can be 8,000,000.

Because of the Cisco ACE Module’s architecture, with distinct paths for new and established connections, the number of existing concurrent connections does not heavily impact the rate at which new connections can be set up. Nevertheless, a very large number of concurrent connections will eventually affect the performance of the system in setting up new connections.

11. Use the command "tcp wan-optimization rtt 0" for slow connections.

The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order to increase performance for a given connection. The LB decision is made in the software (proxy) and then moved to the fastpath (unproxy). In a persistence rebalance scenario, the proxy/unproxy can occur Many times on a given connection. It is possible that if a packet enters the system during the transition Between the proxy and unproxy states, a packet may not be forwarded as expected and a retransmission may be relied upon. This can affect performance. As a workaround, it is possible to configure the ACE such that fastpath forwarding is prohibited This can be accomplished by configuring a parameter map with the following:

"tcp wan-optimization rtt 0"