Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Performance Issues
From DocWiki
m |
|||
| Line 8: | Line 8: | ||
__TOC__ | __TOC__ | ||
| + | |||
| + | |||
| + | |||
Revision as of 21:16, 29 May 2009
This article describes how to troubleshoot performance issues with your ACE.
Contents |
Overview of Troubleshooting Performance Issues
Before you begin to troubleshoot ACE performance issues, check and record the following items:
1. Be sure that the correct licenses are installed in your ACE.
2. Record the number of flows that you are sending to the ACE.
3. Record the performance of a single flow.
4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)
5. Identify the ACE context that is receiving the traffic.
6. Enter the following Exec mode commands and save the output to a file:
- clear stats all
- show clock
- show tech-support
- show clock
7. Be familiar with your application setup.
Troubleshooting Performance Issues
To troubleshoot performance issues with your ACE, follow these steps:
1. Display the resources allocated to each resource class in the ACE by entering the following command:
ACE_module5/Admin# show resource allocation
---------------------------------------------------------------------------
Parameter Min Max Class
---------------------------------------------------------------------------
acl-memory 0.00% 100.00% default
0.00% 100.00% RC1
syslog buffer 0.00% 100.00% default
0.00% 100.00% RC1
conc-connections 0.00% 100.00% default
0.00% 100.00% RC1
mgmt-connections 0.00% 100.00% default
0.00% 100.00% RC1
proxy-connections 0.00% 100.00% default
0.00% 100.00% RC1
bandwidth 0.00% 100.00% default
0.00% 100.00% RC1
connection rate 0.00% 100.00% default
0.00% 100.00% RC1
inspect-conn rate 0.00% 100.00% default
0.00% 100.00% RC1
syslog rate 0.00% 100.00% default
0.00% 100.00% RC1
regexp 0.00% 100.00% default
0.00% 100.00% RC1
sticky 0.00% 100.00% default
5.00% 5.00% RC1
xlates 0.00% 100.00% default
0.00% 100.00% RC1
ssl-connections rate 0.00% 100.00% default
0.00% 100.00% RC1
mgmt-traffic rate 0.00% 100.00% default
0.00% 100.00% RC1
mac-miss rate 0.00% 100.00% default
0.00% 100.00% RC1
throughput 0.00% 100.00% default
0.00% 100.00% RC1
2. Display the resources allocated to the context in question by entering the following command:
ACE_module5/Admin# show resource usage context C1
Allocation
Resource Current Peak Min Max Denied
-------------------------------------------------------------------------------
Context: C1
conc-connections 0 0 0 8000000 0
mgmt-connections 0 0 0 100000 0
proxy-connections 0 0 0 1048574 0
xlates 0 0 0 1048574 0
bandwidth 0 0 0 625000000 0
throughput 0 0 0 500000000 0
mgmt-traffic rate 0 0 0 125000000 0 <------- 1 GBps bandwidth reserved for management traffic
connection rate 0 0 0 1000000 0
ssl-connections rate 0 0 0 5000 0
mac-miss rate 0 0 0 2000 0
inspect-conn rate 0 0 0 6000 0
acl-memory 0 0 0 78610432 0
sticky 0 0 209714 0 0
regexp 0 0 0 1048576 0
syslog buffer 0 0 0 4194304 0
syslog rate 0 0 0 100000 0
 | Note: | All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply the displayed bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management (to-the-ACE) traffic. |
3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:
cat6k# show fabric status
slot channel speed module fabric
status status
2 0 8G OK OK
3 0 8G OK OK
4 0 8G OK OK
5 0 8G OK OK <-------Shows 8 Gbps connectivity to the chassis back plane
6 0 20G OK OK
8 0 8G OK OK
4. Check the fabric utilization by entering the following command:
cat6k# show fabric utilization
slot channel speed Ingress % Egress %
2 0 8G 3 2
3 0 8G 0 0
4 0 8G 0 0
5 0 8G 0 0
6 0 20G 0 0
8 0 8G 2 3
5. Display the load of the network processors (NPs) in terms of packets and connection processing for each microengine (ME) by entering the following command:
ACE_module5/Admin# show np 1 me-stats -cpu 0 proxies open. ME Utilization Statistics -------------- RECEIVE: 7 FASTPATH: 44 SLOWTX: 0 TCP_RX: 0 HTTP: 0 IH_RX 0 SSL_ME: 0 CM_CLOSE: 36 X_TO_ME: 0 FIXUP: 0 REASSEMBLY: 0 OCM: 0 TCP_TX: 0 ICM: 39
ACE/Admin# show np 2 me-stats -cpu 0 proxies open. ME Utilization Statistics -------------- RECEIVE: 9 FASTPATH: 46 SLOWTX: 2 TCP_RX: 0 HTTP: 0 IH_RX 0 SSL_ME: 0 CM_CLOSE: 43 X_TO_ME: 0 FIXUP: 0 REASSEMBLY: 0 OCM: 0 TCP_TX: 0 ICM: 46
| Note: | All show np commands must be entered for both NP1 and NP2 to obtain the total combined results. NPs operate safely at any percentage of utilization. As ME functions within the NPs approach 100 percent, the traffic load is stressing the system close to its architectural limits. Any ME function that reaches 100 percent utilization can cause back pressure and lead to dropped packets or dropped connections. |
6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering the following command:
ACE_module5/Admin# show cde health | include Fifo Fifo Full drop count 0
Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally. Queues that can be affected and create backpressure are as follows:
- FIFOs for the CDE, NPs, and the Crypto Module
- Internal queues for each ME
It is possible that some packets that are received by the ACE could be dropped internally if backpressure is applied.
7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQ Transmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are not incrementing by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-s fp" | include Backpressure FastQ Transmit Backpressure: 0 SlowQ Transmit Backpressure: 0 Drop: Transmit Backpressure: 0
ACE/Admin# show np 1 me-stats "-s fp" | include queue Drop: Next-Hop queue full: 0
8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpath queue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Drops due to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, it can drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.
ACE/Admin# show np 1 me-stats "-s tcp" | include queue Drop reproxy msg queue full: 0 Drops due to FastTX queue full: 0 Drops due to Fastpath queue full: 0 Drops due to HTTP queue full: 0 Drops due to SSL queue full: 0 Drops due to AI queue full: 0 Drops due to Fixup queue full: 0
The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP, probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforce configuration dependencies and requirements before pushing the configuration to the data plane.
9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by entering the following command:
ACE_module5/Admin# show processes cpu | inc util CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%
The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections (such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing or inspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectional concurrent connection limit regardless of the features enabled.
Table 1. Concurrent Connection Support
| Connection Type | ACE Module Limit |
| Layer 4 | 4,000,000 |
| Layer 7 | 512,000 |
The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained with distinct connection objects.
10. Display the connection table by entering the following command:
ACE_module5/Admin# show conn total current connections : 6 conn-id np dir proto vlan source destination state ----------+--+---+-----+----+---------------------+---------------------+------+ 1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB 2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB 4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB 3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB 4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB 3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB
| Note: | You can add the detail command option to provide the following additional fields: connection idle time, elapsed time of the connection, byte count, and packet count for each connection object. |
The total current connections counter is also maintained in the output of the following command:
switch/Admin# show stats connection +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 124 Total Connections Current : 6 Total Connections Destroyed: 62 Total Connections Timed-out: 58 Total Connections Failed : 0
| Note: | The Total Connections Current counter counts the number of used connection objects, not the number of TCP flows. The number of TCP flows can be roughly determined as half the number of connection objects minus any UDP connections. The Total Connections Current counter is always up to date and the maximum value can be 8,000,000. |
Because of the Cisco ACE Module’s architecture, with distinct paths for new and established connections, the number of existing concurrent connections does not heavily impact the rate at which new connections can be set up. Nevertheless, a very large number of concurrent connections will eventually affect the performance of the system in setting up new connections.
11. Use the command "tcp wan-optimization rtt 0" for slow connections.
The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order to increase performance for a given connection. The LB decision is made in the software (proxy) and then moved to the fastpath (unproxy). In a persistence rebalance scenario, the proxy/unproxy can occur Many times on a given connection. It is possible that if a packet enters the system during the transition Between the proxy and unproxy states, a packet may not be forwarded as expected and a retransmission may be relied upon. This can affect performance. As a workaround, it is possible to configure the ACE such that fastpath forwarding is prohibited This can be accomplished by configuring a parameter map with the following:
"tcp wan-optimization rtt 0"
