Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Connectivity
From DocWiki
This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with your ACE.
Contents |
Overview of ACE Connection Handling
This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections, the ACE receives a TCP packet from a client and load balances the connection to a server on the first packet (see Figure 1). The SYN-ACK from the server matches an existing flow and the rest of the connection is handled in the fast path (hardware accelerated path in the network processors), which is represented here as "shortcut." The ACE completes the TCP handshake . This process applies to the following functions:
- Basic load balancing
- Source IP sticky
- TCP/IP normalization
- Figure 1. Layer 4 Flow Setup
For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as a proxy (spoofs the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCP connection. See Figure 2. The ACE sends a SYN-ACK to the client in response to the client's TCP SYN. The client responds with an ACK to complete the TCP handshake and an L7 request method (for example, HTTP GET or POST).
- Figure 2. Layer 7 Flow Setup -- Client Connection
After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to the real server based on the load-balancing method and other criteria. See Figure 3.
- Figure 3. Layer 7 Flow Setup -- Server Connection
Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection to the server. For the life of the HTTP flow, the client communicates directly with the server through the fast path (hardware-accelerated path in the network processors), which is depicted in the figures as "Shortcut." See Figures 4.
- Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together
Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flows together.
- Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers
With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parses subsequent HTTP 1.1 requests over the same TCP connection. In this case, the ACE again spoofs the server and ACKs the HTTP GET as shown in Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for each new HTTP 1.1 request over the same TCP connection.
- Figure 6. Layer 7 Flow Setup -- Reproxy
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fully terminates the client TCP connection. This connection remains fully proxied because the ACE is acting on behalf of the real server. For SSL termination, the ACE completes an SSL handshake after it establishes the TCP connection with the server. See Figure 7.
- Figure 7. SSL Handshake
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client and server connections are completely independent and flows are handled in the software, not in the fast path. See Figure 8.
- Figure 8. Layer 7 Flow Setup -- Full Proxy
Internal Mapping of ACE TCP and UDP Flows
The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You can display the current connections in the ACE by entering the show connections command. See Figure 8.
- Figure 8. Internal Flow Mapping
ACE Connection Table Entries
Understanding ACE’s Conn Table Entries During:
- L4 TCP Connection Setup (3 Way Handshake)
- Normalisation Enabled
- Normalisation Disabled
- L7 TCP Connection Setup (3 Way Handshake)
- TCP Connection Teardown
- 3 Way Handshake
- 4 Way Handshake
- Reset
Tracking Connections Through the ACE
You can display the IDs for the request and response connections in the ACE by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-c 9"
Connection ID:seq: 9[0x9].6 <------- Request and response connection ID
Other ConnID : 3[0x3].4
Proxy ConnID : 0[0x0].0
Next Q : 1124073484[0x4300000c]
192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX]
Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No
L3 Protocol : IPv4 L4 Protocol : 17
Inbound Flag : 0
Interface Match : Yes
Interface MatchID: 0
EncapsID:ver : 0:0 TCP ACK delta : 0x0
MSS : 0 TOS Stamp : 0
Repeat mode : No ARP Lookup : No
TOS Stamp : No TCP Window Check: No
ACE ID : 152 NAT Policy ID : 0
Post NAT hop : 0
Packet Count : 0 Byte Count : 0
TCP Information: (State = 0)
Window size : 0 Window scale : 0
FIN seen : No FIN/ACK seen : No
FIN/ACK exp : No Close initiator : No
FIN/ACK expval: 0 Last seq : 0
timestamp_delta: 0 Last ack : 1ce862
No Trigger : 0 Trigger Status : 0
Timestamp : 26ebc96d
TCP options negotiated:
Sack:Allow TS:Allow Windowscale: Allow
Reserved: Allow Exceed MSS: Allow Window var: Allow
You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option of the show np command as follows:
ACE_module5/Admin# show np 1 me-stats "-c 9 -v"
Connection ID:seq: 9[0x9].2
Other ConnID : 7[0x7].14
Proxy ConnID : 0[0x0].0
Next Q : 0[0x0]
10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP]
Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No
L3 Protocol : IPv4 L4 Protocol : 6
Inbound Flag : 0
Interface Match : Yes
Interface MatchID: 7
EncapsID:ver : 3:0 TCP ACK delta : 0x0
MSS : 1260 TOS Stamp : 0
Repeat mode : No ARP Lookup : No
TOS Stamp : No TCP Window Check: No
ACE ID : 148 NAT Policy ID : 0
Post NAT hop : 4
Packet Count : 347 Byte Count : 24476
TCP Information: (State = 3)
Window size : 5840 Window scale : 0
FIN seen : No FIN/ACK seen : No
FIN/ACK exp : No Close initiator : No
FIN/ACK expval: 5b40000 Last seq : 53768a51
timestamp_delta: 0 Last ack : 658c1f72
No Trigger : 0 Trigger Status : 0
Timestamp : 459f781e
TCP options negotiated:
Sack:Clear TS:Clear Windowscale: Clear
Reserved: Allow Exceed MSS: Deny Window var: Allow
Flags: debug: 0 TCP Normalize: Yes
Syslog: No Reproxy Request: No Policying Reqd: No
Inbound IPSec: No Replicated: No Data Channel: No
L7: No Fin Detect: Yes FP Timeout: No
Standby: No ConnState: 2
ACA Method: 0 ReqTS: 00000000 RspTS: 00000000
Raw Connection Entry
0000 0x00000000 0x0a56d786 0xa12c438f 0x06210007
0010 0x001712e5 0x00000000 0x00030000 0x04ec1004
0020 0x4e000007 0x00000000 0x00080480 0x24450000
0030 0x0000015b 0x00005f9c 0x16d00030 0x05b40000
0040 0x53768a51 0x658c1f72 0x459f781e 0x00000000
0050 0x00000094 0x00000000 0x45729985 0x00000000
0060 0x00000000 0x00000000 0x00000000 0x00000000
Doing verbose output for proxy id: 0
No valid proxy entry.
No valid TCB proxy entry.
No valid HTTP proxy entry.
No valid SSL proxy entry.
No valid AI proxy entry.
Connection ID:seq: 7[0x7].14
Other ConnID : 9[0x9].2
Proxy ConnID : 0[0x0].0
Next Q : 0[0x0]
172.27.16.143:4837 -> 10.1.1.5:23 [RX-NextHop: CP] [TX-NextHop: TX]
Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No
L3 Protocol : IPv4 L4 Protocol : 6
Inbound Flag : 1
Interface Match : Yes
Interface MatchID: 7
EncapsID:ver : 3:0 TCP ACK delta : 0x0
MSS : 1460 TOS Stamp : 0
Repeat mode : No ARP Lookup : No
TOS Stamp : No TCP Window Check: No
ACE ID : 148 NAT Policy ID : 0
Post NAT hop : 0
Packet Count : 486 Byte Count : 19810
TCP Information: (State = 3)
Window size : 65371 Window scale : 0
FIN seen : No FIN/ACK seen : No
FIN/ACK exp : No Close initiator : No
FIN/ACK expval: 5b40000 Last seq : 658c1f72
timestamp_delta: 0 Last ack : 53768a51
No Trigger : 0 Trigger Status : 0
Timestamp : 459f781e
TCP options negotiated:
Sack:Clear TS:Clear Windowscale: Clear
Reserved: Allow Exceed MSS: Deny Window var: Allow
Flags: debug: 0 TCP Normalize: Yes
Syslog: No Reproxy Request: No Policying Reqd: No
Inbound IPSec: No Replicated: No Data Channel: No
L7: No Fin Detect: Yes FP Timeout: No
Standby: No ConnState: 2
ACA Method: 0 ReqTS: 00000000 RspTS: 00000000
Raw Connection Entry
0000 0x00000000 0xa12c438f 0x0a56d786 0x06e90007
0010 0x12e50017 0x00000000 0x00030000 0x05b41000
0020 0x02000009 0x00000000 0x00080481 0x24450000
0030 0x000001e6 0x00004d62 0xff5b0030 0x05b40000
0040 0x658c1f72 0x53768a51 0x459f781e 0x00000000
0050 0x00000094 0x00000000 0x45729985 0x00000000
0060 0x00000000 0x00000000 0x00000000 0x00000000
Doing verbose output for proxy id: 0
No valid proxy entry.
No valid TCB proxy entry.
No valid HTTP proxy entry.
No valid SSL proxy entry.
No valid AI proxy entry.
Troubleshooting Connections
To troubleshoot suspected connectivity issues, follow these steps:
1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, go to Step 2. Otherwise, verify that the access list is configured properly to permit traffic.
ACE_module5/Admin# show access-list anyone detail access-list:anyone, elements: 1, status: ACTIVE remark : access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count
2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0, verify that the service policy is active (show service-policy command) and the server farm is up (show server-farm detail command). If the service policy is large, use the show service-policy policy_name summary command for more information as follows:
ACE_module5/Admin# show service-policy VIP summary
service-policy: VIP
Class VIP Prot Port VLAN
State Curr Conns Hit Count Conns Drop
VIP 192.168.12.192 tcp eq 443 100
IN-SRVC 0 0 0
192.168.12.192 tcp eq 80 100
IN-SRVC 0 0 0
192.168.12.193 tcp eq 80 100
IN-SRVC 0 0 0
192.168.12.193 tcp eq 443 100
IN-SRVC 0 0 0
VIP2 192.168.12.194 tcp eq 80 100
IN-SRVC 0 0 0
192.168.12.194 tcp eq 443 100
IN-SRVC 0 0 0
3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7 rejections or the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for any misconfiguration.
ACE_module5/Admin# show stats loadbalance +------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 3 <-------| Total Layer7 decisions : 0 |------- Failed connections due to traffic not matching the configured class maps Total Layer7 rejections : 7 <-------| Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 10 <------- Failed connections due to no real server' Total ACL denied : 0 Total IDMap Lookup Failures : 0
To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalance command.
4. If none of the error statistics is increasing, check the connection record by entering the show conn detail command and checking the connections for the affected VIP.
ACE_module5/Admin# show conn detail
total current connections : 6
conn-id np dir proto vlan source destination state
----------+--+---+-----+----+---------------------+---------------------+------+
7 1 in TCP 130 10.1.1.2:1171 10.1.1.134:23 ESTAB
[ idle time : 00:00:00, byte count : 60055 ]
[ elapsed time: 04:15:29, packet count: 1473 ]
9 1 out TCP 130 10.1.1.134:23 10.1.2.74:1171 ESTAB
[ conn in reuse pool : FALSE]
[ idle time : 00:00:00, byte count : 64880 ]
[ elapsed time: 04:15:29, packet count: 1086 ]
5. Display existing ACE connection statistics by entering the following command:
ACE_module5/Admin# show stats connection +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 628950 Total Connections Current : 7 Total Connections Destroyed: 389 Total Connections Timed-out: 3958 Total Connections Failed : 624596 <------- Server did not reply to a SYN within the pending timeout period or it replied with a RST
The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with the server. To clear the statistical information stored in the ACE buffer, enter the clear stats connection command.
6. Display service policy statistics by entering the following command:
ACE/Context# show service-policy client-vips detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211
service-policy: client-vips
class: VIP-HTTPS
VIP Address: Protocol: Port:
172.16.11.190 tcp eq 443 <------- Shows the VIP address, port, and protocol
loadbalance:
L7 loadbalance policy: HTTPS-POLICY
VIP Route Metric : 77
VIP Route Advertise : DISABLED
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE <------- Service is INSERVICE
curr conns : 22 , hit count : 22
dropped conns : 0
client pkt count : 0 , client byte count: 0
server pkt count : 0 , server byte count: 0
max-conn-limit : 0 , drop-count : 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : HTTPS-POLICY
class/match : class-default
LB action :
primary serverfarm: backend-ssl
backup serverfarm : -
hit count : 22 <------- Shows the hit count
dropped conns : 0
7. Display server farm connection statistics by entering the following command:
ACE/Context# show serverfarm HTTPS-FARM detail
serverfarm : HTTPS-FARM, type: HOST
total rservers : 4
active rservers: 4
description : -
state : ACTIVE
predictor : ROUNDROBIN <------- Shows the load-balancing predictor that was used
failaction : -
back-inservice : 0
partial-threshold : 0
num times failover : 0
num times back inservice : 0
total conn-dropcount : 0
---------------------------------
----------connections-----------
real weight state current total failures
---+---------------------+--------+---------------------+-----------+----------+---------
rserver: linux-1
192.168.1.11:0 8 OPERATIONAL 0 0 0 <------- Shows connection statistics for
each real server
max-conns : - , out-of-rotation count : -
min-conns : -
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
retcode out-of-rotation count : -
The Connections Failures counter for a real server in a server farm may increment for one of the following reasons:
- SYN timeout (the three-way handshake fails to complete)
- RST received (a client sends an RST to the server)
- Internal exception (internal software issue)
8. Display the statistics for a connection parameter map by entering the following command:
ACE_module5/Admin# show parameter-map CONN_PARAMMAP
Number of parameter-maps : 1
Parameter-map : CONN_PARAMMAP
Type : connection
nagle : disabled
slow start : disabled
buffer-share size : 32768
inactivity timeout (seconds) : TCP: 3600, UDP: 120, ICMP: 2
embryonic timeout (seconds) : 5
ack-delay (milliseconds) : 200
WAN Optimization RTT (milliseconds): 65535
half-closed timeout (seconds) : 3600
TOS rewrite : disabled
syn retry count : 4
TCP MSS min : 0
TCP MSS max : 1460
tcp-options drop range : 0-0
tcp-options allow range : 0-0
tcp-options clear range : 1-255
selective-ack : clear
timestamp : clear
window-scale : clear
window-scale factor : 0
reserved-bits : allow
random-seq-num : enabled
SYN data : drop
exceed-mss : drop
urgent-flag : allow
conn-rate-limit : disabled
bandwidth-rate-limit : disabled
9. Reset the ACE connection statistics by entering the following commands:
- clear conn [all | flow {icmp | tcp | udp} | rserver server_name]
- clear stats conn
- clear tcp statistics
- clear udp statistics
