Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Connectivity
From DocWiki
This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with your ACE.
Contents |
Overview of ACE Connection Handling
This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections, the ACE receives a TCP packet from a client and load balances the connection to a server on the first packet (see Figure 1). The SYN-ACK from the server matches an existing flow and the rest of the connection is handled in the fast path (hardware accelerated path in the network processors), which is represented here as "shortcut." The ACE completes the TCP handshake . This process applies to the following functions:
- Basic load balancing
- Source IP sticky
- TCP/IP normalization
- Figure 1. Layer 4 Flow Setup
For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as a proxy (spoofs the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCP connection. See Figure 2. The ACE sends a SYN-ACK to the client in response to the client's TCP SYN. The client responds with an ACK to complete the TCP handshake and an L7 request method (for example, HTTP GET or POST).
- Figure 2. Layer 7 Flow Setup -- Client Connection
After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to the real server based on the load-balancing method and other criteria. See Figure 3.
- Figure 3. Layer 7 Flow Setup -- Server Connection
Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection to the server. For the life of the HTTP flow, the client communicates directly with the server through the fast path (hardware-accelerated path in the network processors), which is depicted in the figures as "Shortcut." See Figures 4.
- Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together
Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flows together.
- Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers
With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parses subsequent HTTP 1.1 requests over the same TCP connection. In this case, the ACE again spoofs the server and ACKs the HTTP GET as shown in Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for each new HTTP 1.1 request over the same TCP connection.
- Figure 6. Layer 7 Flow Setup -- Reproxy
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fully terminates the client TCP connection. This connection remains fully proxied because the ACE is acting on behalf of the real server. For SSL termination, the ACE completes an SSL handshake after it establishes the TCP connection with the server. See Figure 7.
- Figure 7. SSL Handshake
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client and server connections are completely independent and flows are handled in the software, not in the fast path. See Figure 8.
- Figure 8. Layer 7 Flow Setup -- Full Proxy
Internal Mapping of ACE TCP and UDP Flows
The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You can display the current connections in the ACE by entering the show connections command. See Figure 8.
- Figure 8. Internal Flow Mapping
ACE Connection Table Entries
Understanding ACE’s Conn Table Entries During:
- L4 TCP Connection Setup (3 Way Handshake)
- Normalisation Enabled
- Normalisation Disabled
- L7 TCP Connection Setup (3 Way Handshake)
- TCP Connection Teardown
- 3 Way Handshake
- 4 Way Handshake
- Reset
Tracking Connections Through the ACE
You can display the IDs for the request and response connections in the ACE by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-c 9" Connection ID:seq: 9[0x9].6 <------- Request and response connection ID Other ConnID : 3[0x3].4 Proxy ConnID : 0[0x0].0 Next Q : 1124073484[0x4300000c] 192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 17 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 0 EncapsID:ver : 0:0 TCP ACK delta : 0x0 MSS : 0 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 152 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 0 Byte Count : 0 TCP Information: (State = 0) Window size : 0 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 0 Last seq : 0 timestamp_delta: 0 Last ack : 1ce862 No Trigger : 0 Trigger Status : 0 Timestamp : 26ebc96d TCP options negotiated: Sack:Allow TS:Allow Windowscale: Allow Reserved: Allow Exceed MSS: Allow Window var: Allow
You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option of the show np command as follows:
ACE_module5/Admin# show np 1 me-stats "-c 9 -v" Connection ID:seq: 9[0x9].2 Other ConnID : 7[0x7].14 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0] 10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1260 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 4 Packet Count : 347 Byte Count : 24476 TCP Information: (State = 3) Window size : 5840 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 53768a51 timestamp_delta: 0 Last ack : 658c1f72 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000 Raw Connection Entry 0000 0x00000000 0x0a56d786 0xa12c438f 0x06210007 0010 0x001712e5 0x00000000 0x00030000 0x04ec1004 0020 0x4e000007 0x00000000 0x00080480 0x24450000 0030 0x0000015b 0x00005f9c 0x16d00030 0x05b40000 0040 0x53768a51 0x658c1f72 0x459f781e 0x00000000 0050 0x00000094 0x00000000 0x45729985 0x00000000 0060 0x00000000 0x00000000 0x00000000 0x00000000 Doing verbose output for proxy id: 0 No valid proxy entry. No valid TCB proxy entry. No valid HTTP proxy entry. No valid SSL proxy entry. No valid AI proxy entry. Connection ID:seq: 7[0x7].14 Other ConnID : 9[0x9].2 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0] 172.27.16.143:4837 -> 10.1.1.5:23 [RX-NextHop: CP] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 1 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1460 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 486 Byte Count : 19810 TCP Information: (State = 3) Window size : 65371 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 658c1f72 timestamp_delta: 0 Last ack : 53768a51 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000 Raw Connection Entry 0000 0x00000000 0xa12c438f 0x0a56d786 0x06e90007 0010 0x12e50017 0x00000000 0x00030000 0x05b41000 0020 0x02000009 0x00000000 0x00080481 0x24450000 0030 0x000001e6 0x00004d62 0xff5b0030 0x05b40000 0040 0x658c1f72 0x53768a51 0x459f781e 0x00000000 0050 0x00000094 0x00000000 0x45729985 0x00000000 0060 0x00000000 0x00000000 0x00000000 0x00000000 Doing verbose output for proxy id: 0 No valid proxy entry. No valid TCB proxy entry. No valid HTTP proxy entry. No valid SSL proxy entry. No valid AI proxy entry.
Troubleshooting Connections
To troubleshoot suspected connectivity issues, follow these steps:
1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, go to Step 2. Otherwise, verify that the access list is configured properly to permit traffic.
ACE_module5/Admin# show access-list anyone detail access-list:anyone, elements: 1, status: ACTIVE remark : access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count
2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0, verify that the service policy is active (show service-policy command) and the server farm is up (show server-farm detail command). If the service policy is large, use the show service-policy policy_name summary command for more information as follows:
ACE_module5/Admin# show service-policy VIP summary service-policy: VIP Class VIP Prot Port VLAN State Curr Conns Hit Count Conns Drop VIP 192.168.12.192 tcp eq 443 100 IN-SRVC 0 0 0 192.168.12.192 tcp eq 80 100 IN-SRVC 0 0 0 192.168.12.193 tcp eq 80 100 IN-SRVC 0 0 0 192.168.12.193 tcp eq 443 100 IN-SRVC 0 0 0 VIP2 192.168.12.194 tcp eq 80 100 IN-SRVC 0 0 0 192.168.12.194 tcp eq 443 100 IN-SRVC 0 0 0
3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7 rejections or the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for any misconfiguration.
ACE_module5/Admin# show stats loadbalance +------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 3 <-------| Total Layer7 decisions : 0 |------- Failed connections due to traffic not matching the configured class maps Total Layer7 rejections : 7 <-------| Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 10 <------- Failed connections due to no real server' Total ACL denied : 0 Total IDMap Lookup Failures : 0
To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalance command.
4. If none of the error statistics is increasing, check the connection record by entering the show conn detail command and checking the connections for the affected VIP.
ACE_module5/Admin# show conn detail total current connections : 6 conn-id np dir proto vlan source destination state ----------+--+---+-----+----+---------------------+---------------------+------+ 7 1 in TCP 130 10.1.1.2:1171 10.1.1.134:23 ESTAB [ idle time : 00:00:00, byte count : 60055 ] [ elapsed time: 04:15:29, packet count: 1473 ] 9 1 out TCP 130 10.1.1.134:23 10.1.2.74:1171 ESTAB [ conn in reuse pool : FALSE] [ idle time : 00:00:00, byte count : 64880 ] [ elapsed time: 04:15:29, packet count: 1086 ]
5. Display existing ACE connection statistics by entering the following command:
ACE_module5/Admin# show stats connection +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 628950 Total Connections Current : 7 Total Connections Destroyed: 389 Total Connections Timed-out: 3958 Total Connections Failed : 624596 <------- Server did not reply to a SYN within the pending timeout period or it replied with a RST
The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with the server. To clear the statistical information stored in the ACE buffer, enter the clear stats connection command.
6. Display service policy statistics by entering the following command:
ACE/Context# show service-policy client-vips detail Status : ACTIVE Description: - ----------------------------------------- Interface: vlan 211 service-policy: client-vips class: VIP-HTTPS VIP Address: Protocol: Port: 172.16.11.190 tcp eq 443 <------- Shows the VIP address, port, and protocol loadbalance: L7 loadbalance policy: HTTPS-POLICY VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED-WHEN-ACTIVE VIP State: INSERVICE <------- Service is INSERVICE curr conns : 22 , hit count : 22 dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 max-conn-limit : 0 , drop-count : 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : HTTPS-POLICY class/match : class-default LB action : primary serverfarm: backend-ssl backup serverfarm : - hit count : 22 <------- Shows the hit count dropped conns : 0
7. Display server farm connection statistics by entering the following command:
ACE/Context# show serverfarm HTTPS-FARM detail serverfarm : HTTPS-FARM, type: HOST total rservers : 4 active rservers: 4 description : - state : ACTIVE predictor : ROUNDROBIN <------- Shows the load-balancing predictor that was used failaction : - back-inservice : 0 partial-threshold : 0 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 --------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+--------+---------------------+-----------+----------+--------- rserver: linux-1 192.168.1.11:0 8 OPERATIONAL 0 0 0 <------- Shows connection statistics for each real server max-conns : - , out-of-rotation count : - min-conns : - conn-rate-limit : - , out-of-rotation count : - bandwidth-rate-limit : - , out-of-rotation count : - retcode out-of-rotation count : -
The Connections Failures counter for a real server in a server farm may increment for one of the following reasons:
- SYN timeout (the three-way handshake fails to complete)
- RST received (a client sends an RST to the server)
- Internal exception (internal software issue)
8. Display the statistics for a connection parameter map by entering the following command:
ACE_module5/Admin# show parameter-map CONN_PARAMMAP Number of parameter-maps : 1 Parameter-map : CONN_PARAMMAP Type : connection nagle : disabled slow start : disabled buffer-share size : 32768 inactivity timeout (seconds) : TCP: 3600, UDP: 120, ICMP: 2 embryonic timeout (seconds) : 5 ack-delay (milliseconds) : 200 WAN Optimization RTT (milliseconds): 65535 half-closed timeout (seconds) : 3600 TOS rewrite : disabled syn retry count : 4 TCP MSS min : 0 TCP MSS max : 1460 tcp-options drop range : 0-0 tcp-options allow range : 0-0 tcp-options clear range : 1-255 selective-ack : clear timestamp : clear window-scale : clear window-scale factor : 0 reserved-bits : allow random-seq-num : enabled SYN data : drop exceed-mss : drop urgent-flag : allow conn-rate-limit : disabled bandwidth-rate-limit : disabled
9. Reset the ACE connection statistics by entering the following commands:
- clear conn [all | flow {icmp | tcp | udp} | rserver server_name]
- clear stats conn
- clear tcp statistics
- clear udp statistics