Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Connectivity

From DocWiki

(Difference between revisions)
Jump to: navigation, search
(ACE Connection Table Entries)
(ACE Connection Table Entries)
Line 128: Line 128:
**Reset
**Reset
-
[[Image:L4-connection-setup1.jpg|480px]]
+
[[Image:L4-connection-setup1.jpg|600px]]
[[Image:L4-connection-setup2.jpg]]
[[Image:L4-connection-setup2.jpg]]
[[Image:L4-connection-setup3.jpg]]
[[Image:L4-connection-setup3.jpg]]

Revision as of 09:17, 1 April 2010

This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with your ACE.

Guide Contents
Main Article
Overview of ACE Module Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Module Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting Remote Access
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Layer 7 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Performance Issues
ACE Module Resource Limits
Managing ACE Resources

Contents












Overview of ACE Connection Handling

This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections, the ACE receives a TCP packet from a client and load balances the connection to a server on the first packet (see Figure 1). The SYN-ACK from the server matches an existing flow and the rest of the connection is handled in the fast path (hardware accelerated path in the network processors), which is represented here as "shortcut." The ACE completes the TCP handshake . This process applies to the following functions:

  • Basic load balancing
  • Source IP sticky
  • TCP/IP normalization


Figure 1. Layer 4 Flow Setup


Layer 4 Flow Setup.jpg


For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as a proxy (spoofs the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCP connection. See Figure 2. The ACE sends a SYN-ACK to the client in response to the client's TCP SYN. The client responds with an ACK to complete the TCP handshake and an L7 request method (for example, HTTP GET or POST).


Figure 2. Layer 7 Flow Setup -- Client Connection


Layer 7 Flow Setup1.jpg


After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to the real server based on the load-balancing method and other criteria. See Figure 3.


Figure 3. Layer 7 Flow Setup -- Server Connection


Layer 7 Flow Setup2 Server.jpg


Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection to the server. For the life of the HTTP flow, the client communicates directly with the server through the fast path (hardware-accelerated path in the network processors), which is depicted in the figures as "Shortcut." See Figures 4.


Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together


Layer 7 Flow Setup3 Unproxy.jpg


Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flows together.


Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers


Layer 7 Flow Setup4 Splice.jpg


With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parses subsequent HTTP 1.1 requests over the same TCP connection. In this case, the ACE again spoofs the server and ACKs the HTTP GET as shown in Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for each new HTTP 1.1 request over the same TCP connection.


Figure 6. Layer 7 Flow Setup -- Reproxy


Layer 7 Flow Setup5 Reproxy.jpg


For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fully terminates the client TCP connection. This connection remains fully proxied because the ACE is acting on behalf of the real server. For SSL termination, the ACE completes an SSL handshake after it establishes the TCP connection with the server. See Figure 7.


Figure 7. SSL Handshake


SSL handshake.jpg


For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client and server connections are completely independent and flows are handled in the software, not in the fast path. See Figure 8.


Figure 8. Layer 7 Flow Setup -- Full Proxy


Layer 7 Flow Setup6 FullProxy.jpg

Internal Mapping of ACE TCP and UDP Flows

The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You can display the current connections in the ACE by entering the show connections command. See Figure 8.


Figure 8. Internal Flow Mapping


Internal Flow Mapping.jpg


ACE Connection Table Entries

Understanding ACE’s Conn Table Entries During:

  • L4 TCP Connection Setup (3 Way Handshake)
    • Normalisation Enabled
    • Normalisation Disabled
  • L7 TCP Connection Setup (3 Way Handshake)
  • TCP Connection Teardown
    • 3 Way Handshake
    • 4 Way Handshake
    • Reset

L4-connection-setup1.jpg L4-connection-setup2.jpg L4-connection-setup3.jpg L4-connection-setup4.jpg L7-connection-setup1.jpg

Tracking Connections Through the ACE

You can display the IDs for the request and response connections in the ACE by entering the following command:

ACE_module5/Admin# show np 1 me-stats "-c 9"
Connection ID:seq: 9[0x9].6 <------- Request and response connection ID
  Other ConnID    : 3[0x3].4
  Proxy ConnID    : 0[0x0].0
  Next Q    : 1124073484[0x4300000c]

192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX]
  Flags:  PAT: No  DynNAT: No  Implicit PAT: No On_Reuse: No
  L3 Protocol     : IPv4                L4 Protocol    : 17
  Inbound Flag    : 0
  Interface Match : Yes
    Interface MatchID: 0
  EncapsID:ver    : 0:0         TCP ACK delta  : 0x0
  MSS             : 0           TOS Stamp       : 0
  Repeat mode     : No          ARP Lookup      : No
  TOS Stamp       : No          TCP Window Check: No
  ACE ID          : 152         NAT Policy ID       : 0
  Post NAT hop    : 0
  Packet Count    : 0           Byte Count          : 0
  TCP Information: (State = 0)
    Window size   : 0           Window scale    : 0
    FIN seen      : No          FIN/ACK seen    : No
    FIN/ACK exp   : No          Close initiator : No
    FIN/ACK expval: 0           Last seq        : 0
   timestamp_delta: 0           Last ack        : 1ce862
    No Trigger    : 0           Trigger Status   : 0
    Timestamp : 26ebc96d
  TCP options negotiated:
    Sack:Allow          TS:Allow        Windowscale:  Allow
    Reserved: Allow     Exceed MSS: Allow       Window var: Allow


You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option of the show np command as follows:

ACE_module5/Admin# show np 1 me-stats "-c 9 -v"
Connection ID:seq: 9[0x9].2
  Other ConnID    : 7[0x7].14
  Proxy ConnID    : 0[0x0].0
  Next Q    : 0[0x0]

10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP]
  Flags:  PAT: No  DynNAT: No  Implicit PAT: No On_Reuse: No
  L3 Protocol     : IPv4                L4 Protocol    : 6
  Inbound Flag    : 0
  Interface Match : Yes
    Interface MatchID: 7
  EncapsID:ver    : 3:0         TCP ACK delta  : 0x0
  MSS             : 1260                TOS Stamp       : 0
  Repeat mode     : No          ARP Lookup      : No
  TOS Stamp       : No          TCP Window Check: No
  ACE ID          : 148         NAT Policy ID       : 0
  Post NAT hop    : 4
  Packet Count    : 347         Byte Count          : 24476
  TCP Information: (State = 3)
    Window size   : 5840                Window scale    : 0
    FIN seen      : No          FIN/ACK seen    : No
    FIN/ACK exp   : No          Close initiator : No
    FIN/ACK expval: 5b40000             Last seq        : 53768a51
   timestamp_delta: 0           Last ack        : 658c1f72
    No Trigger    : 0           Trigger Status   : 0
    Timestamp : 459f781e
  TCP options negotiated:
    Sack:Clear          TS:Clear        Windowscale:  Clear
    Reserved: Allow     Exceed MSS:  Deny       Window var: Allow
  Flags:  debug: 0              TCP Normalize: Yes
          Syslog: No    Reproxy Request: No   Policying Reqd: No
          Inbound IPSec: No  Replicated: No  Data Channel: No
          L7: No  Fin Detect: Yes  FP Timeout: No
          Standby: No  ConnState: 2
          ACA Method: 0  ReqTS: 00000000  RspTS: 00000000

Raw Connection Entry
0000  0x00000000  0x0a56d786  0xa12c438f  0x06210007
0010  0x001712e5  0x00000000  0x00030000  0x04ec1004
0020  0x4e000007  0x00000000  0x00080480  0x24450000
0030  0x0000015b  0x00005f9c  0x16d00030  0x05b40000
0040  0x53768a51  0x658c1f72  0x459f781e  0x00000000
0050  0x00000094  0x00000000  0x45729985  0x00000000
0060  0x00000000  0x00000000  0x00000000  0x00000000

Doing verbose output for proxy id: 0

No valid proxy entry.
No valid TCB proxy entry.
No valid HTTP proxy entry.
No valid SSL proxy entry.
No valid AI proxy entry.
Connection ID:seq: 7[0x7].14
  Other ConnID    : 9[0x9].2
  Proxy ConnID    : 0[0x0].0
  Next Q    : 0[0x0]

172.27.16.143:4837 -> 10.1.1.5:23 [RX-NextHop: CP] [TX-NextHop: TX]
  Flags:  PAT: No  DynNAT: No  Implicit PAT: No On_Reuse: No
  L3 Protocol     : IPv4                L4 Protocol    : 6
  Inbound Flag    : 1
  Interface Match : Yes
    Interface MatchID: 7
  EncapsID:ver    : 3:0         TCP ACK delta  : 0x0
  MSS             : 1460                TOS Stamp       : 0
  Repeat mode     : No          ARP Lookup      : No
  TOS Stamp       : No          TCP Window Check: No
  ACE ID          : 148         NAT Policy ID       : 0
  Post NAT hop    : 0
  Packet Count    : 486         Byte Count          : 19810
  TCP Information: (State = 3)
    Window size   : 65371               Window scale    : 0
    FIN seen      : No          FIN/ACK seen    : No
    FIN/ACK exp   : No          Close initiator : No
    FIN/ACK expval: 5b40000             Last seq        : 658c1f72
   timestamp_delta: 0           Last ack        : 53768a51
    No Trigger    : 0           Trigger Status   : 0
    Timestamp : 459f781e
  TCP options negotiated:
    Sack:Clear          TS:Clear        Windowscale:  Clear
    Reserved: Allow     Exceed MSS:  Deny       Window var: Allow
  Flags:  debug: 0              TCP Normalize: Yes
          Syslog: No    Reproxy Request: No   Policying Reqd: No
          Inbound IPSec: No  Replicated: No  Data Channel: No
          L7: No  Fin Detect: Yes  FP Timeout: No
          Standby: No  ConnState: 2
          ACA Method: 0  ReqTS: 00000000  RspTS: 00000000

Raw Connection Entry
0000  0x00000000  0xa12c438f  0x0a56d786  0x06e90007
0010  0x12e50017  0x00000000  0x00030000  0x05b41000
0020  0x02000009  0x00000000  0x00080481  0x24450000
0030  0x000001e6  0x00004d62  0xff5b0030  0x05b40000
0040  0x658c1f72  0x53768a51  0x459f781e  0x00000000
0050  0x00000094  0x00000000  0x45729985  0x00000000
0060  0x00000000  0x00000000  0x00000000  0x00000000 

Doing verbose output for proxy id: 0

No valid proxy entry.
No valid TCB proxy entry.
No valid HTTP proxy entry.
No valid SSL proxy entry.
No valid AI proxy entry.

Troubleshooting Connections

To troubleshoot suspected connectivity issues, follow these steps:

1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, go to Step 2. Otherwise, verify that the access list is configured properly to permit traffic.

ACE_module5/Admin# show access-list anyone detail
access-list:anyone, elements: 1, status: ACTIVE
  remark :
access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count

2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0, verify that the service policy is active (show service-policy command) and the server farm is up (show server-farm detail command). If the service policy is large, use the show service-policy policy_name summary command for more information as follows:

ACE_module5/Admin# show service-policy VIP summary
 
service-policy: VIP
Class                            VIP             Prot  Port        VLAN
 State    Curr Conns   Hit Count  Conns Drop
VIP                              192.168.12.192   tcp   eq 443      100
 IN-SRVC           0           0          0
                                 192.168.12.192   tcp   eq 80       100
 IN-SRVC           0           0          0
                                 192.168.12.193   tcp   eq 80       100
 IN-SRVC           0           0          0
                                 192.168.12.193   tcp   eq 443      100
 IN-SRVC           0           0          0
VIP2                             192.168.12.194   tcp   eq 80       100
 IN-SRVC           0           0          0
                                 192.168.12.194   tcp   eq 443      100
 IN-SRVC           0           0          0

3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7 rejections or the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for any misconfiguration.

ACE_module5/Admin# show stats loadbalance

 +------------------------------------------+
 +------- Loadbalance statistics -----------+
 +------------------------------------------+
  Total version mismatch              : 0
  Total Layer4 decisions              : 0
  Total Layer4 rejections             : 3 <-------|
  Total Layer7 decisions              : 0         |------- Failed connections due to traffic not matching the configured class maps 
  Total Layer7 rejections             : 7 <-------|
  Total Layer4 LB policy misses       : 0
  Total Layer7 LB policy misses       : 0
  Total times rserver was unavailable : 10 <------- Failed connections due to no real server'
  Total ACL denied                    : 0
  Total IDMap Lookup Failures         : 0

To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalance command.

4. If none of the error statistics is increasing, check the connection record by entering the show conn detail command and checking the connections for the affected VIP.

ACE_module5/Admin# show conn detail

total current connections : 6

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
7          1  in  TCP   130  10.1.1.2:1171     10.1.1.134:23      ESTAB
          [ idle time   : 00:00:00,   byte count  : 60055      ]
          [ elapsed time: 04:15:29,   packet count: 1473       ]
9          1  out TCP   130  10.1.1.134:23      10.1.2.74:1171     ESTAB
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:00,   byte count  : 64880      ]
          [ elapsed time: 04:15:29,   packet count: 1086       ]

5. Display existing ACE connection statistics by entering the following command:

ACE_module5/Admin# show stats connection

+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
 Total Connections Created  : 628950
 Total Connections Current  : 7
 Total Connections Destroyed: 389
 Total Connections Timed-out: 3958
 Total Connections Failed   : 624596 <------- Server did not reply to a SYN within the pending timeout period or it replied with a RST


The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with the server. To clear the statistical information stored in the ACE buffer, enter the clear stats connection command.

6. Display service policy statistics by entering the following command:

ACE/Context# show service-policy client-vips detail
Status     : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211 
  service-policy: client-vips
    class: VIP-HTTPS
     VIP Address:    Protocol:  Port:
     172.16.11.190   tcp        eq    443  <------- Shows the VIP address, port, and protocol
      loadbalance:
        L7 loadbalance policy: HTTPS-POLICY
        VIP Route Metric     : 77
        VIP Route Advertise  : DISABLED
        VIP ICMP Reply       : ENABLED-WHEN-ACTIVE
        VIP State: INSERVICE  <------- Service is INSERVICE
        curr conns       : 22        , hit count        : 22        
        dropped conns    : 0         
        client pkt count : 0         , client byte count: 0                   
        server pkt count : 0         , server byte count: 0                   
        max-conn-limit       : 0         , drop-count : 0         
        conn-rate-limit      : 0         , drop-count : 0         
        bandwidth-rate-limit : 0         , drop-count : 0         
        L7 Loadbalance policy : HTTPS-POLICY
          class/match : class-default
            LB action : 
               primary serverfarm: backend-ssl
               backup serverfarm : -
            hit count        : 22  <------- Shows the hit count   
            dropped conns    : 0


7. Display server farm connection statistics by entering the following command:

ACE/Context# show serverfarm HTTPS-FARM detail
serverfarm     : HTTPS-FARM, type: HOST
 total rservers : 4
 active rservers: 4
 description    : -
 state          : ACTIVE
 predictor      : ROUNDROBIN  <------- Shows the load-balancing predictor that was used
 failaction     : -
 back-inservice    : 0
 partial-threshold : 0
 num times failover       : 0
 num times back inservice : 0
 total conn-dropcount : 0
 ---------------------------------
                                                ----------connections-----------
       real                   weight  state                 current     total      failures 
   ---+---------------------+--------+---------------------+-----------+----------+---------
   rserver: linux-1
       192.168.1.11:0           8     OPERATIONAL           0            0          0  <------- Shows connection statistics for
                                                                                                each real server
         max-conns            : -         , out-of-rotation count : -
         min-conns            : -         
         conn-rate-limit      : -         , out-of-rotation count : -
         bandwidth-rate-limit : -         , out-of-rotation count : -
         retcode out-of-rotation count : -

The Connections Failures counter for a real server in a server farm may increment for one of the following reasons:

  • SYN timeout (the three-way handshake fails to complete)
  • RST received (a client sends an RST to the server)
  • Internal exception (internal software issue)

8. Display the statistics for a connection parameter map by entering the following command:

ACE_module5/Admin# show parameter-map CONN_PARAMMAP

 Number of parameter-maps : 1


 Parameter-map : CONN_PARAMMAP
 Type : connection
    nagle                              : disabled
    slow start                         : disabled
    buffer-share size                  : 32768
    inactivity timeout (seconds)       : TCP: 3600, UDP: 120, ICMP: 2
    embryonic timeout (seconds)        : 5
    ack-delay (milliseconds)           : 200
    WAN Optimization RTT (milliseconds): 65535
    half-closed timeout (seconds)      : 3600
    TOS rewrite                        : disabled
    syn retry count                    : 4
    TCP MSS min                        : 0
    TCP MSS max                        : 1460
    tcp-options drop range             : 0-0
    tcp-options allow range            : 0-0
    tcp-options clear range            : 1-255
    selective-ack                      : clear
    timestamp                          : clear
    window-scale                       : clear
    window-scale factor                : 0
    reserved-bits                      : allow
    random-seq-num                     : enabled
    SYN data                           : drop
    exceed-mss                         : drop
    urgent-flag                        : allow
    conn-rate-limit                    : disabled
    bandwidth-rate-limit               : disabled

9. Reset the ACE connection statistics by entering the following commands:

  • clear conn [all | flow {icmp | tcp | udp} | rserver server_name]
  • clear stats conn
  • clear tcp statistics
  • clear udp statistics

Rating: 4.8/5 (18 votes cast)

Personal tools