Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Layer 4 Load Balancing

From DocWiki

Jump to: navigation, search

This article describes how to troubleshoot Layer 4 (L4) load balancing on the ACE.

Guide Contents
Main Article
Overview of ACE Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting ACE Appliance Ethernet Ports
Troubleshooting Remote Access
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Layer 7 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Compression
Troubleshooting Performance Issues
ACE Resource Limits
Managing ACE Resources
Show Counter Reference


Overview of ACE L4 Load Balancing

Load balancing at L4 involves selecting a server in a server farm to service a client request based on the VIP address and protocol in the request. You configure a class map to classify (match) interesting traffic arriving at the ACE and associate the class map with a policy map to perform an action on the traffic based on the classification. With L4 load balancing, the ACE selects a server based on the first packet it receives in a particular flow. See the "Overview of ACE Connection Handling" section in the Troubleshooting Connectivity article.

For detailed information about ACE load balancing, see the Cisco Application Control Engine Module Server Load Balancing Configuration Guide.

Classifying L4 Traffic for Server Load Balancing

You classify inbound network traffic destined to or passing through the ACE based on a series of flow match criteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is of interest to you. A policy map defines a series of actions (functions) that you want applied to a set of classified inbound or outbound traffic.

ACE L3 and L4 traffic policies support the following server load-balancing (SLB) traffic attributes:

  • Source or destination IP address
  • Source or destination port
  • Virtual IP (VIP) address
  • IP protocol

The three major steps in the traffic classification process are as follows:

  1. Create a class map using the class-map command and the associated match commands, which comprise a set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocol classifications.
  2. Create a policy map using the policy-map command, which refers to the class maps and identifies a series of actions to perform based on the traffic match criteria.
  3. Activate the policy map by associating it with a specific VLAN interface or globally with all VLAN interfaces using the service-policy command to filter the traffic received by the ACE.

Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7 policies that the ACE uses for SLB. The figure also shows how you associate the various components of the SLB policy configuration with each other.

Figure 1. SLB Flow Diagram
SLB flow.jpg

Example of a Layer 4 Load-Balancing Configuration

The following example shows a L4 load-balancing configuration:

access-list ACL1 line 10 extended permit ip any any

rserver host SERVER1
  ip address

rserver host SERVER2
  ip address

rserver host SERVER3
  ip address

rserver host SERVER4
  ip address

rserver host SERVER5
  ip address

rserver host SERVER6
  ip address

serverfarm host SFARM1
  probe TCP_PROBE
  predictor roundrobin
  rserver SERVER1
    weight 10
  rserver SERVER2
    weight 20
  rserver SERVER3
    weight 30

serverfarm host SFARM2
  probe TCP_PROBE
  predictor roundrobin
  rserver SERVER4
    weight 10
  rserver SERVER5
    weight 20
  rserver SERVER6
    weight 30

class-map match-all L4WEB_CLASS
  2 match virtual-address tcp eq www 

policy-map type loadbalance first-match LB_WEB_POLICY
  class class-default
    serverfarm SFARM1 backup SFARM2

policy-map multi-match L4WEB_POLICY
  class L4WEB_CLASS
    loadbalance vip inservice
    loadbalance policy LB_WEB_POLICY
    loadbalance vip icmp-reply active
    nat dynamic 1 VLAN 120

interface vlan 100
  description Upstream VLAN_100 - Clients and VIPs
  ip address
  fragment chain 20
  fragment min-mtu 68
  access-group input ACL1
  nat-pool 1 netmask pat
  service-policy input L4WEB_POLICY
  no shutdown

ip route

Troubleshooting L4 Load Balancing on the ACE

To troubleshoot L4 load-balancing issues, follow these steps:

1. Ensure that your load-balancing configuration is correct and that the following conditions exist:

  • Real servers have valid IP addresses and are in service.
  • Servers are associated with server farms of the same type.
  • A load-balancing policy exists with an associated server farm and is associated with a L4 multimatch policy.
  • An L4 class map contains a valid match virtual-address command and is associated with the L4 multimatch policy map.
  • The L4 policy is applied to the appropriate active interface using a service policy.
  • A static route is configured for the server network.

Use the following show commands:

  • show running-config rserver
  • show running-config serverfarm
  • show running-config policy-map
  • show running-config class-map
  • show running-config interface
  • show ip route

2. Check the ACE connectivity. See the "Troubleshooting Connectivity" section.

3. Verify that the L4 VIP class map is referenced in a L4 policy by entering the following command. Also, check the following fields:

  • VIP address and port
  • VIP state
  • Hit count
  • Dropped connections
ACE_module5/Admin# show service-policy L4WEB_POLICY detail

Status     : ACTIVE
Description: -
Interface: vlan 100
  service-policy: L4WEB_POLICY <------- L4 multimatch policy map
    class: L4WEB_CLASS <------- L4 VIP class map
     VIP Address:      Protocol:   Port:   tcp         eq    80 <------- VIP address, protocol, and port
        L7 loadbalance policy: LB_WEB_POLICY
        VIP Route Metric     : 77
        VIP Route Advertise  : DISABLED
        VIP ICMP Reply       : ENABLED
        VIP State: INSERVICE <------- VIP state should be INSERVICE
        curr conns       : 0         , hit count        : 56
        dropped conns    : 14 <------- Number of attempted connections to this VIP that the ACE discarded
        client pkt count : 6297      , client byte count: 1047583
        server pkt count : 1238      , server byte count: 1325495
        L7 Loadbalance policy : LB_WEB_POLICY
          class/match : class-default
            LB action :
               serverfarm: SFARM1
            hit count        : 0 <-------|-- Check these counters to see if they are increasing
            dropped conns    : 0 <-------|

The dropped conns counter under a VIP in the output of the show service policy detail command is incremented whenever the ACE discards a connection request destined to that VIP. There are several reasons why the ACE discards such connection requests. For example:

  • If all the real servers in the server farm associated with the VIP go down, then the VIP will go down. So, all the incoming connections to that VIP are discarded.
  • If the URL in a connection request to the VIP is unknown, then the connection request is discarded.
  • If the server to which the ACE load balances the connection does not respond to the request, then, after the maximum number of retries, the ACE discards the connection.

The dropped conns counter is cumulative and the value may comprise entries from any of the following show command counters:

  • show stats loadbalance
- Total Layer4 rejections
- Total Layer7 rejections
- Total Layer4 LB policy misses
- Total Layer7 LB policy misses
- Total times rserver was unavailable
  • show stats connection
- Total Connections Timed-out
- Total Connections Failed
  • The failures counter of the show serverfarm serverfarm_name command
  • The Total drop decisions counter of the show stats inspect command

4. Verify that the L4 policy is applied as a service policy to an active interface by entering the following command:

ACE_module5/Admin# show running-config interface
Generating configuration....

interface vlan 100
  ip address
  access-group input ACL1
  access-group output anyone
  service-policy input L4WEB_POLICY
  no shutdown

5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also, check the IP address, state, and the connection statistics for each real server that is configured in the server farm.

ACE_module5/Admin# show serverfarm SFARM1 detail

serverfarm     : SFARM1, type: HOST
total rservers : 3
active rservers: 3
description    : -
state          : ACTIVE <------- Current state of the server farm
predictor      : ROUNDROBIN <------- Load-balancing method
  weight            : -
  autoadjust        : MAXLOAD
failaction     : -
back-inservice    : 40
partial-threshold : 40
num times failover       : 0
num times back inservice : 0
total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded
      real                  weight state        current    total      failures
  rserver: SERVER1     10     INSERVICE    0          0          0 <------- Real server IP address, state, and connection statistics
        max-conns            : 4000000   , out-of-rotation count : 0
        min-conns            : 4000000
        conn-rate-limit      : -         , out-of-rotation count : -
        bandwidth-rate-limit : -         , out-of-rotation count : -
        retcode out-of-rotation count : -
        load value           : 0

  rserver: SERVER2     20     INSERVICE    0          0          0
        max-conns            : 4000000   , out-of-rotation count : 0
        min-conns            : 4000000
        conn-rate-limit      : -         , out-of-rotation count : -
        bandwidth-rate-limit : -         , out-of-rotation count : -
        retcode out-of-rotation count : -
        load value           : 0

  rserver: SERVER3     30     INSERVICE    0          0          0
        max-conns            : 4000000   , out-of-rotation count : 0
        min-conns            : 4000000
        conn-rate-limit      : -         , out-of-rotation count : -
        bandwidth-rate-limit : -         , out-of-rotation count : -
        retcode out-of-rotation count : -
        load value           : 0

6. Check the L4 load-balance statistics by entering the following command:

ACE_module5/Admin# show stats loadbalance

+------- Loadbalance statistics -----------+
 Total version mismatch              : 0
 Total Layer4 decisions              : 0
 Total Layer4 rejections             : 0
 Total Layer7 decisions              : 0
 Total Layer7 rejections             : 0
 Total Layer4 LB policy misses       : 0
 Total Layer7 LB policy misses       : 0
 Total times rserver was unavailable : 0
 Total ACL denied                    : 0
 Total IDMap Lookup Failures         : 0
Note Note: The ID Map is used to map real servers and server farms between the local and the remote peers in a redundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails to find the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a proper remote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the ID Map table was not created.

Rating: 4.7/5 (10 votes cast)

Personal tools