PfR:Solutions:AdvancedLoadBalancing

From DocWiki

Jump to: navigation, search

Pfr-banner-solutions.png



PfR Advanced Load Balancing using BGP
Policies based on range, link utilization and delay
Path Enforcement using PBR



Navigation



Contents





PfR Features that Enable Load Balancing

Link Utilization

Usage of this policy sets an upper threshold on the amount of traffic a specific link can carry. For example, if the upper threshold for a link is 90 % of total bandwidth, and it is currently running at 95 % of bandwidth, the link is Out-of-Policy (OOP). Cisco PfR will attempt to bring the link back into policy by repeatedly moving prefixes from the over-used link onto other exit links.

Range

Usage of this policy keeps all WAN links within a certain utilization range, relative to each other in order to ensure fair load-sharing across all concerned links. The range functionality allows the network administrator to instruct Cisco PfR to keep the usage on a set of exit links with in a certain percentage range of each other. If the difference between the links becomes too great, Cisco PfR will attempt to bring the link back in to policy by distributing data traffic among the available exit links.

Traffic Class Performance

Usage of this policy enables the customer to define multiple paths that a set of traffic (ie voice traffic) could use as long as all the paths maintain the performance SLA ’s that are needed forth at set of traffic. Hence, a policy that determines voice traffic to have a delay threshold of less than 250 msecs can utilize multiple paths in the network if available, as long as all the paths deliver the traffic within its performance bounds.



Enterprise Needs

Pfr-enterprise-challenge1.png


This solution is for an Enterprise which is dual-attached to two IP-VPN Service Providers. The load-balancing used here is taking place between external interfaces and specific policies are to be applied depending on the traffic type. In most (all) cases, Qos is already in place and classification/marking procedures were studied a while back. Therefore packets entering on the Border Router are already classified and marked directly on the access switch connecting the station, IP Phone or multimedia terminal. That means PfR can probably use the dscp field as the most efficient way for the traffic profiling phase.

  • BUSINESS: this group encompass all the critical applications. In most cases, these are transactional applications and by nature are delay intolerant. One of the key goal is to protect these applications and to be able to track variations in SLA. Traffic in this group is marked with DSCP=AF31.
  • BEST-EFFORT (BE): traffic that has a low priority but could have a high bandwidth. Traffic in this group is marked with DSCP=0.
  • VOICE: voice traffic that is delay intolerant and UDP based, which means that passive monitoring cannot be used here. Traffic in this group is marked with DSCP=EF.


PfR Solution Used

This solution describes a more advanced load sharing and will be based on automatic application classification based on DSCP, learning-list and specific policies per group of traffic.

Traffic is divided in 3 majors groups which will have specific policies applied. The main PfR features used in this configuration are the following (there are others but these ones are the most important):

Traffic Profiling:

  • Learn-list: a flexible way to define the Traffic Classes, in this case BUSINESS, BE and VOICE
  • Learning phase: automatic. Top talkers based on Netflow reports from the Border Routers
  • BUSINESS group: match the critical traffic. PfR policies are based on utilization and delay. That’s why we use resolve delay, resolve utilization and range. Also define the delay and utilization thresholds. The values defined here are just for lab purpose and are not firm recommendations.
  • BE Group: match the low priority traffic. PfR policies are based on range and link utilization and disable delay.
  • VOICE Group: match the voice traffic. PfR policies are based on delay only and disable the others.


Note: If the marking is done on the Border Router and it is also used as the classifier in PfR then it will not work. Because Netflow will see the packet before the marking and hence would not account for the BW and performance under correct TC. So marking should be done before it reaches to BR.


Measurement and optimization:

  • BUSINESS
Monitoring mode is set to both which is the default. Passive monitoring based on Netflow and active probes when choosing new exits.
Policies based on delay, then range and utilization
  • BE
Monitoring mode is set to both which is the default. Passive monitoring based on Netflow and active probes when choosing new exits.
Policies based on range then utilization
  • VOICE
Monitoring mode is set to active throughput. Active probes are generated to check the path and netflow is used to gather the bandwidth usage per Traffic Class
Policies based on delay


Enforcement:

Compared to the Basic Load Balancing Solution (see PfR:Solutions:BasicLoadBalancing) where classification is based on prefixes, the classification described here is based on DSCP, therefore the only enforcement that can be used is Policy-based Routing.
Performance Routing is Application Based
PfR will then dynamically generate dynamic route-map on the Border Routers.



PfR Network Topology Used

The central site has two Border Routers, connected to two separate Service Providers using eBGP. R2, R4 and R5 are iBGP peers.

  • R3 is the Master Controller
  • R4 and R5 the Border Routers
  • R1 and R11 are packet generators that send/receive traffic (http, ssh, etc).
  • Traffic Simulator tool is used between R1 and R11 to emulates traffic, both TCP and UDP.
  • R6 and R7 are delay generators that add delay/loss to the path through SP1 and SP2. By default, 100 ms through SP1 and 50 ms through SP2.


Pfr-topology1.png


PfR Components Configuration

pfr master
 ! ===============================
 ! For Each Traffic Class, Associate a Policy
 ! ===============================
 !
policy-rules MYMAP
!
 max-range-utilization percent 10
 logging
 !
 ! =========================
 ! Border Routers Definition
 ! =========================
 !
 border 10.4.5.4 key-chain pfr
  interface Ethernet0/0 internal
  interface Ethernet0/1 external
   max-xmit-utilization percentage 90
 !
 border 10.4.5.5 key-chain pfr
  interface Ethernet0/0 internal
  interface Ethernet0/1 external
   max-xmit-utilization percentage 90
 !
 ! ============================
 ! Learning based on learn-list
 ! ============================
 ! 
 learn
  throughput
  delay
  periodic-interval 0
  monitor-period 1
  list seq 10 refname BUSINESS
   traffic-class access-list BUSINESS
   throughput
  list seq 20 refname BE
   traffic-class access-list BE
   throughput
  list seq 30 refname VOICE
   traffic-class access-list VOICE
   throughput
!
! ============================
! For test purpose, decrease the hold timer
! ============================
!
 holddown 90
 mode route control
 periodic 180
!
!
! ============================
! ACL used by learn-list
! ============================
!
ip access-list extended BE
 permit ip any any dscp default
ip access-list extended BUSINESS
 permit ip any any dscp af31
ip access-list extended VOICE
 permit ip any any dscp ef
!
!
! ===============================
! Policies applied to Business TCs
! resolve based on delay, range, then utilization
! monitoring mode is both
! ===============================
!
pfr-map MYMAP 10
 match pfr learn list BUSINESS
 set mode select-exit good
 set delay threshold 200
 set mode route control
 set mode monitor both
 set resolve delay priority 1 variance 20
 set resolve range priority 5
 set resolve utilization priority 10 variance 20
 set probe frequency 30
!
!
! ============================
! Policies applied to BE TCs
! resolve based on range, then utilization
! monitoring mode is both
! ============================
!
pfr-map MYMAP 20
 match pfr learn list BE
 set mode select-exit good
 set mode route control
 set mode monitor both
 set resolve range priority 5
 set resolve utilization priority 10 variance 20
 no set resolve delay
 set probe frequency 30
!
!
! ============================
! Policies applied to VOICE TCs
! resolve based on delay only
! monitoring mode is active throughput
! ============================
!
pfr-map MYMAP 30
 match pfr learn list VOICE
 set mode select-exit good
 set delay threshold 200
 set mode route control
 set mode monitor active throughput
 set resolve delay priority 1 variance 20
 no set resolve range
 no set resolve utilization
 set probe frequency 30
!


Flexible Netflow

While configuring Netflow is not a mandatory task for PfR to work, it allows to have a good understanding of the traffic flows across the border routers.


Flow Record Definition

!
flow record MYRECORD
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match interface input
 collect ipv4 dscp
 collect interface output
 collect counter bytes
 collect counter packets
!

Flow Monitor Definition

flow monitor MYMONITOR
 record MYRECORD
!

And then apply the FNF Monitor on the interface

interface Ethernet0/0
 ip flow monitor MYMONITOR input
!




Checking Statistics and Flows

As explained before, explicitly enabling Netflow is not required for PfR to run but is a good practice to check active flows crossing the Border Routers, verify the ingress/egress interfaces used (must be internal to external or vice-versa).

Here is the output on R2 which sees all flows:


R2# sh flow monitor MYMONITOR cache format table
  Cache type:                               Normal
  Cache size:                                 4096
  Current entries:                             148
  High Watermark:                              169

  Flows added:                               38770
  Flows aged:                                38622
    - Active timeout      (  1800 secs)         14
    - Inactive timeout    (    15 secs)      38608
    - Event aged                                 0
    - Watermark aged                             0
    - Emergency aged                             0

IPV4 SRC ADDR    IPV4 DST ADDR    TRNS SRC PORT  TRNS DST PORT  INTF INPUT            IP PROT  intf output                bytes        pkts  ip dscp
===============  ===============  =============  =============  ====================  =======  ====================  ==========  ==========  =======
10.2.3.3         10.4.5.4                  3949          31739  Et0/0                       6  Et0/2                      79390        1029  0x00   
10.2.3.3         10.4.5.5                  3949          48115  Et0/0                       6  Et0/2                      75273         988  0x00   
10.1.1.1         30.1.1.11                 6001          51642  Et0/1                      17  Et0/2                     452352        3534  0x2E   
10.1.1.2         30.1.2.11                 6002          61192  Et0/1                      17  Et0/2                     453632        3544  0x2E   
10.1.1.3         30.1.3.11                 6003          50802  Et0/1                      17  Et0/2                     453888        3546  0x2E   
10.1.1.4         30.1.4.11                 6004          61090  Et0/1                      17  Et0/2                     454144        3548  0x2E   
10.1.1.1         30.1.2.11                   80           2012  Et0/1                       6  Et0/2                      10977          12  0x00   
10.1.1.1         30.1.0.11                 7000           7063  Et0/1                       6  Et0/2                      10977          12  0x1A   
10.1.1.1         30.1.2.11                 7000           7064  Et0/1                       6  Et0/2                      10977          12  0x1A   

[snip]




Master Controller Verification

Before starting to look at the results, a few checks are needed to verify that the configuration is correct, that the learning is correctly defined and that policies are well defined and associated with learn-list.

Goal:

  • Verify that the MC is active
  • Verify that the learning process is enabled on the Master Controller
  • Display the policy settings as well as the learning parameters and global timers.


Master Controller and Traffic Classes

The first step is to check the master controller configuration, verify the border routers, verify the parameters used (default and configured) and check the learn-list.


MC#sh pfr master
OER state: ENABLED and ACTIVE
  Conn Status: SUCCESS, PORT: 3949
  Version: 3.0
  Number of Border routers: 2
  Number of Exits: 2
  Number of monitored prefixes: 22 (max 5000)
  Max prefixes: total 5000 learn 2500
  Prefix count: total 22, learn 15, cfg 0
  PBR Requirements met
  Nbar Status: Inactive

Border           Status   UP/DOWN             AuthFail  Version
10.4.5.4         ACTIVE   UP       01:45:35          0  3.0
10.4.5.5         ACTIVE   UP       01:45:35          0  3.0

Global Settings:
  max-range-utilization percent 10 recv 0
  mode route metric bgp local-pref 5000
  mode route metric static tag 5000
  trace probe delay 1000
  logging
  exit holddown time 60 secs, time remaining 0

Default Policy Settings:
  backoff 300 3000 300
  delay relative 50
  holddown 90
  periodic 180
  probe frequency 56
  number of jitter probe packets 100
  mode route control 
  mode monitor both
  mode select-exit good
  loss relative 10
  jitter threshold 20
  mos threshold 3.60 percent 30
  unreachable relative 50
  resolve delay priority 11 variance 20
  resolve range priority 12 variance 0
  resolve utilization priority 13 variance 20

Learn Settings:
  current state : STARTED
  time remaining in current state : 67 seconds
  throughput
  delay
  no inside bgp
  monitor-period 1
  periodic-interval 0
  aggregation-type prefix-length 24
  prefixes 100 appls 100
  expire after time 720

  Learn-List seq 10 refname BUSINESS
    Configuration:
     Traffic-Class Access-list: BUSINESS
     Aggregation-type: prefix-length 24
     Learn type: throughput
     Session count: 50 Max count: 100
     Policies assigned: 10
     Status: ACTIVE
    Stats:
     Traffic-Class Count: 4
  Learn-List seq 20 refname BE
    Configuration:
     Traffic-Class Access-list: BE
     Aggregation-type: prefix-length 24
     Learn type: throughput
     Session count: 50 Max count: 100
     Policies assigned: 20
     Status: ACTIVE
    Stats:
     Traffic-Class Count: 7
  Learn-List seq 30 refname VOICE
    Configuration:
     Traffic-Class Access-list: VOICE
     Aggregation-type: prefix-length 24
     Learn type: throughput
     Session count: 50 Max count: 100
     Policies assigned: 30
     Status: ACTIVE
    Stats:
     Traffic-Class Count: 4
MC#


What to check:

  • Both Border Routers are up and running
  • Number of Monitored Prefixes: 22 and 15 are learned
  • Global Settings: max-range-utilization is 10 (percentage of bandwidth difference between all exit interfaces)
  • All default policy settings are displayed
  • Learn is started (current state : STARTED)
  • Then all learn-list are displayed
  • For each learn-list, check the Traffic Class access-list, the policy number associated (which is the pfr-map it refers to).
  • The Traffic Class count is also displayed which allows to check whether the learning process works well.


Display Learn-list

Because we use learn-list, there is a specific command to have a more detailed view:


MC#sh pfr master learn list

 Learn-List seq 10 refname BUSINESS
   Configuration:
    Traffic-Class Access-list: BUSINESS
    Aggregation-type: prefix-length 24
    Learn type: throughput
    Session count: 50 Max count: 100
    Policies assigned: 10
    Status: ACTIVE
   Stats:
    Traffic-Class Count: 4
    Traffic-Class Learned:
     Appl Prefix 30.1.3.0/24 af31 256
     Appl Prefix 30.1.1.0/24 af31 256
     Appl Prefix 30.1.2.0/24 af31 256
     Appl Prefix 30.1.0.0/24 af31 256
 Learn-List seq 20 refname BE
   Configuration:
    Traffic-Class Access-list: BE
    Aggregation-type: prefix-length 24
    Learn type: throughput
    Session count: 50 Max count: 100
    Policies assigned: 20
    Status: ACTIVE
   Stats:
    Traffic-Class Count: 7
    Traffic-Class Learned:
     Appl Prefix 20.10.11.0/24 defa 256
     Appl Prefix 30.1.0.0/24 defa 256
     Appl Prefix 30.1.2.0/24 defa 256
     Appl Prefix 30.1.3.0/24 defa 256
     Appl Prefix 30.1.1.0/24 defa 256
     Appl Prefix 30.1.5.0/24 defa 256
     Appl Prefix 30.1.4.0/24 defa 256
 Learn-List seq 30 refname VOICE
   Configuration:
    Traffic-Class Access-list: VOICE
    Aggregation-type: prefix-length 24
    Learn type: throughput
    Session count: 50 Max count: 100
    Policies assigned: 30
    Status: ACTIVE
   Stats:
    Traffic-Class Count: 4
    Traffic-Class Learned:
     Appl Prefix 30.1.3.0/24 ef   256
     Appl Prefix 30.1.4.0/24 ef   256
     Appl Prefix 30.1.2.0/24 ef   256
     Appl Prefix 30.1.1.0/24 ef   256
MC# 

For each group, this command displays the prefixes that are dynamically learned.


Policy Configuration

After the Master Controller and Traffic Classes verification, the third step is to check the policies associated with the Traffic Classes.


MC#sh pfr master policy 
Default Policy Settings:
  backoff 300 3000 300
  delay relative 50
  holddown 90
  periodic 180
  probe frequency 56
  number of jitter probe packets 100
  mode route control 
  mode monitor both
  mode select-exit good
  loss relative 10
  jitter threshold 20
  mos threshold 3.60 percent 30
  unreachable relative 50
  resolve delay priority 11 variance 20
  resolve range priority 12 variance 0
  resolve utilization priority 13 variance 20
oer-map MYMAP 10
  sequence no. 8444249301975040, provider id 1, provider priority 30
    host priority 0, policy priority 10, Session id 0
  match oer learn list BUSINESS
  backoff 300 3000 300
 *delay threshold 200
  holddown 90
  periodic 180
 *probe frequency 30
  number of jitter probe packets 100
 *mode route control 
 *mode monitor both
 *mode select-exit good
  loss relative 10
  jitter threshold 20
  mos threshold 3.60 percent 30
  unreachable relative 50
  next-hop not set
  forwarding interface not set
 *resolve delay priority 1 variance 20
 *resolve range priority 5 variance 0
 *resolve utilization priority 10 variance 20
oer-map MYMAP 20
  sequence no. 8444249302630400, provider id 1, provider priority 30
    host priority 0, policy priority 20, Session id 0
  match oer learn list BE
  backoff 300 3000 300
  delay relative 50
  holddown 90
  periodic 180
 *probe frequency 30
  number of jitter probe packets 100
 *mode route control 
 *mode monitor both
 *mode select-exit good
  loss relative 10
  jitter threshold 20
  mos threshold 3.60 percent 30
  unreachable relative 50
  next-hop not set
  forwarding interface not set
 *resolve range priority 5 variance 0
 *resolve utilization priority 10 variance 20
oer-map MYMAP 30
  sequence no. 8444249303285760, provider id 1, provider priority 30
    host priority 0, policy priority 30, Session id 0
  match oer learn list VOICE
  backoff 300 3000 300
 *delay threshold 200
  holddown 90
  periodic 180
 *probe frequency 30
  number of jitter probe packets 100
 *mode route control 
 *mode monitor active throughput
 *mode select-exit good
  loss relative 10
  jitter threshold 20
  mos threshold 3.60 percent 30
  unreachable relative 50
  next-hop not set
  forwarding interface not set
 *resolve delay priority 1 variance 20

* Overrides Default Policy Setting
MC# 

Verify Traffic Classes Statistics

As soon as you see:


MC#
*Sep  3 16:01:22.924: %OER_MC-5-NOTICE: Prefix Learning WRITING DATA
*Sep  3 16:01:22.966: %OER_MC-5-NOTICE: Prefix Learning STARTED
MC#

You should be able to see the traffic classes on the Master Controller. You will need a few cycles before having all prefixes in INPOLICY state. You will start getting OOP (out of policy messages) from PfR regarding range, because R4 (10.4.5.4) is currently the only exit point while R5 (10.4.5.5) has no traffic:


MC#
*Sep  3 16:01:39.232: %OER_MC-5-NOTICE: Range OOP BR 10.4.5.4, i/f Et0/1, percent 76. Other BR 10.4.5.5, i/f Et0/1, percent 0
*Sep  3 16:01:39.232: %OER_MC-5-NOTICE: Exit 10.4.5.4 intf Et0/1 OOP, Tx BW 383, Rx BW 54, Tx Load 76, Rx Load 10
MC#


Traffic Classes

On the Master Controller, you have all the Traffic Classes (in this case based on DSCP values) learnt as well as statistics:

Using show pfr master traffic-class:


MC#sh pfr master traffic-class 
OER Prefix Statistics:
 Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
 P - Percentage below threshold, Jit - Jitter (ms), 
 MOS - Mean Opinion Score
 Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
 E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
 U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
 # - Prefix monitor mode is Special, & - Blackholed Prefix
 % - Force Next-Hop, ^ - Prefix is denied

DstPrefix           Appl_ID Dscp Prot     SrcPort     DstPort SrcPrefix         
           Flags             State     Time            CurrBR  CurrI/F Protocol
         PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos      EBw      IBw
         ActSDly  ActLDly   ActSUn   ActLUn  ActSJit  ActPMOS  ActSLos  ActLLos
--------------------------------------------------------------------------------

330.1.0.0/24               N af31  256           N           N 0.0.0.0/0         
                          INPOLICY       76          10.4.5.5 Et0/1           PBR     
              52       52        0        0        0        0       65        7
              52       51        0        0        N        N        N        N

30.1.1.0/24               N   ef  256           N           N 0.0.0.0/0         
               #         INPOLICY*      @39          10.4.5.4 Et0/1           U       
               U        U        0        0        0        0        9        9
              56       52        0        0        N        N        N        N

30.1.1.0/24               N defa  256           N           N 0.0.0.0/0         
                          INPOLICY      @33          10.4.5.4 Et0/1           PBR     
              53       53        0        0        0        0       31        3
               U       54        0        0        N        N        N        N

30.1.1.0/24               N af31  256           N           N 0.0.0.0/0         
                          INPOLICY      @47          10.4.5.4 Et0/1           PBR     
              53       53        0        0        0        0       35        4
               U       53        0        0        N        N        N        N


[snip]

MC# 


A closer look at the results

Let’s have a look at a few entries.


From the command “show oer master traffic-class. Let’s focus on a specific TC which belongs to BUSINESS group to understand the interesting values reported here:


MC#sh pfr master traffic-class 
OER Prefix Statistics:
 Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
 P - Percentage below threshold, Jit - Jitter (ms), 
 MOS - Mean Opinion Score
 Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
 E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
 U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
 # - Prefix monitor mode is Special, & - Blackholed Prefix
 % - Force Next-Hop, ^ - Prefix is denied

DstPrefix           Appl_ID Dscp Prot     SrcPort     DstPort SrcPrefix         
           Flags             State     Time            CurrBR  CurrI/F Protocol
         PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos      EBw      IBw
         ActSDly  ActLDly   ActSUn   ActLUn  ActSJit  ActPMOS  ActSLos  ActLLos
--------------------------------------------------------------------------------

30.1.0.0/24               N af31  256           N           N 0.0.0.0/0         
                          INPOLICY       76          10.4.5.5 Et0/1           PBR     
              52       52        0        0        0        0       65        7
              52       51        0        0        N        N        N        N


  • Line1: prefix
  • Line2: State of INPOLICY—this prefix is controlled by PfR and is currently inpolicy. PBR is used to enforce the path. Forwarding decisions based on other packet fields (such as TCP port numbers, DSCP field) cannot be done via the traditional routing table. For this reason, PfR will create a dynamic Policy Based Routing (PBR) policy and apply it to the PfR internal interfaces of the border routers.
  • Line2: R5 is the exit point (SP2)


  • Line3: Passive results
  • Line3 (PasSDly, PasLDly): short-term and long-term passive delay, measured from the TCP Syn/Ack. As explained previously, TCP Syn/Ack are used to check the reachability of the prefix and to collect the delay and loss information. The delay is around 100 ms, which is the delay through SP2 (BR R5).
  • Line3 (PasSUn, PasLUn): short-term and long-term statistics for Unreachable.
  • Line3 (PasSLos, PasLLos): short-term and long-term statistics for Loss.
  • Line3 (EBw/IBw): the bandwidth for this prefix in egress and ingress direction.


  • Line4: Active results
  • Line4 (ActSDly, ActLDly): short-term and long-term active delay, measured by active probes that are automatically learned.
  • Line4 (ActSUn, ActLUn): short-term and long-term Unreachable results.
  • Line4 (ActSLos, ActLLoss): short-term and long-term Loss results.



From the command “show oer master traffic-class. Let’s focus on a specific TC which belongs to VOICE group to understand the interesting values reported here. Remember that the monitor mode used for this group is "active throughput".


MC#sh pfr master traffic-class 
OER Prefix Statistics:
 Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
 P - Percentage below threshold, Jit - Jitter (ms), 
 MOS - Mean Opinion Score
 Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
 E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
 U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
 # - Prefix monitor mode is Special, & - Blackholed Prefix
 % - Force Next-Hop, ^ - Prefix is denied

DstPrefix           Appl_ID Dscp Prot     SrcPort     DstPort SrcPrefix         
           Flags             State     Time            CurrBR  CurrI/F Protocol
         PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos      EBw      IBw
         ActSDly  ActLDly   ActSUn   ActLUn  ActSJit  ActPMOS  ActSLos  ActLLos
--------------------------------------------------------------------------------

30.1.1.0/24               N   ef  256           N           N 0.0.0.0/0         
               #         INPOLICY*      @39          10.4.5.4 Et0/1           U       
               U        U        0        0        0        0        9        9
              56       52        0        0        N        N        N        N


  • Line1: prefix
  • Line2: State of INPOLICY*—The asteric (*) indicates this prefix is uncontrolled by PfR (the parent route controls routing) , but is currently inpolicy. The '#' indicates that mode active. The ‘at sign’ (@) on the Time Remaining value means the prefix is being actively probed. The numerical value is a countdown timer indicating when this state will expire. throughput is used here.
  • Line2: R4 is the exit point (SP1)
  • Line2: Not Applicable, this TC is uncontrolled and therefore follows the routing table for this prefix.


  • Line3: Passive results
  • Line3 (PasSDly, PasLDly): active mode used, so Not Applicable here.
  • Line3 (PasSUn, PasLUn): active mode used, so Not Applicable here..
  • Line3 (PasSLos, PasLLos): active mode used, so Not Applicable here.
  • Line3 (EBw/IBw): mode active throughput is used, therefore the bandwidth for this prefix is reported in egress and ingress direction.


  • Line4: Active results
  • Line4 (ActSDly, ActLDly): short-term and long-term active delay, measured by active probes that are automatically learned here.
  • Line4 (ActSUn, ActLUn): short-term and long-term Unreachable results.
  • Line4 (ActSLos, ActLLoss): short-term and long-term Loss results.


Active Probes

In this configuration active probes are automatically calculated and generated, but active probes can be manually defined per TC in the pfr-map section.


MC#sh pfr master active-probes 
        OER Master Controller active-probes
Border   = Border Router running this Probe
State    = Un/Assigned to a Prefix
Prefix   = Probe is assigned to this Prefix
Type     = Probe Type
Target   = Target Address
TPort    = Target Port
How      = Was the probe Learned or Configured
N - Not applicable

The following Probes exist:

State      Prefix             Type     Target          TPort   How     Codec
Assigned   20.10.11.0/24      echo     20.10.11.11         N  Lrnd         N
Assigned   30.1.2.0/24        echo     30.1.2.11           N  Lrnd         N
Assigned   30.1.3.0/24        echo     30.1.3.11           N  Lrnd         N
Assigned   30.1.1.0/24        echo     30.1.1.11           N  Lrnd         N
Assigned   30.1.5.0/24        echo     30.1.5.11           N  Lrnd         N
Assigned   30.1.4.0/24        echo     30.1.4.11           N  Lrnd         N
Assigned   30.1.0.0/24        echo     30.1.0.11           N  Lrnd         N

The following Probes are running:

Border          State    Prefix             Type     Target          TPort


MC#



Verify Enforcement

Policy-Based Routing used

Forwarding decisions based on other packet fields (in this case DSCP field) cannot be done via the traditional routing table. For this reason, PfR will create a dynamic Policy Based Routing (PBR) policy and apply it to the PfR internal interfaces of the border routers. But for PBR to be successful, BRs have to be adjacent either through a direct connection or via a GRE tunnel.

This requirement can be verified with the "show pfr master" command and looking for: PBR Requirements met


MC#sh pfr master
OER state: ENABLED and ACTIVE
  Conn Status: SUCCESS, PORT: 3949
  Version: 3.0
  Number of Border routers: 2
  Number of Exits: 2
  Number of monitored prefixes: 22 (max 5000)
  Max prefixes: total 5000 learn 2500
  Prefix count: total 22, learn 15, cfg 0
  PBR Requirements met                                     <---- Check this
  Nbar Status: Inactive

Border           Status   UP/DOWN             AuthFail  Version
10.4.5.4         ACTIVE   UP       03:09:01          0  3.0
10.4.5.5         ACTIVE   UP       03:09:02          0  3.0


BGP Routes by default

PfR being not active, the parent routes are BGP based on R2, R4 and R5. R4 is the preferred exit point as seen below:


R2#sh ip bgp
BGP table version is 27, local router ID is 10.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r i10.4.5.0/24      10.5.5.5                 0    100      0 i
r>i                 10.4.4.4                 0    100      0 i
*>i20.10.11.0/24    100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i20.11.11.11/32   100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i30.1.0.0/24      100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i30.1.1.0/24      100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i30.1.2.0/24      100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i30.1.3.0/24      100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i30.1.4.0/24      100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i30.1.5.0/24      100.4.8.8                0    100      0 100 20 i
   Network          Next Hop            Metric LocPrf Weight Path
* i                 100.5.9.9                0    100      0 200 20 i
r>i100.4.8.0/24     10.4.4.4                 0    100      0 i
r>i100.5.9.0/24     10.5.5.5                 0    100      0 i
*>i100.8.10.0/24    100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
*>i100.9.10.0/24    100.4.8.8                0    100      0 100 20 i
* i                 100.5.9.9                0    100      0 200 20 i
R2#



Dynamic Route Maps generated

PfR creates several dynamic route-maps to enforce the path for BUSINESS, BE and VOICE application TCs. Depending on two dynamic ACLs, PBR enforce a next-hop to SP1 or SP2.

Let's check on the first border router R4:


R4#sh route-map dynamic 
route-map OER_INTERNAL_RMAP, permit, sequence 0, identifier 3640655874
  Match clauses:
    ip address (access-lists): oer#2 
  Set clauses:
    ip next-hop 100.4.8.8
    interface Ethernet0/1
  Policy routing matches: 398009 packets, 232904449 bytes
route-map OER_INTERNAL_RMAP, permit, sequence 1, identifier 4060086275
  Match clauses:
    ip address (access-lists): oer#3 
  Set clauses:
    ip next-hop 10.4.5.5
    interface Ethernet0/0
  Policy routing matches: 421999 packets, 250233694 bytes
Current active dynamic routemaps = 1
R4#

This route-map is applied on the ingress interface of the Border Router.


R4#sh ip policy
Interface      Route map
Ethernet0/0    OER_INTERNAL_RMAP (Dynamic)
R4#

The first route-map sets the next-hop to R8, which means traffic matching the ACL will be sent to SP1. The second route-map sets the next-hop to R5, which means that a packet received on R4 will be forwarded to the other BR for traffic that match the corresponding ACL.

The corresponding ACL are the following:


R4#sh ip access-lists dynamic 
Extended IP access list oer#2
    2047 permit ip any 30.1.1.0 0.0.0.255 dscp default (14190 matches)
    9215 permit ip any 30.1.3.0 0.0.0.255 dscp default (6803 matches)
    32767 permit ip any 30.1.1.0 0.0.0.255 dscp af31 (36545 matches)
    65535 permit ip any 30.1.5.0 0.0.0.255 dscp default (74573 matches)
Extended IP access list oer#3
    2047 permit ip any 30.1.3.0 0.0.0.255 dscp af31 (2322 matches)
    4095 permit ip any 30.1.2.0 0.0.0.255 dscp af31 (6666 matches)
    8191 permit ip any 30.1.2.0 0.0.0.255 dscp default (8352 matches)
    262143 permit ip any 30.1.4.0 0.0.0.255 dscp default (73840 matches)
    1048575 permit ip any 30.1.0.0 0.0.0.255 dscp af31 (103854 matches)
R4#

And it's easy to understand and troubleshoot if needed.

A similar behavior can be observed on the second Border Router R5.



Add Delay on SP1

Normal delay across SP1

BUSINESS and VOICE groups have a delay resolver configured with an absolute threshold of 150 ms while BE group has only range and utilization. Increasing the delay on SP1 would then only impact business and voice traffic.

In this section, we’ll take the SP1 path and add delay to it and observe what PfR does.

  • For TCP sessions, PfR is able to detect the delay along the path using passive monitoring only. Because the monitoring mode is both, only the passive delay can trigger an OOP event.
  • For UDP session, active mode is used and PfR generates probes. Active delay will trigger an OOP message.


PfR maintain delay statistics for the short-term delay (5 min) and the long-term delay (60 min).

Remember that there are two ways to specify what is flagged as an OOP condition in PfR:

  • Relative: Relative implies the use of short/long term statistics. You specify a percentage and that is used to gauge if a TC is OOP.
  • Absolute: You now have a threshold that is compared only to the short-term delay.


Before adding delay, let’s verify the RTT from R4 across the SP1 link:


R4#ping 100.4.8.8

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.4.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 51/53/54 ms
R4#


Adding a delay of 500ms on SP1

Now, we are adding 500 ms delay on SP1. Again verify on R4:


R4#ping 100.4.8.8

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.4.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 502/504/508 ms
R4#


The delay threshold being defined in an absolute mode, is therefore compared to the short-term delay. By issuing "sh pfr master traffic-class", you can notice that the short-term delay (PasSDly, ActSDly) is increasing for traffic going over the SP1 path.


MC#sh pfr master traffic-class 
OER Prefix Statistics:
 Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
 P - Percentage below threshold, Jit - Jitter (ms), 
 MOS - Mean Opinion Score
 Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
 E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
 U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
 # - Prefix monitor mode is Special, & - Blackholed Prefix
 % - Force Next-Hop, ^ - Prefix is denied

DstPrefix           Appl_ID Dscp Prot     SrcPort     DstPort SrcPrefix         
           Flags             State     Time            CurrBR  CurrI/F Protocol
         PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos      EBw      IBw
         ActSDly  ActLDly   ActSUn   ActLUn  ActSJit  ActPMOS  ActSLos  ActLLos
--------------------------------------------------------------------------------

30.1.0.0/24               N af31  256           N           N 0.0.0.0/0         
                          INPOLICY      107          10.4.5.5 Et0/1           PBR     
              52       52        0        0        0        0       66        7
              48       51        0        0        N        N        N        N

30.1.1.0/24               N defa  256           N           N 0.0.0.0/0         
                          INPOLICY       53          10.4.5.4 Et0/1           PBR     
             169       70        0        0        0        0       18        1
             507      143        0        0        N        N        N        N

30.1.1.0/24               N af31  256           N           N 0.0.0.0/0         
                          INPOLICY      @24          10.4.5.4 Et0/1           PBR     
              79       55        0        0        0        0       30        3
             503       74        0        0        N        N        N        N

30.1.3.0/24               N   ef  256           N           N 0.0.0.0/0         
               #         INPOLICY*      @43          10.4.5.4 Et0/1           U       
               U        U        0        0        0        0        9        9
             233       66        0        0        N        N        N        N



Move TC to new exit – HOLDDOWN State

After a few cycles we can notice that the Short Delay is now above 200 ms. The absolute delay threshold is compared against the short-term delay value and therefore trigger an OOP message for traffic classes belonging BUSINESS and VOICE groups going over the SP1 link:


*Oct 22 14:40:34.907: %OER_MC-5-NOTICE: Active ABS Delay OOP Appl Prefix 30.1.4.0/24 ef   256, delay 236, BR 10.4.5.4, i/f Et0/1
*Oct 22 14:40:35.865: %OER_MC-5-NOTICE: Active ABS Delay OOP Appl Prefix 30.1.3.0/24 ef   256, delay 233, BR 10.4.5.4, i/f Et0/1

*Oct 22 14:42:47.739: %OER_MC-5-NOTICE: Passive ABS Delay OOP Appl Prefix 30.1.2.0/24 af31 256, delay 504, BR 10.4.5.4, i/f Et0/1


When OOP condition is detected, PfR tries to find an alternate path. When PfR makes a route change to a specific TC, this TC will move to the HOLDDOWN state to avoid erratic behavior. The timer associated is the holddown timer defined in the pfr master configuration. If the current path is the best path but it is OOPOLICY then it will go to OOPOLICY state. A syslog message indicates that a new exit is found. The primary path being OOP, PfR moves BUSINESS and VOICE Traffic Classes to the SP2 path.


*Oct 22 14:41:07.919: %OER_MC-5-NOTICE: Route changed Appl Prefix 30.1.2.0/24 ef   256, BR 10.4.5.5, i/f Et0/1, Reason Delay, OOP Reason Delay
*Oct 22 14:41:32.356: %OER_MC-5-NOTICE: Route changed Appl Prefix 30.1.4.0/24 ef   256, BR 10.4.5.5, i/f Et0/1, Reason Delay, OOP Reason Delay
*Oct 22 14:41:33.305: %OER_MC-5-NOTICE: Route changed Appl Prefix 30.1.3.0/24 ef   256, BR 10.4.5.5, i/f Et0/1, Reason Delay, OOP Reason Delay

*Oct 22 14:43:45.040: %OER_MC-5-NOTICE: Route changed Appl Prefix 30.1.2.0/24 af31 256, BR 10.4.5.5, i/f Et0/1, Reason Delay, OOP Reason Delay


As seen below, BUSINESS and VOICE traffic classes are now all on SP2 path while BE Traffic Classes stay INPOLICY and therefore stay on the SP1 path.


MC#sh pfr master traffic-class 
OER Prefix Statistics:
 Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
 P - Percentage below threshold, Jit - Jitter (ms), 
 MOS - Mean Opinion Score
 Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
 E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
 U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
 # - Prefix monitor mode is Special, & - Blackholed Prefix
 % - Force Next-Hop, ^ - Prefix is denied

DstPrefix           Appl_ID Dscp Prot     SrcPort     DstPort SrcPrefix         
           Flags             State     Time            CurrBR  CurrI/F Protocol
         PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos      EBw      IBw
         ActSDly  ActLDly   ActSUn   ActLUn  ActSJit  ActPMOS  ActSLos  ActLLos
--------------------------------------------------------------------------------

30.1.0.0/24               N af31  256           N           N 0.0.0.0/0         
                          INPOLICY       18          10.4.5.5 Et0/1           PBR     
              52       52        0        0        0        0       66        7
              52       51        0        0        N        N        N        N

30.1.1.0/24               N   ef  256           N           N 0.0.0.0/0         
               #          HOLDDOWN       82          10.4.5.5 Et0/1           PBR     
               U        U        0        0        0        0        9        9
               U        U        0        0        N        N        N        N

30.1.1.0/24               N defa  256           N           N 0.0.0.0/0         
                          INPOLICY      @14          10.4.5.4 Et0/1           PBR     
              61       54        0        0        0        0       24        3
             510      102        0        0        N        N        N        N

30.1.2.0/24               N   ef  256           N           N 0.0.0.0/0         
               #          HOLDDOWN       75          10.4.5.5 Et0/1           PBR     
               U        U        0        0        0        0        9        9
               U        U        0        0        N        N        N        N
          


Conclusion

Performance Routing is an advanced technology that allows multiple deployments, one of them being the one described in this document. The configuration is straightforward and very efficient. Based on that, a customer can evolve to a more complex Performance Routing solution or can fine-tune the Traffic Class definition.

Rating: 5.0/5 (6 votes cast)

Personal tools