PfR:Solutions:CostPolicies

From DocWiki

Jump to: navigation, search

Pfr-banner-solutions.png



PfR Internet Presence - Implementing Cost based Policies


Navigation



Contents




Introduction

This solution guide describes how to configure and apply Cisco IOS Performance Routing (PfR) cost policies. A PfR policy can be configured to optimize traffic based on the monetary cost of the exit links. The PfR Cost Based Optimization feature provides financial benefits by directing traffic to lower cost links, while at the same time honoring other configured policies such as delay, loss, and utilization.

There are two main methods of billing:

  • Fixed-rate billing is used when the ISP bills one flat rate for a link regardless of bandwidth usage. If fixed-rate billing only is configured on the exit links, all exits are considered equal with regard to cost-optimization and other policy parameters (such as delay, loss, and utilization) are used to determine if the prefix or exit link is in-policy.
Configured under external interfaces using: cost-minimization {fixed fee cost}
  • Tier-based billing is used when the ISP bills at a tiered rate based on the percentage of exit link utilization. Each cost tier is configured separately with an associated monetary cost and a percentage of bandwidth utilization that activates the tier is defined. The lowest cost tier for an exit using tier-based billing is charged each month regardless of the bandwidth actually utilized. An allowance is made for bursting in the algorithm used to determine the tier-based billing. In this situation, bursting is defined as short periods of high bandwidth usage that would be expensive under fixed-rate billing.
Configured under external interfaces using: cost-minimization {tier percentage fee fee}

Cost Based Optimization can be applied to links that are billed using a fixed or tiered billing method. Load balancing based on cost can also be achieved.



Link Policies

Overview

PfR link policies are a set of rules that are applied against PfR-managed external links (an external link is an interface on a border router on the network edge). Link policies define the desired performance characteristics of the links. Instead of defining the performance of an individual traffic class entry that uses the link (as in traffic class performance policies), link policies are concerned with the performance of the link as a whole. Link policies are applied both to exit (egress) links and entrance (ingress) links. The following link policy types describe the different performance characteristics that can be managed using link policies.


Link Utilization Policy

A traffic load (also referred to as utilization) policy consists of an upper threshold on the amount of traffic that a specific link can carry. Cisco IOS PfR supports per traffic class load distribution. Every 20 seconds, by default, the border router reports the link utilization to the master controller, after an external interface is configured for a border router. Both exit link traffic and entrance link traffic load thresholds can be configured as a PfR policy. If the exit or entrance link utilization is above the configured threshold, or the default threshold of 75-percent, the exit or entrance link is in an out-of-policy (OOP) state and PfR starts the monitoring process to find an alternative link for the traffic class. The link utilization threshold can be manually configured either as an absolute value in kilobytes per second (kbps) or as a percentage. A load utilization policy for an individual interface is configured on the master controller under the border router configuration.

When configuring load distribution, we recommend that you set the interface load calculation on external interfaces to 30-second intervals with the load-interval interface configuration command. The default calculation interval is 300 seconds. The load calculation is configured under interface configuration mode on the border routers.


Range Policy

A range policy is defined to maintain all links within a certain utilization range, relative to each other in order to ensure that the traffic load is distributed. For example, if a network has multiple exit links, and there is no financial reason to choose one link over another, the optimal choice is to provide an even load distribution across all links. The load-sharing provided by traditional routing protocols is not always evenly distributed, because the load-sharing is flow-based rather than performance- or policy-based. Cisco PfR range functionality allows you to configure PfR to maintain the traffic utilization on a set of links within a certain percentage range of each other. If the difference between the links becomes too great, PfR will attempt to bring the link back to an in-policy state by distributing traffic classes among the available links. The master controller sets the maximum range utilization to 20 percent for all PfR-managed links by default, but the utilization range can be configured using a maximum percentage value.

Both exit link and entrance link utilization ranges can be configured as a PfR policy.


Cost Policy

PfR support for cost-based optimization was introduced in Cisco IOS Release 12.3(14)T, 12.2(33)SRB, and later releases. Cost-based optimization allows you to configure policies based on the monetary cost of each exit link in your network. To implement PfR cost-based optimization the PfR master controller is configured to send traffic over exit links that provide the most cost-effective bandwidth utilization, while still maintaining the desired performance characteristics. The load balancing algorithm is modified to allow for more efficient bandwidth utilization while minimizing the link cost.


Cost Policy Billing Models

Link Utilization Rollup Calculations

The first step in determining the billing fee for each exit link per month is to calculate the link utilization rollup values.

Link utilization rollup values are the averages of the link utilization readings taken at regular intervals (sampling period) from the ingress and egress interfaces at the border routers for a given rollup period. For example, if a sampling period was set to 60 minutes, and the rollup was set at 1440 minutes (24 hours), we would have 24 ingress and 24 egress link utilization samples used for calculating the link utilization rollup. An average is taken for each set of ingress and egress samples from that rollup period to get a link utilization rollup value for the ingress and egress links.


Monthly Sustained Utilization Calculation

After the link utilization rollup calculation is performed, the monthly sustained utilization is calculated. The specific details of tier-based billing models vary by ISP. However, most ISPs use some variation of the following algorithm to calculate what an enterprise should pay in a tiered billing plan:

  • Gather periodic measurements of egress and ingress traffic carried on the enterprise connection to the ISP network and aggregate the measurements to generate a rollup value for a rollup period.
  • Calculate one or more rollup values per billing period.
  • Rank the rollup values for the billing period into a stack from the largest value to the smallest.
  • Discard the top X percent (5% percent is the default) to accommodate bursting (Any bandwidth above the sustained monthly utilization.
  • Apply the highest remaining rollup value in the stack, referred to as the sustained Monthly Target Link Utilization (MTLU), to a tiered structure to determine a tier associated with the rollup value.
  • Charge the customer based on a set cost associated with the identified tier.


The monthly sustained utilization rollup calculations can be configured to use one of the following three techniques:

  • Combined: the monthly sustained utilization calculation is based on a combination of the egress and ingress rollup samples on a single sorted stack, the highest X rollup values are discarded, and the next highest rollup value is the MTLU.
Configured under the external interface: cost-minimization calc combined
  • Separate: the egress and ingress rollup samples for a link are sorted into separate stacks and the highest X rollup values for each stack are discarded. The highest remaining rollup value of the two stacks is selected as the MTLU.
Configured under the external interfaces: cost-minimization calc separate
  • Summed: the egress and ingress rollup samples are added together. The summed values of each rollup sample are placed into one stack, the top X rollup values are discarded, leaving the next highest rollup value as the MTLU.
Configured under the external interfaces: cost-minimization calc sum



Example:

In the following example, we use the separate technique, which means we have all samples sorted in two different columns. We also define an absolute value of 5 highest value to discard. Therefore, PfR will remove the highest 5 rollup values in each column. After the discarded values, the next highest value is 52 and this becomes the Sustained Monthly Utilization.

Egress Rollup Ingress Rollup Rollups are Sorted from Highest Bandwidth to Lowest Bandwidth in Billing Period
96 40
85 35
70 34
65 32
60 30
52 26 This value becomes the highest value after discarded the 5 highest rollup.
50 25
48 23
40 22
35 20
34 19



PfR Network Topology Used

The central site has three Border Routers, connected to three separate Service Providers using eBGP. R2, R4, R5 and R6 are iBGP peers. For an Internet Presence solution, it may be recommended to have a dedicated Master Controller given the possible high number of prefixes that have to be optimized and managed.

  • R2, R4, R5 and R6 are iBGP peers in AS 100
  • R3 is the Master Controller
  • R4, R5 and R6 are the Border Routers
  • Traffic Simulator tool is used between R1 and R11, R12 to emulates traffic
  • R1, R11 and R12 are traffic generators (to send/receive http, ssh, etc.).


Pfr-topology2.jpg



Checking Statistics and Flows

Explicitly enabling Netflow is not required for PfR to run but here we enable Flexible NetFlow to check active flows crossing the Border Routers, verify the ingress/egress interfaces used (must be internal to external or vice-versa).


Configuring Flexible Netflow

The following configuration is just an example of a flow monitor definition in order to monitor active flows based on the 5-tuples + source interface and collect IP Source and Destination Address, ports and DSCP values.

Flow Record Definition

!
flow record MYRECORD
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match interface input
 collect ipv4 dscp
 collect interface output
 collect counter bytes
 collect counter packets
!

Flow Monitor Definition

flow monitor MYMONITOR
 record MYRECORD
!

And then apply the Flexible NetFlow Monitor on the Border Routers as well as R2:

interface Ethernet0/0
 ip flow monitor MYMONITOR input
!


Checking flows on R2

Here is the output on R2 which sees all flows:


R2#sh flow monitor MYMONITOR cache format table
 R2#shflow
  Cache type:                               Normal
  Cache size:                                 4096
  Current entries:                             208
  High Watermark:                              208

  Flows added:                                 208
  Flows aged:                                    0
    - Active timeout      (  1800 secs)          0
    - Inactive timeout    (    15 secs)          0
    - Event aged                                 0
    - Watermark aged                             0
    - Emergency aged                             0

IPV4 SRC ADDR    IPV4 DST ADDR    TRNS SRC PORT  TRNS DST PORT  INTF INPUT            IP PROT  intf output                bytes        pkts  ip dscp
===============  ===============  =============  =============  ====================  =======  ====================  ==========  ==========  =======
10.10.3.1        30.30.16.11                 80           2037  Et0/1                       6  Et0/2                       7894           8  0x00   
10.10.1.1        30.30.20.11               7000           7001  Et0/1                       6  Et0/2                      10934          11  0x00   
10.10.1.1        30.30.8.11                7000           7005  Et0/1                       6  Et0/2                         80           2  0x00   
10.10.2.1        30.30.9.11                  25           1028  Et0/1                       6  Et0/2                       9394           9  0x00   
10.10.1.1        30.30.17.11               7000           7004  Et0/1                       6  Et0/2                       6394           7  0x00   
10.10.2.1        20.20.8.12                  25           1016  Et0/1                       6  Et0/2                         40           1  0x00   
10.10.3.1        30.30.18.11                 80           2038  Et0/1                       6  Et0/2                       2501           5  0x00   
10.10.4.1        20.20.20.12                 80           2045  Et0/1                       6  Et0/2                        394           3  0x00   
10.10.1.1        20.20.12.12               7000           7005  Et0/1                       6  Et0/2                       2461           4  0x00   
10.10.4.1        20.20.6.12                  80           2001  Et0/1                       6  Et0/2                       2501           5  0x00   
10.10.1.1        30.30.5.11                7000           7008  Et0/1                       6  Et0/2                       2461           4  0x00   
10.10.2.1        20.20.5.12                  25           1029  Et0/1                       6  Et0/2                       2545           6  0x00   
10.10.1.1        30.30.11.11               7000           7006  Et0/1                       6  Et0/2                       9478          11  0x00   
10.10.2.1        30.30.4.11                  25           1027  Et0/1                       6  Et0/2                         80           2  0x00   
10.10.1.1        20.20.4.12                7000           7003  Et0/1                       6  Et0/2                         80           2  0x00   
10.10.1.1        20.20.15.12               7000           7009  Et0/1                       6  Et0/2                         40           1  0x00   
10.10.4.1        20.20.19.12                 80           2048  Et0/1                       6  Et0/2                       6394           7  0x00   

[SNIP]



Display Routing Table (Central Site)

Let's have a look at the routing table before applying a PfR configuration. Central site has prefixes 10.10.0.0/16 in Autonomous System 100, remote sites have prefixes 20.20.0.0/16 in Autonomous System 200 and 30.30.0.0/16 in Autonomous System 300. For clarity, only interesting part matching the destination prefixes of the routing tables are displayed. The servers subnets (inside prefixes) are 10.10.1.0/24, 10.10.2.0/24, 10.10.3.0/24 and 10.10.4.0/24. (Remember that these prefixes must be in the BGP table in case of inbound optimization).

Note: there is a BGP policy in place to enforce the path through R6. This is to demonstrate that even if there is a BGP local preference policy in place, PfR is able to override this when needed.

  • A Local Preference of 50 is assigned on R4 for 20.20.0.0/16 and 30.30.0.0/16 routes
  • A Local Preference of 100 is assigned on R5 for 20.20.0.0/16 and 30.30.0.0/16 routes
  • A Local Preference of 200 is assigned on R6 for 20.20.0.0/16 and 30.30.0.0/16 routes


On the Border Router R4:


R4#sh bgp
BGP table version is 1476, local router ID is 10.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 * i 10.10.1.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.2.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.3.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.4.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 *>i 20.20.0.0/16     100.6.83.1               0    200      0 300 20 i <-- REMOTE AS200
 *                    100.4.81.1                     50      0 100 20 i
 *>i 30.30.0.0/16     100.6.83.1               0    200      0 300 30 i <-- REMOTE AS300
 *                    100.4.81.1                     50      0 100 30 i

[SNIP]

R4#


On the Border Router R5:


R5#sh bgp
BGP table version is 1442, local router ID is 10.5.5.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 * i 10.10.1.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.2.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.3.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.4.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 *>i 20.20.0.0/16     100.6.83.1               0    200      0 300 20 i <-- REMOTE AS200
 *                    100.5.82.1                    100      0 200 20 i
 *>i 30.30.0.0/16     100.6.83.1               0    200      0 300 30 i <-- REMOTE AS300
 *                    100.5.82.1                    100      0 200 30 i

[SNIP]

R5# 


On the Border Router R6:


R6#sh bgp
BGP table version is 1436, local router ID is 10.6.6.6
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 * i 10.10.1.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.2.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.3.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 * i 10.10.4.0/24     10.4.5.2                21    100      0 i <-- INSIDE
 * i                  10.4.5.2                21    100      0 i
 *>                   10.4.5.2                21         32768 i
 *>  20.20.0.0/16     100.6.83.1                    200      0 300 20 i <------ HERE
 *>  30.30.0.0/16     100.6.83.1                    200      0 300 30 i <------ HERE

[SNIP]

R6# 



PfR Configuration

Main configuration commands are the following:

Border Router External Interfaces:

  • cost-minimization nickname: configures a nickname for a border router interface within a cost-based optimization policy on a master controller. This is useful to quickly have the show command outputs (see later).
  • cost-minimization calc {combined | separate | sum}: choose the mode you want to use between combined, separate and summed (as explained in the previous chapter Monthly Sustained Utilization Calculation). Separate is the default here.
  • cost-minimization {fixed fee <cost>| tier <percentage> fee <fee>}: choose between a fixed cost billing cycle or a tier-based billing cycle. If you choose a tier-based model, then define the various tiers and the associated fees.
  • cost-minimization sampling period <minutes> [rollup <minutes>]: define the rollup period and the sampling interval. For this test, we have defined a very short rollup period and sampling interval which are not representatives of a real-life scenario.

Global policies:

  • resolve cost priority X: configure the cost policy
  • no resolve xx: disable other policies to avoid optimization conflicts.



!
pfr master
 max-range-utilization percent 100
 logging
 !
 border 10.4.5.6 key-chain pfr
  interface Ethernet0/1 external
   ! -----------------------------------------
   ! Nickname for R6-E0/1 is COST-ISP3
   ! Rollup period is 10 min, sampling per minute
   ! Tier-based model: 
   !   up to 80% -> fee=50 
   !   then fee=300
   ! -----------------------------------------
   cost-minimization nickname COST-ISP3
   cost-minimization tier 100 fee        300
   cost-minimization tier  80 fee         50
   cost-minimization sampling period 1 rollup 10
  interface Ethernet0/0 internal
 !
 border 10.4.5.5 key-chain pfr
  interface Ethernet0/1 external
   ! -----------------------------------------
   ! Nickname for R6-E0/1 is COST-ISP2
   ! Rollup period is 10 min, sampling per minute
   ! Tier-based model: 
   !   up to 20% -> fee=200
   !   from 20% to 60 -> fee=250
   !   then fee=300
   ! -----------------------------------------
   cost-minimization nickname COST-ISP2
   cost-minimization tier 100 fee        300
   cost-minimization tier  60 fee        250
   cost-minimization tier  20 fee        200
   cost-minimization sampling period 1 rollup 10
  interface Ethernet0/0 internal
 !        
 border 10.4.5.4 key-chain pfr
  interface Ethernet0/1 external
   ! -----------------------------------------
   ! Nickname for R6-E0/1 is COST-ISP1
   ! Rollup period is 10 min, sampling per minute
   ! Tier-based model: 
   !   up to 20% -> fee=200
   !   from 20% to 60 -> fee=250
   !   then fee=300
   ! -----------------------------------------
   cost-minimization nickname COST-ISP1
   cost-minimization tier 100 fee        300
   cost-minimization tier  60 fee        250
   cost-minimization tier  20 fee        200
   cost-minimization sampling period 1 rollup 10
  interface Ethernet0/0 internal
 !
 learn
  expire after time 300
 max prefix total 10000 learn 10000
 mode monitor passive
!---------------------
! Enable cost based policy only
!---------------------
 resolve cost priority 1
 no resolve delay
 no resolve range
 no resolve utilization
 !



Verifying PfR Cost Minimization Policies

After cost-minimization policies are configured and applied to traffic the show command below allow you to verify that the policy configuration is working as expected.

  • sh pfr master exits: will give an overall view and a summary of all metrics for all border routers and external interfaces. This command has been defined explicitly to check the link usage in case of load balancing or cost optimization policies (new command available from 15.2(2)T).
  • show pfr master cost-minimization border <ip-address> <interface>
  • show pfr master cost-minimization nickname <name>
both commands display the same cost-minimization information. Example here:
sh pfr master cost border 10.4.5.4 Ethernet 0/1 (or sh pfr master cost-minimization nickname COST-ISP1)
sh pfr master cost border 10.4.5.5 Ethernet 0/1 (or sh pfr master cost-minimization nickname COST-ISP2)
sh pfr master cost border 10.4.5.6 Ethernet 0/1 (or sh pfr master cost-minimization nickname COST-ISP3)


Overall: the first step is to check the bandwidth usage on the Border Router external interfaces. There is a new command that summarize all informations relative to external interface bandwidth utilization. You can also use the sh pfr master border detail command.


MC# sh pfr master exits 
==============================================================================================
PfR Master Controller Exits:

General Info:
=============
  E - External
  I - Internal
  N/A - Not Applicable
                                                                                           Up/
   ID Name         Border          Interface   ifIdx IP Address      Mask Policy      Type Down
  --- ------------ --------------- ----------- ----- --------------- ---- ----------- ---- ----
    6              10.4.5.6        Et0/1           2 100.6.83.6        24 Cost & Util   E  UP  
    5              10.4.5.5        Et0/1           2 100.5.82.5        24 Cost & Util   E  UP  
    4              10.4.5.4        Et0/1           2 100.4.81.4        24 Cost & Util   E  UP  

Global Exit Policy:
===================
    Range Egress:      In Policy - Max difference 43% between Exits 4 & 6 - Policy 100%
   Range Ingress:  Out of Policy - Max difference 12% between Exits 6 & 4 - Policy 0%
     Util Egress:      In Policy
    Util Ingress:      In Policy
            Cost:      In Policy

Exits Performance:
==================
                   Egress                                            Ingress
    ---------------------------------------------------- ------------------------------------
 ID Capacity  MaxUtil    Usage   %      RSVP POOL    OOP Capacity  MaxUtil    Usage   %  OOP
 --- -------- -------- -------- --- -------------- ----- -------- -------- -------- --- -----
  6     2000     1800     1164  58            N/A   Cost     2000     2000        0   0  N/A
  5     2000     1800      330  16            N/A   Cost     2000     2000        0   0  N/A
  4     2000     1800      305  15            N/A   Cost     2000     2000      241  12  N/A

TC and BW Distribution:
=======================
                       # of TCs                  BW (kbps)            Probe   Active
     Name/ID   Current Controlled InPolicy    Controlled       Total  Failed  Unreach
                                                                     (count) (fpm)
        ----   ----------------------------   ----------------------  ------  --------
           6       31         31       31         1130         1164       0          0
           5        6          6        6          317          330       0          0
           4        5          5        5          330          305       0          0

Exit Related TC Stats:
======================
                                  Priority
                             highest       nth
                            ------------------
  Number of TCs with range:        0         0
   Number of TCs with util:        0         0
   Number of TCs with cost:        1        41

       Total number of TCs:       42
MC#

Notes:

  • Global Exit Policy: we do not care about the range policy because we use the cost policy. Outgoing range we defined as 100% which explains why it is marked as In Policy.
  • Exits Performance: this part is more interesting here because it gives the bandwidth repartition between all external interfaces. No surprise here, R6 has more bandwidth because it's fee is lower (50) than the other up to 80%. Note that we only have egress bandwidth optimization defined, and R4 is the only ingress BR used (follows BGP rules).
  • TC and BW Distribution: gives the Traffic Classes repartition on all exits.



Details per ISP: the next step would be to check the details for each ISP


MC# sh pfr master cost-minimization nickname COST-ISP1
 pM - per Month, pD - per Day
-------------------------------------------------------------------------------------------
 Nickname  : COST-ISP1        Border: 10.4.5.4         Interface: Et0/1           
 Calc type : Separate
 End Date  : 0 
 Summer time: Disabled
 Fee       : Tier Based
             Tier 1: 100, fee:        300
             Tier 2:  60, fee:        250
             Tier 3:  20, fee:        200
 Period    : Sampling 1, Rollup 10
 Discard   : Type Absolute, Value 5

 Rollup Information:
 Total(pM)       Discard(pM)     Remaining(pM)   Collected(pM)   
 4320            5               1222            11              

 Current Rollup Information:
   MomentaryTgtUtil:          400 Kbps    CumRxBytes:         13243971
  StartingRollupTgt:          400 Kbps    CumTxBytes:         17672897
   CurrentRollupTgt:          400 Kbps    TimeRemain:         00:02:59

 Rollup Utilization (Kbps):
 Egress Utilization Rollups (Descending order)

 1   : 0            2   : 1108         3   : 758          4   : 686         
 5   : 319          6   : 319          7   : 318          8   : 318         
 9   : 316          10  : 0            11  : 0            12  : 0           
 Ingress Utilization Rollups (Descending order)

 1   : 0            2   : 283          3   : 269          4   : 240         
 5   : 239          6   : 239          7   : 238          8   : 237         
 9   : 209          10  : 0            11  : 0            12  : 0           
MC#

Notes:

  • Calc type : Separate (default)
  • Period: Sampling 1, Rollup 10 - 10 minutes rollup with a sampling interval of 1 minute (this is just for the test and is not a recommended value for real life deployment).
  • Discard: Type Absolute, Value 5 - the top 5 values are discarded in each column (egress, ingress).
  • Rollup Utilization: gives you the rollup values



MC# sh pfr master cost-minimization nickname COST-ISP2 
 pM - per Month, pD - per Day
-------------------------------------------------------------------------------------------
 Nickname  : COST-ISP2        Border: 10.4.5.5         Interface: Et0/1           
 Calc type : Separate
 End Date  : 0 
 Summer time: Disabled
 Fee       : Tier Based
             Tier 1: 100, fee:        300
             Tier 2:  60, fee:        250
             Tier 3:  20, fee:        200
 Period    : Sampling 1, Rollup 10
 Discard   : Type Absolute, Value 5

 Rollup Information:
 Total(pM)       Discard(pM)     Remaining(pM)   Collected(pM)   
 4320            5               1222            11              

 Current Rollup Information:
   MomentaryTgtUtil:          400 Kbps    CumRxBytes:             5579
  StartingRollupTgt:          400 Kbps    CumTxBytes:         16557729
   CurrentRollupTgt:          400 Kbps    TimeRemain:         00:02:59

 Rollup Utilization (Kbps):
 Egress Utilization Rollups (Descending order)

 1   : 0            2   : 476          3   : 358          4   : 317         
 5   : 306          6   : 302          7   : 300          8   : 300         
 9   : 105          10  : 0            11  : 0            12  : 0           
 Ingress Utilization Rollups (Descending order)

 1   : 0            2   : 0            3   : 0            4   : 0           
 5   : 0            6   : 0            7   : 0            8   : 0           
 9   : 0            10  : 0            11  : 0            12  : 0           
MC#




MC# sh pfr master cost-minimization nickname COST-ISP3
 pM - per Month, pD - per Day
-------------------------------------------------------------------------------------------
 Nickname  : COST-ISP3        Border: 10.4.5.6         Interface: Et0/1           
 Calc type : Separate
 End Date  : 0 
 Summer time: Disabled
 Fee       : Tier Based
             Tier 1: 100, fee:        300
             Tier 2:  80, fee:         50
 Period    : Sampling 1, Rollup 10
 Discard   : Type Absolute, Value 5

 Rollup Information:
 Total(pM)       Discard(pM)     Remaining(pM)   Collected(pM)   
 4320            5               1222            11              

 Current Rollup Information:
   MomentaryTgtUtil:         1600 Kbps    CumRxBytes:             5820
  StartingRollupTgt:         1600 Kbps    CumTxBytes:         59928524
   CurrentRollupTgt:         1600 Kbps    TimeRemain:         00:02:59

 Rollup Utilization (Kbps):
 Egress Utilization Rollups (Descending order)

 1   : 0            2   : 1147         3   : 1141         4   : 1105        
 5   : 1103         6   : 1087         7   : 744          8   : 738         
 9   : 592          10  : 0            11  : 0            12  : 0           
 Ingress Utilization Rollups (Descending order)

 1   : 0            2   : 0            3   : 0            4   : 0           
 5   : 0            6   : 0            7   : 0            8   : 0           
 9   : 0            10  : 0            11  : 0            12  : 0           
MC#


Verifying Enforcement

The first step is to look at the traffic classes and check that everything is in policy and what is the BR and external interface used. Only part of the output is displayed below with a focus on a few prefixes only:



MC#sh pfr master traffic-class 
OER Prefix Statistics:
 Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms),
 P - Percentage below threshold, Jit - Jitter (ms), 
 MOS - Mean Opinion Score
 Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million),
 E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable
 U - unknown, * - uncontrolled, + - control more specific, @ - active probe all
 # - Prefix monitor mode is Special, & - Blackholed Prefix
 % - Force Next-Hop, ^ - Prefix is denied

DstPrefix           Appl_ID Dscp Prot     SrcPort     DstPort SrcPrefix         
           Flags             State     Time            CurrBR  CurrI/F Protocol
         PasSDly  PasLDly   PasSUn   PasLUn  PasSLos  PasLLos      EBw      IBw
         ActSDly  ActLDly   ActSUn   ActLUn  ActSJit  ActPMOS  ActSLos  ActLLos
--------------------------------------------------------------------------------
20.20.0.0/24              N    N    N           N           N N                 
                          INPOLICY        0          10.4.5.5 Et0/1           BGP     
             111      106        0        0      337      140       52        5
               N        N        N        N        N        N

20.20.8.0/24              N    N    N           N           N N                 
                          INPOLICY        0          10.4.5.6 Et0/1           BGP     
             153      155        0        0      715      414       37        4
               N        N        N        N        N        N

20.20.16.0/24             N    N    N           N           N N                 
                          INPOLICY        0          10.4.5.4 Et0/1           BGP     
              52       52        0        0      121       77       67        7
               N        N        N        N        N        N

30.30.0.0/24              N    N    N           N           N N                 
                          INPOLICY        0          10.4.5.6 Et0/1           BGP     
             153      154        0        0      477      401       35        4
               N        N        N        N        N        N

30.30.8.0/24              N    N    N           N           N N                 
                          INPOLICY        0          10.4.5.5 Et0/1           BGP     
             104      104        0        0        0      125       52        6
               N        N        N        N        N        N

[SNIP]


Let's focus on the following prefixes:

  • 20.20.0.0/24: exit is R5
  • 20.20.8.0/24: exit is R6
  • 20.20.16.0/24: exit is R4
  • 30.30.0.0/24: exit is R6
  • 30.30.8.0/24: exit is R5



The next step is to look at the BGP routing table. By default the path for a specific prefix is enforced by setting the Local Preference to 5000 on the BR/Interface that was chosen by the PfR Master Controller.


R2#sh bgp
BGP table version is 184, local router ID is 10.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path

 *>i 20.20.0.0/24     100.5.82.1               0   5000      0 200 20 i <-- HERE: exit is through R5
 *>i 20.20.0.0/16     100.6.83.1               0    200      0 300 20 i <-- PARENT ROUTE
 *>i 20.20.1.0/24     100.6.83.1               0   5000      0 300 20 i
 *>i 20.20.2.0/24     100.6.83.1               0   5000      0 300 20 i
 *>i 20.20.3.0/24     100.6.83.1               0   5000      0 300 20 i
 *>i 20.20.4.0/24     100.4.81.1               0   5000      0 100 20 i
 *>i 20.20.5.0/24     100.6.83.1               0   5000      0 300 20 i
 *>i 20.20.6.0/24     100.6.83.1               0   5000      0 300 20 i
 *>i 20.20.7.0/24     100.4.81.1               0   5000      0 100 20 i
 *>i 20.20.8.0/24     100.6.83.1               0   5000      0 300 20 i <-- HERE: exit is through R6
[SNIP]
 *>i 20.20.16.0/24    100.4.81.1               0   5000      0 100 20 i <-- HERE: exit is through R4
 *>i 30.30.0.0/24     100.6.83.1               0   5000      0 300 30 i <-- HERE: exit is through R6
 *>i 30.30.0.0/16     100.6.83.1               0    200      0 300 30 i <-- PARENT ROUTE
 *>i 30.30.1.0/24     100.6.83.1               0   5000      0 300 30 i
 *>i 30.30.2.0/24     100.6.83.1               0   5000      0 300 30 i
 *>i 30.30.3.0/24     100.6.83.1               0   5000      0 300 30 i
 *>i 30.30.4.0/24     100.6.83.1               0   5000      0 300 30 i
 *>i 30.30.5.0/24     100.6.83.1               0   5000      0 300 30 i
 *>i 30.30.6.0/24     100.6.83.1               0   5000      0 300 30 i
 *>i 30.30.7.0/24     100.5.82.1               0   5000      0 200 30 i
 *>i 30.30.8.0/24     100.5.82.1               0   5000      0 200 30 i <-- HERE: exit is through R5
[SNIP]
 *>i 30.30.16.0/24    100.6.83.1               0   5000      0 300 30 i
 
[SNIP]
 
R2# 


Then, we can check on each BR the BGP routes that are enforced by the Master Controller:


R4#sh pfr border routes bgp
BGP table version is 178, local router ID is 10.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
OER Flags: C - Controlled, X - Excluded, E - Exact, N - Non-exact, I - Injected

   Network          Next Hop        OER    LocPrf Weight Path
*>i20.20.0.0/24     100.5.82.1      XN      5000      0 200 20 i <-- CONTROLLED BY R5
*>i20.20.1.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.2.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.3.0/24     100.6.83.1      XN      5000      0 300 20 i
*> 20.20.4.0/24     100.4.81.1      CEI       50      0 100 20 i
*>i20.20.5.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.6.0/24     100.6.83.1      XN      5000      0 300 20 i
*> 20.20.7.0/24     100.4.81.1      CEI       50      0 100 20 i
*>i20.20.8.0/24     100.6.83.1      XN      5000      0 300 20 i
[SNIP]
*> 20.20.16.0/24    100.4.81.1      CEI       50      0 100 20 i <-- LOCALLY CONTROLLED
[SNIP]
*>i30.30.0.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.1.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.2.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.3.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.4.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.5.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.6.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.7.0/24     100.5.82.1      XN      5000      0 200 30 i
*>i30.30.8.0/24     100.5.82.1      XN      5000      0 200 30 i

[SNIP]

R4#  

Notes:

  • The ‘X’ under the OER column for the 20.20.0.0/24 route on R4 means that the route is not locally controlled. Meaning that the local preference 5000 is being injected from another router (in that case R5). When the ‘X’ attribute is set, the exact vs. non-exact is meaningless.
  • The prefixe 20.20.16.0/24 is locally controlled by R4. The ‘exact’ means that the 20.20.16.0/24 route is in the BGP table and there are no more specific subnets underneath. This route is also marked as 'injected', because only the parent route 20.20.0.0/16 was in the BGP routing table before PfR kicked in.
  • Also note that in the LocPrf column, the local preference value displayed for locally controlled prefixes is the one that is manually configured in the BGP configuration (in R4 case, local preference of 50 was assigned to 20.20.0.0/16 and 30.30.0.0/16 routes).



R5#sh pfr border routes bgp
BGP table version is 156, local router ID is 10.5.5.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
OER Flags: C - Controlled, X - Excluded, E - Exact, N - Non-exact, I - Injected

   Network          Next Hop        OER    LocPrf Weight Path
*> 20.20.0.0/24     100.5.82.1      CEI      100      0 200 20 i <-- LOCALLY CONTROLLED
*>i20.20.1.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.2.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.3.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.4.0/24     100.4.81.1      XN      5000      0 100 20 i
*>i20.20.5.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.6.0/24     100.6.83.1      XN      5000      0 300 20 i
*>i20.20.8.0/24     100.6.83.1      XN      5000      0 300 20 i <-- CONTROLLED BY R6
[SNIP]
*>i30.30.0.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.1.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.2.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.3.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.4.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.5.0/24     100.6.83.1      XN      5000      0 300 30 i
*>i30.30.6.0/24     100.6.83.1      XN      5000      0 300 30 i
*> 30.30.7.0/24     100.5.82.1      CEI      100      0 200 30 i
*> 30.30.8.0/24     100.5.82.1      CEI      100      0 200 30 i

[SNIP]

R5#  


Notes:

  • The ‘X’ under the OER column for the 20.20.8.0/24 route on R5 means that the route is not locally controlled. Meaning that the local preference 5000 is being injected from another router (in that case R6). When the ‘X’ attribute is set, the exact vs. non-exact is meaningless.
  • The prefixe 20.20.0.0/24 is locally controlled by R5. The ‘exact’ means that the 20.20.0.0/24 route is in the BGP table and there are no more specific subnets underneath. This route is also marked as 'injected', because only the parent route 20.20.0.0/16 was in the BGP routing table before PfR kicked in.
  • Also note that in the LocPrf column, the local preference value displayed for locally controlled prefixes is the one that is manually configured in the BGP configuration (in R5 case, local preference of 100 was assigned to 20.20.0.0/16 and 30.30.0.0/16 routes).



R6#sh pfr border routes bgp
BGP table version is 185, local router ID is 10.6.6.6
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
OER Flags: C - Controlled, X - Excluded, E - Exact, N - Non-exact, I - Injected

   Network          Next Hop        OER    LocPrf Weight Path
*>i20.20.0.0/24     100.5.82.1      XN      5000      0 200 20 i <-- CONTROLLED BY R5
*> 20.20.1.0/24     100.6.83.1      CEI      200      0 300 20 i
*> 20.20.2.0/24     100.6.83.1      CEI      200      0 300 20 i
*> 20.20.3.0/24     100.6.83.1      CEI      200      0 300 20 i
*>i20.20.4.0/24     100.4.81.1      XN      5000      0 100 20 i
*> 20.20.5.0/24     100.6.83.1      CEI      200      0 300 20 i
*> 20.20.6.0/24     100.6.83.1      CEI      200      0 300 20 i
*>i20.20.7.0/24     100.4.81.1      XN      5000      0 100 20 i
*> 20.20.8.0/24     100.6.83.1      CEI      200      0 300 20 i <-- LOCALLY CONTROLLED
[SNIP]
*>i20.20.16.0/24    100.4.81.1      XN      5000      0 100 20 i <-- CONTROLLED BY R4
[SNIP]
*> 30.30.0.0/24     100.6.83.1      CEI      200      0 300 30 i
*> 30.30.1.0/24     100.6.83.1      CEI      200      0 300 30 i
*> 30.30.2.0/24     100.6.83.1      CEI      200      0 300 30 i
*> 30.30.3.0/24     100.6.83.1      CEI      200      0 300 30 i
*> 30.30.4.0/24     100.6.83.1      CEI      200      0 300 30 i
*> 30.30.5.0/24     100.6.83.1      CEI      200      0 300 30 i
*> 30.30.6.0/24     100.6.83.1      CEI      200      0 300 30 i
*>i30.30.7.0/24     100.5.82.1      XN      5000      0 200 30 i
*>i30.30.8.0/24     100.5.82.1      XN      5000      0 200 30 i

[SNIP]

R6# 


Notes:

  • The ‘X’ under the OER column for the 20.20.0.0/24 route on R6 means that the route is not locally controlled. Meaning that the local preference 5000 is being injected from another router (in that case R5). When the ‘X’ attribute is set, the exact vs. non-exact is meaningless.
  • The prefixe 20.20.8.0/24 is locally controlled by R6. The ‘exact’ means that the 20.20.8.0/24 route is in the BGP table and there are no more specific subnets underneath. This route is also marked as 'injected', because only the parent route 20.20.0.0/16 was in the BGP routing table before PfR kicked in.
  • Also note that in the LocPrf column, the local preference value displayed for locally controlled prefixes is the one that is manually configured in the BGP configuration (in R4 case, local preference of 200 was assigned to 20.20.0.0/16 and 30.30.0.0/16 routes).


Conclusion

Configuring and using PfR cost policies is straight forward. If you want to review more information about PfR Cost Policies, see the Configuration Guide

Rating: 5.0/5 (1 vote cast)

Personal tools