IWAN Solution - Advanced topology

From DocWiki

Jump to: navigation, search

Banner2.png



Contents



IWAN Overview

The Cisco Intelligent WAN (IWAN) solution provides design and implementation guidance for organizations looking to deploy a transport independent WAN with intelligent path control, application optimization, and secure connectivity to the Internet and branch locations while reducing the operating cost of the WAN. IWAN takes full advantage of premium WAN and cost-effective Internet services to increase bandwidth capacity without compromising performance, reliability, or security of collaboration or cloud-based applications.

For more information, refer to IWAN Overview and read about IWAN at www.cisco.com/go/iwan or contact your local Cisco account representative.


DMVPN Multiple Tunnel Termination (MTT)

The DMVPN MTT provides additional network access resiliency at a single hub router in IWAN without having to add network devices just for this. This involves terminating multiple WAN links on the same router, which, in the IWAN design, boils down to support for multiple tunnel terminations (interfaces) in the same VRF on the same hub router. This also becomes important for small deployments (and PoCs) where the intention may be to terminate two (or more) provider connections on the same device.

MTT.png

Enterprise Deployment example which uses 3 DCs and MTT

Overview

An enterprise is beginning their WAN network redesign and is very interested in augmenting the bandwidth with an Internet based link. Their primary path is MPLS and they want to use this preferred path for all critical applications and fallback to the secondary/third link when there is a performance issue.

IWAN uses a prescriptive design with an Hybrid Transport Independent design based on DMVPN. DMVPN supports Multiple Tunnel termination at the DC which helps to terminate multiple and different type of WAN links at one DC - Border router. This greatly simplifies the routing by using a single routing domain that encompasses all the transports. The DMVPN routers use tunnel interfaces that support IP unicast as well as IP multicast and broadcast traffic, including the use of dynamic routing protocols. After the initial spoke-to-hub tunnel is active, it is possible to create dynamic spoke-to-spoke tunnels when site-to-site IP traffic flows require it.

The Transport Independent Design is based on one DMVPN cloud per provider. In this guide we are gonna use 5 providers for transport, which includes 2 MPLS links, 2 Internet and one 4G link. Branch sites are connected to all the 3 DCs leveraging the above transport options.


Transport and Overlay Backbones

The Transport Topology is based on five Service Providers two MPLS and Internet links each & a 4G link. A DMVPN overlay is built on top of each provider WAN. Each Hub Border Router also supports multiple tunnel termination. MPLS being considered as the primary one with known SLAs. Therefore MPLS is the preferred path for voice/video and critical applications:

The below topology uses a DCI Switch for Inter-DC POPs communication.

3DC-Topo1.png


This overlay design with five DMVPN clouds can accommodate any kind of transports. The primary path can connect to an MPLS-VPN or even to the Public Internet. The configuration of PfR (and QoS) will remain the same even if the transport design changes.


Notes:

  • Multiple transports supported on a hub/transit BR (XE 16.3.3 onwards)
  • Multiple transports supported on a single branch BR
  • Dual CPE branch: BRs have to be directly connected. R51 and R52 are directly connected in the topology used in this guide.



IWAN Topology Design details

DC Transport:

  • MPLS Transport1 (MPLS1): 10.4.81.x/30
  • MPLS Transport2 (MPLS2): 10.4.81.x/30
  • Internet Transport1 (INET1): 172.16.1.x/30
  • Internet Transport2 (INET2): 1172.16.1.x/30
  • Internet Transport - 4G (INET4G): 172.26.1.x/30


DC Overlay - Transport Independent Design (DMVPN)

  • DMVPN Overlay for MPLS1: 10.13.1.x/16
  • DMVPN Overlay for MPLS2: 10.23.1.x/16
  • DMVPN Overlay for INET1: 10.33.1.x/16
  • DMVPN Overlay for INET2: 10.43.1.x/16
  • DMVPN Overlay for INET4G: 10.60.1.x/16


DC Site1: Datacenter1 (DC-POP1) (10.1.0.0/16):

  • PfR: A dedicated Master Controller (MC) R10 and two Border Routers (BRs) R11 (connected to MPLS1 and INET1) and R12 (connected to MPLS2, INET2, INET4G)
  • Routing: R11 and R12 advertise 10.1.0.0/16, the enterprise summary 10.0.0.0/8 as well as a default route.


DC Site2: Datacenter2 (DC-POP1) (10.2.0.0/16):

  • PfR: A dedicated Master Controller (MC) R20 and two Border Routers (BRs) R21 (connected to MPLS1 and INET1) and R22 (connected to MPLS2, INET2, INET4G)
  • Routing: R21 and R22 advertise 10.2.0.0/16, the enterprise summary 10.0.0.0/8 as well as a default route.


DC Site3: Datacenter3 (DC-POP3) (10.2.0.0/16):

  • PfR: A dedicated Master Controller (MC) R30 and two Border Routers (BRs) R31 (connected to MPLS1 and INET1) and R32 (connected to MPLS2, INET2, INET4G)
  • Routing: R21 and R22 advertise 10.2.0.0/16, the enterprise summary 10.0.0.0/8 as well as a default route.


Branch Sites - Transport:

  • MPLS Transport1 (MPLS1): 10.4.81.x/30
  • MPLS Transport2 (MPLS2): 10.4.99.x/30
  • Internet Transport1 (INET1): 172.19.11.x/30
  • Internet Transport2 (INET2): 172.20.10.x/30
  • Internet Transport - 4G (INET4G): 172.30.100.x/30


Branch Sites - DMVPN Overlay:

  • DMVPN Overlay for MPLS1: 10.13.1.x/16
  • DMVPN Overlay for MPLS2: 10.23.1.x/16
  • DMVPN Overlay for INET1: 10.33.1.x/16
  • DMVPN Overlay for INET2: 10.43.1.x/16
  • DMVPN Overlay for INET4G: 10.60.1.x/16


Branch Sites - Routers involved:

  • Branch Site1: Single CPE branch. R41(MC/BR), connected to MPLS1, INET1 and INET4G
  • Branch Site2: Dual CPE branch. R51 (MC/BR) connected to MPLS1stiINET1 and INET4G and R52 (BR) connected to MPLS1, INET1 and INET4G
  • Branch Site3: Single CPE branch. R61 (MC/BR), 10.3.3.0/24
  • Branch Site4: Single CPE branch. R771 (MC/BR), 10.3.3.0/24



Configuring DMVPN with MTT enabled at DC Border routers

First, the DMVPN hub requires a connection to the MPLS/Internet, and the DMVPN hub is usually connected through a Firewall using a DMZ interface specifically created and configured for a VPN termination router. This is not represented here.

The Front Door VRF implementation requires the following steps:

  • Creating the VRF
  • Assigning the external interface to the FVRF
  • Defining a default route in the FVRF to allow the creation of the DMVPN tunnel


Front Door VRF Configuration on DC-POP1-BR1 (R11):

!
vrf definition IWAN-TRANSPORT-1
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-3
 !
 address-family ipv4
 exit-address-family
!
interface TenGigabitEthernet0/0/0
 description ** Connected to WAN1 MPLS1 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-1
 ip address 10.4.81.2 255.255.255.252
!
interface TenGigabitEthernet0/1/0
 description ** Connected to WAN3 INET1 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-3
 ip address 172.16.1.2 255.255.255.252
ip route vrf MPLS01 0.0.0.0 0.0.0.0 172.16.11.2
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 10.4.81.1
ip route vrf IWAN-TRANSPORT-3 0.0.0.0 0.0.0.0 172.16.1.1
!


Front Door VRF Configuration on DC-POP1-BR2 (R12):

!
vrf definition IWAN-TRANSPORT-2
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-4
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-5
 !
 address-family ipv4
 exit-address-family
!
interface TenGigabitEthernet0/1/0
 description ** Connected to WAN2 MPLS2 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-2
 ip address 10.4.81.14 255.255.255.252
!
interface TenGigabitEthernet0/0/0
 description ** Connected to WAN4 INET2 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-4
 ip address 172.16.1.18 255.255.255.252
!

interface GigabitEthernet0/0/0
 description ** Connected to WAN5 INET4G **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-5
 ip address 172.26.1.10 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-2 0.0.0.0 0.0.0.0 10.4.81.13
ip route vrf IWAN-TRANSPORT-4 0.0.0.0 0.0.0.0 172.16.1.17
ip route vrf IWAN-TRANSPORT-5 0.0.0.0 0.0.0.0 172.26.1.9
!


Front Door VRF Configuration on DC-POP2-BR1 (R21):

!
vrf definition IWAN-TRANSPORT-1
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-3
 !
 address-family ipv4
 exit-address-family
!
interface TenGigabitEthernet0/2/0
 description ** Connected to WAN1 MPLS1 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-1
 ip address 10.4.81.6 255.255.255.252
!
interface TenGigabitEthernet0/3/0
 description ** Connected to WAN3 INET1 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-3
 ip address 172.16.1.6 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 10.4.81.5
ip route vrf IWAN-TRANSPORT-3 0.0.0.0 0.0.0.0 172.16.1.5
!


Front Door VRF Configuration on DC-POP2-BR2 (R22):

!
vrf definition IWAN-TRANSPORT-2
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-4
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-5
 !
 address-family ipv4
 exit-address-family
!
interface TenGigabitEthernet0/1/0
 description ** Connected WAN2 MPLS2 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-2
 ip address 10.4.81.18 255.255.255.252
!
interface TenGigabitEthernet0/1/1
 description ** Connected to WAN4 INET2 **
 bandwidth 10000000
 vrf forwarding IWAN-TRANSPORT-4
 ip address 172.16.1.22 255.255.255.252
!
interface GigabitEthernet0/0/0
 description ** Connected to WAN5 INET4G **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-5
 ip address 172.26.1.14 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-2 0.0.0.0 0.0.0.0 10.4.81.17
ip route vrf IWAN-TRANSPORT-4 0.0.0.0 0.0.0.0 172.16.1.21
ip route vrf IWAN-TRANSPORT-5 0.0.0.0 0.0.0.0 172.26.1.13
!


Front Door VRF Configuration on DC-POP3-BR1 (R31):

!
vrf definition IWAN-TRANSPORT-1
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-3
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet0/0/1
 description ** Connected to WAN3 INET1 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-3
 ip address 172.17.1.2 255.255.255.252
!
interface GigabitEthernet0/0/2
 description ** Connected to WAN1 MPLS1 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-1
 ip address 10.4.85.2 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 10.4.85.1
ip route vrf IWAN-TRANSPORT-3 0.0.0.0 0.0.0.0 172.17.1.1
!


Front Door VRF Configuration on DC-POP3-BR2(R32):

!
vrf definition IWAN-TRANSPORT-2
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-4
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-5
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet0/0/1
 description ** Connected to WAN2 MPLS2 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-2
 ip address 10.4.85.6 255.255.255.252
!
interface GigabitEthernet0/0/2
 description ** Connected to WAN4 INET2 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-4
 ip address 172.17.1.6 255.255.255.252
!
interface GigabitEthernet0/0/3
 description ** Connected to WAN5 INET4G **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-5
 ip address 172.25.1.6 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-2 0.0.0.0 0.0.0.0 10.4.85.5
ip route vrf IWAN-TRANSPORT-4 0.0.0.0 0.0.0.0 172.17.1.5
ip route vrf IWAN-TRANSPORT-5 0.0.0.0 0.0.0.0 172.25.1.5
!


The DMVPN spoke routers at the WAN remote sites connect to the Internet directly through a router interface without a separate firewall. This connection is secured in two ways. Because the Internet interface is in a separate VRF, no traffic can access the global VRF except traffic sourced through the DMVPN tunnel. This design provides implicit security. Additionally, an IP access list permits only the traffic required for an encrypted tunnel, as well as DHCP and various ICMP protocols for troubleshooting. The IP access list must permit the protocols specified in the following configuration sample. The access list is applied inbound on the WAN interface, so filtering is done on traffic destined to the router.


interface GigabitEthernet0/1
 ip access-group ACL-INET-PUBLIC in
!
ip access-list extended ACL-INET-PUBLIC
 permit udp any any eq non500-isakmp         ! IPsec via NAT-T 
 permit udp any any eq isakmp                ! ISAKMP (UDP 500)
 permit esp any any                          ! IPSEC
 permit udp any any eq bootpc                ! DHCP


The additional protocols listed in the following table may assist in troubleshooting, but are not explicitly required to allow DMVPN to function properly.

ip access-list extended ACL-INET-PUBLIC
 permit icmp any any echo                     ! Allow remote pings 
 permit icmp any any echo-reply               ! Allow ping replies (from our requests) 
 permit icmp any any ttl-exceeded             ! Allow traceroute replies (from our requests) 
 permit icmp any any port-unreachable         ! Allow traceroute replies (from our requests) 
 permit udp any any gt 1023 ttl eq 1          ! Allow remote traceroute 


The example below illustrates the Front Door VRF Configuration on Branch1 MC-BR (R41 ) which is connected to 3 WAN links (MPLS1, INET1, INET4G).

!
vrf definition IWAN-TRANSPORT-1
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-3
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-5
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet0/0/0
 description ** Connected to WAN1 MPLS1 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-1
 ip address 10.4.81.110 255.255.255.252
!
interface GigabitEthernet0/0/1
 description ** Connected to WAN2 INET1 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-3
!
interface GigabitEthernet0/1/0
 description ** Connected to WAN3 INET4G **
 bandwidth 30000
 vrf forwarding IWAN-TRANSPORT-5
 ip address 172.30.100.1 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 10.4.81.109
ip route vrf IWAN-TRANSPORT-5 0.0.0.0 0.0.0.0 172.30.100.2
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/1 dhcp 10
!


Front Door VRF Configuration on Branch2 MC-BR (R51):

!
vrf definition IWAN-TRANSPORT-1
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-3
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet0/0/0
 description ** Connected to WAN1 MPLS1 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-1
 ip address 10.4.81.102 255.255.255.252
!
interface GigabitEthernet0/0/2
 description ** Connected to WAN3 INET1 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-3
 ip address 172.19.11.253 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-1 0.0.0.0 0.0.0.0 10.4.81.101
ip route vrf IWAN-TRANSPORT-3 0.0.0.0 0.0.0.0 172.19.11.254
!


Front Door VRF Configuration on Branch2 BR2 (R52):

!
vrf definition IWAN-TRANSPORT-2
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-4
 !
 address-family ipv4
 exit-address-family
!
vrf definition IWAN-TRANSPORT-5
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet0/0/0
 description ** Connected to WAN4 INET2 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-4
 ip address 172.20.10.2 255.255.255.252
!
interface GigabitEthernet0/0/2
 description ** Connected to WAN2 MPLS2 **
 bandwidth 1000000
 vrf forwarding IWAN-TRANSPORT-2
 ip address 10.4.99.1 255.255.255.252
!
interface GigabitEthernet0/0/3
 description ** Connected to WAN5 INET4G **
 bandwidth 30000
 vrf forwarding IWAN-TRANSPORT-5
 ip address 172.30.100.5 255.255.255.252
!
ip route vrf IWAN-TRANSPORT-2 0.0.0.0 0.0.0.0 10.4.99.2
ip route vrf IWAN-TRANSPORT-4 0.0.0.0 0.0.0.0 172.20.10.1
ip route vrf IWAN-TRANSPORT-5 0.0.0.0 0.0.0.0 172.30.100.6
!


IKEv2 and IPSec Configuration at DC routers

The primary goal of encryption is to provide data confidentiality, integrity, and authenticity by encrypting IP packets as the data travels across a network. This is not mandatory in the IWAN design, especially for DMVPN over MPLS transport. The encrypted payloads are then encapsulated with a new header (or multiple headers) and transmitted across the network.


NOTE:

  • R11, R21 and R31 are DMVPN hubs for the DMVPN cloud over MPLS 1 and also for the DMVPN cloud over INET1
  • R12, R22 and R32 are DMVPN hubs for the DMVPN cloud over MPLS 2, the DMVPN cloud over INET2 and the DMVPN cloud over INET4G link.
  • Branch 1- MCBR has one DMVPN tunnel interface each over MPLS1, INET1 and INET4G links.
  • Branch 2- MCBR has one DMVPN tunnel interface each over MPLS1 and INET1 links.
  • Branch 2- BR2 has one DMVPN tunnel interface each over MPLS2, INET2 and INET4G links.
  • Branch 3 - MCBR has one DMVPN tunnel interface each over MPLS1, INET1 and INET4G links.
  • Branch 4 - MCBR has one DMVPN tunnel interface each over MPLS2, INET2 and INET4G links.


In this solution guide, we are using IKEv2 with smart defaults and simplified commands (NHRP) to further simplify the configuration. Pre-shared key are used here for simplicity sake and as a first step to test IWAN. The PKI infrastructure and design will be fully described and explained in the IWAN Cisco Validated Design (CVD).

1. Configure the crypto keyring

  • The crypto keyring defines a pre-shared key (or password) valid for IP sources reachable within a particular VRF.
  • This key is a wildcard pre-shared key if it applies to any IP source.
  • A wildcard key is configured using the 0.0.0.0 0.0.0.0 network/mask combination.

2. IKE Proposal

  • The IKE proposal is based on smart defaults and therefore not defined here.

3. Configure the IKE Profile

  • The IKE profile creates an association between an identity address, a VRF, and a crypto keyring.
  • A wildcard address within a VRF is referenced with 0.0.0.0.

4. Define the IPSec transform set

  • A transform set is an acceptable combination of security protocols, algorithms, and other settings to apply to IPsec-protected traffic.
  • Peers agree to use a particular transform set when protecting a particular data flow.

5. Create the IPSec profile

  • The IPsec profile creates an association between an IKE profile and an IPsec transform-set.


The example below illustrates the configuration on R11, DMVPN Hub (DC-POP1-BR1). The similar configuration applies to R21 (DC-POP2-BR1) and R31 (DC-POP3-BR1).

!------------------------------------------------------------
! KEYRING 
! Use pre-share key here
!------------------------------------------------------------
!
crypto ikev2 keyring DMVPN-KEYRING-1
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-3
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !

!
!------------------------------------------------------------
! IKEv2 PROPOSAL
!
! Removed IKEv2 proposal, will use smart default
!------------------------------------------------------------
!
!
!------------------------------------------------------------
! IKEv2 PROFILE
!------------------------------------------------------------
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1
 match fvrf IWAN-TRANSPORT-1
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-1
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-3
 match fvrf IWAN-TRANSPORT-3
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-3
!
!
!------------------------------------------------------------
! IPSEC
!------------------------------------------------------------
!
! It is recommended that you use the maximum window size to eliminate future anti-replay problems. 
! If you do not increase the window size, the router may drop packets 
! and you may see the following error message on the router CLI:
! %CRYPTO-4-PKT_REPLAY_ERR:  decrypt: replay check failed
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-1
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-3
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-3
!
!


Configuration on R12, DMVPN Hub (DC-POP1-BR2) is shown below. The similar configuration applies to R22 (DC-POP2-BR2) and R32 (DC-POP3-BR2).

!------------------------------------------------------------
! KEYRING
! Use pre-share key here
!------------------------------------------------------------
!
crypto ikev2 keyring DMVPN-KEYRING-2
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-4
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-5
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
!------------------------------------------------------------
! IKEv2 PROPOSAL
!
! Removed IKEv2 proposal, will use smart default
!------------------------------------------------------------
!
!
!------------------------------------------------------------
! IKEv2 PROFILE
!------------------------------------------------------------
!

crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-2
 match fvrf IWAN-TRANSPORT-2
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-2
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-5
 match fvrf IWAN-TRANSPORT-5
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-5
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-4
 match fvrf IWAN-TRANSPORT-4
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-4
!
!------------------------------------------------------------
! IPSEC
!------------------------------------------------------------
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-2
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-2
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-4
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-4
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-5
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-5
!
!
!


DMVPN Tunnels Configuration at DC routers

The additional headers introduce a certain amount of overhead to the overall packet length. The following table highlights the packet overhead associated with encryption based on the additional headers required for various combinations of IPsec and GRE.

  • GRE only 24 bytes
  • IPsec (Transport Mode): 36 bytes
  • IPsec (Tunnel Mode): 52 bytes
  • IPsec (Transport Mode) + GRE: 60 bytes
  • IPsec (Tunnel Mode) + GRE: 76 bytes


There is a maximum transmission unit (MTU) parameter for every link in an IP network and typically the MTU is 1500 bytes. IP packets larger than 1500 bytes must be fragmented when transmitted across these links. Fragmentation is not desirable and can impact network performance. To avoid fragmentation, the original packet size plus overhead must be 1500 bytes or less, which means that the sender must reduce the original packet size. To account for other potential overhead, Cisco recommends that you configure tunnel interfaces with a 1400 byte MTU.

There are dynamic methods for network clients to discover the path MTU, which allow the clients to reduce the size of packets they transmit. However, in many cases, these dynamic methods are unsuccessful, typically because security devices filter the necessary discovery traffic. This failure to discover the path MTU drives the need for a method that can reliably inform network clients of the appropriate packet size. The solution is to implement the ip tcp adjust mss [size] command on the WAN routers, which influences the TCP maximum segment size (MSS) value reported by end hosts.

The MSS defines the maximum amount of data that a host is willing to accept in a single TCP/IP datagram. The MSS value is sent as a TCP header option only in TCP SYN segments. Each side of a TCP connection reports its MSS value to the other side. The sending host is required to limit the size of data in a single TCP segment to a value less than or equal to the MSS reported by the receiving host.

The IP and TCP headers combine for 40 bytes of overhead, so the typical MSS value reported by network clients will be 1460. This design includes encrypted tunnels with a 1400 byte MTU, so the MSS used by endpoints should be configured to be 1360 to minimize any impact of fragmentation. In this solution, you implement the ip tcp adjust mss 1360 command on all WAN facing router interfaces.


DMVPN uses multipoint GRE (mGRE) tunnels. This type of tunnel requires a source interface only.

  • Use the same source interface that you use to connect to the Internet.
  • Set the tunnel vrf command to the VRF defined previously for FVRF.
  • Configure basic interface settings
    • The bandwidth setting should be set to match the bandwidth of the respective primary or secondary carrier.
    • The IP MTU should be configured to 1400
    • The ip tcp adjust-mss should be configured to 1360.
    • There is a 40 byte difference which corresponds to the combined IP and TCP header length.
  • Configure NHRP
    • Define Next Hop Server (NHS)
    • Define static mapping to R94 and vice-versa. Note the new configuration used which combines multiple lines that we used to have.
    • Dynamic mapping for spokes
    • Enable nhrp redirect for direct spoke to spoke tunnels
    • Set NHRP holdtime to 600
  • Apply the IPSec profile to the tunnel


Below is the configuration on R11 (DC-POP1-BR1) for DMVPN tunnels over MPLS1 and INET1.

!
interface Tunnel100
 description ** Tunnel through WAN1 MPLS1 **
 bandwidth 10000000
 ip address 10.13.1.1 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 101
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 101
 tunnel vrf IWAN-TRANSPORT-1
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-1
 domain iwan path MPLS1 path-id 1
 hold-queue 4096 in
 hold-queue 4096 out
!
interface Tunnel300
 description ** Tunnel through WAN3 INET1 **
 bandwidth 10000000
 ip address 10.33.1.1 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 103
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/1/0
 tunnel mode gre multipoint
 tunnel key 103
 tunnel vrf IWAN-TRANSPORT-3
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-3
 domain iwan path INET1 path-id 3
 hold-queue 4096 in
 hold-queue 4096 out
!


Below is the configuration on R12 (DC-POP1-BR2) for DMVPN over MPLS2, INET2 and INET4G links.

!
interface Tunnel200
 description ** Tunnel Through WAN2 MPLS2 **
 bandwidth 10000000
 ip address 10.23.1.1 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 102
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/1/0
 tunnel mode gre multipoint
 tunnel key 102
 tunnel vrf IWAN-TRANSPORT-2
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-2
 domain iwan path MPLS2 path-id 2
 hold-queue 4096 in
 hold-queue 4096 out
!
interface Tunnel400
 description ** Tunnel Through WAN4 INET2 **
 bandwidth 10000000
 ip address 10.43.1.1 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 104
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 104
 tunnel vrf IWAN-TRANSPORT-4
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-4
 domain iwan path INET2 path-id 4
 hold-queue 4096 in
 hold-queue 4096 out
!
interface Tunnel500
 description ** Tunnel Through WAN5 INET4G **
 bandwidth 1000000
 ip address 10.60.1.1 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 105
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 105
 tunnel vrf IWAN-TRANSPORT-5
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-5
 domain iwan path INET4G path-id 5 path-last-resort
 hold-queue 4096 in
 hold-queue 4096 out
!


Below is the configuration on R21 (DC-POP2-BR1) for DMVPN over MPLS1 and INEt1 links.

!
interface Tunnel100
 description ** Tunnel through WAN1 MPLS1 **
 bandwidth 10000000
 ip address 10.13.1.2 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 101
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/2/0
 tunnel mode gre multipoint
 tunnel key 101
 tunnel vrf IWAN-TRANSPORT-1
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-1
 domain iwan path MPLS1 path-id 1
 hold-queue 4096 in
 hold-queue 4096 out
!
!
interface Tunnel300
 description ** Tunnel through WAN3 INET1 **
 bandwidth 10000000
 ip address 10.33.1.2 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 103
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/3/0
 tunnel mode gre multipoint
 tunnel key 103
 tunnel vrf IWAN-TRANSPORT-3
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-3
 domain iwan path INET1 path-id 3
 hold-queue 4096 in
 hold-queue 4096 out
!
!


Below is the configuration on R22 (DC-POP2-BR2) for DMVPN over MPLS2, INET2 and INET4G links.

!
interface Tunnel200
 description ** Tunnel through WAN2 MPLS2 **
 bandwidth 10000000
 ip address 10.23.1.2 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 102
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/1/0
 tunnel mode gre multipoint
 tunnel key 102
 tunnel vrf IWAN-TRANSPORT-2
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-2
 domain iwan path MPLS2 path-id 2
 hold-queue 4096 in
 hold-queue 4096 out
!
interface Tunnel400
 description ** Tunnel through WAN4 INET2 **
 bandwidth 10000000
 ip address 10.43.1.2 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 104
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source TenGigabitEthernet0/1/1
 tunnel mode gre multipoint
 tunnel key 104
 tunnel vrf IWAN-TRANSPORT-4
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-4
 domain iwan path INET2 path-id 4
 hold-queue 4096 in
 hold-queue 4096 out
!
interface Tunnel500
 description ** Tunnel through WAN5 INET4G **
 bandwidth 1000000
 ip address 10.60.1.2 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip pim nbma-mode
 ip pim sparse-mode
 ip nhrp authentication cisco123
 ip nhrp network-id 105
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 105
 tunnel vrf IWAN-TRANSPORT-5
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-5
 domain iwan path INET4G path-id 5 path-last-resort
 hold-queue 4096 in
 hold-queue 4096 out
!


Below is the configuration on R31 (DC-POP3-BR1) for DMVPN over MPLS1 and INEt1 links.

!
interface Tunnel100
 description ** Tunnel through WAN1 MPLS1 **
 bandwidth 1000000
 ip address 10.13.1.3 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 101
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/2
 tunnel mode gre multipoint
 tunnel key 101
 tunnel vrf IWAN-TRANSPORT-1
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-1
 domain iwan path MPLS1 path-id 1
 hold-queue 4096 in
 hold-queue 4096 out
!
!
interface Tunnel300
 description ** Tunnel through WAN3 INET1 **
 bandwidth 1000000
 ip address 10.33.1.3 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 103
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/1
 tunnel mode gre multipoint
 tunnel key 103
 tunnel vrf IWAN-TRANSPORT-3
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-3
 domain iwan path INET1 path-id 3
 hold-queue 4096 in
 hold-queue 4096 out
!
!


Below is the configuration on R32 (DC-POP3-BR2) for DMVPN over MPLS2, INET2 and INET4G links.

!
interface Tunnel200
 description ** Tunnel through WAN2 MPLS2 **
 bandwidth 1000000
 ip address 10.23.1.3 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 102
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/1
 tunnel mode gre multipoint
 tunnel key 102
 tunnel vrf IWAN-TRANSPORT-2
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-2
 domain iwan path MPLS2 path-id 2
 hold-queue 4096 in
 hold-queue 4096 out
!
!
interface Tunnel400
 description ** Tunnel through WAN4 INET2 **
 bandwidth 1000000
 ip address 10.43.1.3 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 104
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/2
 tunnel mode gre multipoint
 tunnel key 104
 tunnel vrf IWAN-TRANSPORT-4
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-4
 domain iwan path INET2 path-id 4
 hold-queue 4096 in
 hold-queue 4096 out
!
interface Tunnel500
 description ** Tunnel through WAN5 INET4G **
 bandwidth 1000000
 ip address 10.60.1.3 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 105
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/0/3
 tunnel mode gre multipoint
 tunnel key 105
 tunnel vrf IWAN-TRANSPORT-5
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-5
 domain iwan path INET4G path-id 5 path-last-resort
 hold-queue 4096 in
 hold-queue 4096 out
!


IKEv2 & IPSec Configuration at Spokes/Branch routers

IKEv2 and IPSec configuration is basically the same as explained in the hub section with the following additions:

  • A best practice is to enable Dead Peer Detection (DPD) on the spokes. Dead Peer Detection (DPD) detects unreachable IKE peers and Each peer’s DPD state is independent of the others. DPD is not recommended for Hub routers – it causes an increase in CPU overhead with large number of peers.
    • with keepalive intervals sent at 40-second intervals
    • with a 5-second retry interval, which is considered to be a reasonable setting to detect a failed hub.
  • We also enables if-state nhrp to make the tunnel going down.


R41 (Site1-MCBR) Configuration is as follows. NOTE: Similar configs applies to R61 (Site3-MCBR) as well.

!------------------------------------------------------------
! KEYRING
!------------------------------------------------------------
!
crypto ikev2 keyring DMVPN-KEYRING-1
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-3
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-5
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
!------------------------------------------------------------
! IKEv2 PROPOSAL
!
! Removed IKEv2 proposal, will use smart default
!------------------------------------------------------------
!
!
!------------------------------------------------------------
! IKEv2 PROFILE
!------------------------------------------------------------
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1
 match fvrf IWAN-TRANSPORT-1
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-1
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-3
 match fvrf IWAN-TRANSPORT-3
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-3
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-5
 match fvrf IWAN-TRANSPORT-5
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-5
!
crypto ikev2 dpd 40 5 on-demand

!
!------------------------------------------------------------
! IPSEC
!------------------------------------------------------------
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-1
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-3
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-3
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-5
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-5
!
!


R51 (Site2-MCBR) Configuration is as follows.

!------------------------------------------------------------
! KEYRING
!------------------------------------------------------------
!
crypto ikev2 keyring DMVPN-KEYRING-1
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-3
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
!------------------------------------------------------------
! IKEv2 PROPOSAL
!
! Removed IKEv2 proposal, will use smart default
!------------------------------------------------------------
!
!
!------------------------------------------------------------
! IKEv2 PROFILE
!------------------------------------------------------------
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-1
 match fvrf IWAN-TRANSPORT-1
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-1
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-3
 match fvrf IWAN-TRANSPORT-3
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-3
!
crypto ikev2 dpd 40 5 on-demand
!
!
!------------------------------------------------------------
! IPSEC
!------------------------------------------------------------
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-1
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-1
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-3
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-3
!
!
!


R52 (Site2-MCBR) Configuration is as follows. NOTE: Similar configs applies to R71 (Site4-MCBR) as well.

!------------------------------------------------------------
! KEYRING
!------------------------------------------------------------
!
crypto ikev2 keyring DMVPN-KEYRING-2
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-5
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
crypto ikev2 keyring DMVPN-KEYRING-4
 peer ANY
  address 0.0.0.0 0.0.0.0
  pre-shared-key c1sco123
 !
!
!
!------------------------------------------------------------
! IKEv2 PROPOSAL
!
! Removed IKEv2 proposal, will use smart default
!------------------------------------------------------------
!
!
!------------------------------------------------------------
! IKEv2 PROFILE
!------------------------------------------------------------
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-2
 match fvrf IWAN-TRANSPORT-2
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-2
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-5
 match fvrf IWAN-TRANSPORT-5
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-5
!
crypto ikev2 profile FVRF-IKEv2-IWAN-TRANSPORT-4
 match fvrf IWAN-TRANSPORT-4
 match identity remote address 0.0.0.0
 authentication local pre-share
 authentication remote pre-share
 keyring local DMVPN-KEYRING-4
!
crypto ikev2 dpd 40 5 on-demand
!
!
!------------------------------------------------------------
! IPSEC
!------------------------------------------------------------
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-2
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-2
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-4
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-4
!
crypto ipsec profile DMVPN-PROFILE-TRANSPORT-5
 set transform-set AES256/SHA/TRANSPORT
 set ikev2-profile FVRF-IKEv2-IWAN-TRANSPORT-5
!

!


DMVPN Tunnels Configuration at Spokes/Branch routers

DMVPN uses multipoint GRE (mGRE) tunnels. This type of tunnel requires a source interface only. A spoke will have a static definition of all DMVPN Next Hop Servers available on Hub and Transit Sites. NHS are R11/R21 on tunnel100 for the DMVPN network over the MPLS transport and R12/R22 for the DMVPN network over the INET transport.

  • Use the same source interface that you use to connect to the MPLS provider.
  • Set the tunnel vrf command to the VRF defined previously for FVRF.
  • Configure basic interface settings
    • The IP MTU should be configured to 1400
    • The ip tcp adjust-mss should be configured to 1360.
    • There is a 40 byte difference which corresponds to the combined IP and TCP header length.
  • Configure NHRP for DMVPN1 (MPLS) and DMVPN2 (INET)
    • R11 and R21 are defined as Next Hop Servers (NHS) on Tunnel100
    • R12 and R22 are defined as Next Hop Servers (NHS) on Tunnel200
    • Note the simplified syntax that enables NHS with a single line. Both are active, so both tunnels will be up. Routing will decide which one will be used.
    • Note that PfRv3 currently supports only one next-hop per DMVPN interface.
    • Enable NHRP shortcut for direct spoke to spoke tunnels
    • Set NHRP holdtime to 600
  • Apply the IPSec profile to the tunnel


The example below illustrates the configuration on R41 (Site1-MCBR) branch:

!
interface Tunnel100
 description ** DMVPN Tunnel over MPLS1 **
 bandwidth 1000000
 ip address 10.13.1.202 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 101
 ip nhrp nhs 10.13.1.1 nbma 10.4.81.2 multicast
 ip nhrp nhs 10.13.1.2 nbma 10.4.81.6 multicast
 ip nhrp nhs 10.13.1.3 nbma 10.4.85.2 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 101
 tunnel vrf IWAN-TRANSPORT-1
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-1
!
interface Tunnel300
 description ** DMVPN Tunnel over INET1 **
 bandwidth 1000000
 ip address 10.33.1.202 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 103
 ip nhrp nhs 10.33.1.1 nbma 172.16.1.2 multicast
 ip nhrp nhs 10.33.1.2 nbma 172.16.1.6 multicast
 ip nhrp nhs 10.33.1.3 nbma 172.17.1.2 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/1
 tunnel mode gre multipoint
 tunnel key 103
 tunnel vrf IWAN-TRANSPORT-3
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-3
 
!
interface Tunnel500
 description ** DMVPN Tunnel over Internet 4G Cloud **
 bandwidth 30000
 ip address 10.60.1.202 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 105
 ip nhrp nhs 10.60.1.1 nbma 172.26.1.10 multicast
 ip nhrp nhs 10.60.1.2 nbma 172.26.1.14 multicast
 ip nhrp nhs 10.60.1.3 nbma 172.25.1.6 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/1/0
 tunnel mode gre multipoint
 tunnel key 105
 tunnel vrf IWAN-TRANSPORT-5
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-5
!

The example below illustrates the configuration on R51 (Site2-MCBR) branch:

!
interface Tunnel100
 description ** Tunnel through WAN1 MPLS1 **
 bandwidth 1000000
 ip address 10.13.1.200 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 101
 ip nhrp nhs 10.13.1.1 nbma 10.4.81.2 multicast
 ip nhrp nhs 10.13.1.2 nbma 10.4.81.6 multicast
 ip nhrp nhs 10.13.1.3 nbma 10.4.85.2 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 101
 tunnel vrf IWAN-TRANSPORT-1
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-1
!
interface Tunnel300
 description ** Tunnel through WAN3 INET1 **
 bandwidth 1000000
 ip address 10.33.1.200 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 103
 ip nhrp nhs 10.33.1.1 nbma 172.16.1.2 multicast
 ip nhrp nhs 10.33.1.2 nbma 172.16.1.6 multicast
 ip nhrp nhs 10.33.1.3 nbma 172.17.1.2 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/2
 tunnel mode gre multipoint
 tunnel key 103
 tunnel vrf IWAN-TRANSPORT-3
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-3
!

The example below illustrates the configuration on R52 (Site2-BR2) branch:

!
interface Tunnel200
 description ** Tunnel through WAN2 MPLS2 **
 bandwidth 1000000
 ip address 10.23.1.200 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 102
 ip nhrp nhs 10.23.1.1 nbma 10.4.81.14 multicast
 ip nhrp nhs 10.23.1.2 nbma 10.4.81.18 multicast
 ip nhrp nhs 10.23.1.3 nbma 10.4.85.6 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/2
 tunnel mode gre multipoint
 tunnel key 102
 tunnel vrf IWAN-TRANSPORT-2
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-2
 
!
interface Tunnel400
 description ** Tunnel through WAN4 INET2 **
 bandwidth 1000000
 ip address 10.43.1.200 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 104
 ip nhrp nhs 10.43.1.1 nbma 172.16.1.18 multicast
 ip nhrp nhs 10.43.1.2 nbma 172.16.1.22 multicast
 ip nhrp nhs 10.43.1.3 nbma 172.17.1.6 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 104
 tunnel vrf IWAN-TRANSPORT-4
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-4
 
!
interface Tunnel500
 description ** Tunnel through WAN5 INET4G **
 bandwidth 30000
 ip address 10.60.1.200 255.255.0.0
 no ip redirects
 ip mtu 1400
 ip pim dr-priority 0
 ip pim nbma-mode
 ip pim sparse-mode
 ip nbar protocol-discovery
 ip nhrp authentication cisco123
 ip nhrp network-id 105
 ip nhrp nhs 10.60.1.1 nbma 172.26.1.10 multicast
 ip nhrp nhs 10.60.1.2 nbma 172.26.1.14 multicast
 ip nhrp nhs 10.60.1.3 nbma 172.25.1.6 multicast
 ip tcp adjust-mss 1360
 load-interval 30
 no nhrp route-watch
 if-state nhrp
 tunnel source GigabitEthernet0/0/3
 tunnel mode gre multipoint
 tunnel key 105
 tunnel vrf IWAN-TRANSPORT-5
 tunnel protection ipsec profile DMVPN-PROFILE-TRANSPORT-5
 !

Notes:

  • NHRP requires all devices within a DMVPN cloud to use the same network ID and authentication key. The NHRP cache holdtime should be configured to 600 seconds.
  • ip nhrp registration no-unique: for designs where DMVPN spoke routers receive their external IP addresses through DHCP. It is possible for these routers to acquire different IP addresses after a reload. When the router attempts to register with the NHRP server, it may appear as a duplicate to an entry already in the cache and be rejected. The registration no-unique option allows you to overwrite existing cache entries. This feature is only required on NHRP clients (DMVPN spoke routers).
  • The if-state nhrp: This command will display Tunnel interface as down if NHRP is unable to register with the hub(s) due to Hub outage or blackhole impairment.
  • no nhrp route-watch: on spokes only with multiple interfaces, configured on interfaces that are blind. NHRP Route Watch will not install a shortcut if a parent route doesn't exist in the RIB. With IWAN PfR is monitoring the EIGRP Topology Table or BGP Table for parent routes over the specific interface. This allows a shortcut to be installed over this path for PfR controlled traffic.


BGP Routing on the Overlay Backbone

BGP Routing Overview

BGP can be deployed as the IWAN / DMVPN routing protocol as an alternative to EIGRP. BGP is a popular choice for network operators that require a rich set of features to customized path selection in complex topologies and large-scale deployments. While traditionally positioned at the Provider WAN edge, recent enhancements such as BGP dynamic neighbors make it a viable choice for IWAN deployment, as static peers no longer need to be defined, allowing for zero touch deployment.

  • Hub Border Routers are BGP Route Reflectors and use BGP Dynamic Neighbors to simplify the configuration:
    • BGP dynamic neighbor support allows BGP peering to a group of remote neighbors that are defined by a range of IP addresses. Each range can be configured as a subnet IP address.
    • This allows spokes to initiate the BGP peering without having to preconfigure remote peers on the route-reflectors.


Principles:

  • A single iBGP routing domain is used
  • Appropriate Hello/Hold timers are defined for IWAN (20/60)
  • Hub:
    • DMVPN hub routers function as BGP route-reflectors for the spokes.
    • No BGP peering between RR.
    • BGP dynamic peer feature configured on the route-reflectors
    • Site specific prefixes, Enterprise summary prefix and default route advertised to spokes
    • Set local preference for all prefixes
    • Redistribute BGP into local IGP with a defined metric cost to attract traffic from the central sites to the spokes across MPLS.
  • Spokes:
    • Peer to Hub/Transit BRs in each DMVPN cloud
    • Mutual redistribution OSPF/BGP
    • Set a route tag to identify routes redistributed from BGP
    • Preferred path is MPLS due to highest Local Preference


When BGP is used, PfRv3 will be able to check in the BGP database and will use the best path as computed by BGP. This path needs to be via an external interface (WAN interface). If that is not the case, then PfRv3 will choose in sequence the path with biggest weight, then biggest local preference and finally the path with the smallest IP address.


BGP Hub/DC Configuration

R11, R21 and R31 are BGP Route Reflectors over MPLS1 and INET1 links. And R12, R22 and R32 are BGP Route Reflectors over MPLS2, INET2 and INET4G links.

With BGP Dynamic Neighbor, R11, R12, R21, R22, R31 and R32 just listen to incoming BGP connections. This avoids the manual configuration of all remote sites neighbors. In this design, there is no mutual redistribution, BGP is only redistributed into OSPF.

Tasks include the following on the hub routers:

  • Enable the BGP process for DMVPN routing
  • Configure BGP route advertisement with local preference.
  • Configure BGP to OSPF redistribution. The routing policy redistribution design is constructed so that an MPLS outbound DMVPN path is preferred over the Internet DMVPN path, when both are available.
  • Block redistribution of DMVPN Tunnel Interfaces
  • Change the iBGP administrative distance because OSPF has a lower administrative distance (AD), 110, then a route learned from an IBGP peer, 200.


R11 (DC-POP1-BR1) Overlay routing Configuration:


!--------------------------------------------------------------------
! OSPF
!--------------------------------------------------------------------
!
router ospf 100
 router-id 10.8.88.20
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface TenGigabitEthernet0/0/1
 network 10.0.0.0 0.255.255.255 area 0
!
!
!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.8.88.20
 bgp log-neighbor-changes
 bgp listen range 10.33.0.0/16 peer-group INET1-SPOKES
 bgp listen range 10.13.0.0/16 peer-group MPLS1-SPOKES
 bgp listen limit 2100
 neighbor MPLS1-SPOKES peer-group
 neighbor MPLS1-SPOKES remote-as 10
 neighbor MPLS1-SPOKES description MPLS1 Spoke Route Reflector
 neighbor MPLS1-SPOKES update-source Tunnel100
 neighbor MPLS1-SPOKES timers 20 60
 neighbor INET1-SPOKES peer-group
 neighbor INET1-SPOKES remote-as 10
 neighbor INET1-SPOKES description INET1 Spoke Route Reflector
 neighbor INET1-SPOKES update-source Tunnel300
 neighbor INET1-SPOKES timers 20 60
 !
 address-family ipv4
  bgp redistribute-internal
  network 0.0.0.0
  network 10.0.0.0
  network 10.4.200.0 mask 255.255.255.0
  network 10.4.201.0 mask 255.255.255.0
  network 10.5.0.0 mask 255.255.0.0
  network 10.6.0.0 mask 255.255.0.0
  network 10.7.0.0 mask 255.255.0.0
  network 10.8.0.0 mask 255.255.0.0
  network 10.8.88.10 mask 255.255.255.255
  network 10.8.88.20 mask 255.255.255.255
  network 10.9.0.0 mask 255.255.0.0
  network 10.77.0.0 mask 255.255.0.0
  neighbor MPLS1-SPOKES activate
  neighbor MPLS1-SPOKES send-community
  neighbor MPLS1-SPOKES route-reflector-client
  neighbor MPLS1-SPOKES next-hop-self all
  neighbor MPLS1-SPOKES weight 50000
  neighbor MPLS1-SPOKES soft-reconfiguration inbound
  neighbor MPLS1-SPOKES route-map WAN-IN in
  neighbor MPLS1-SPOKES route-map MPLS1-OUT out
  neighbor INET1-SPOKES activate
  neighbor INET1-SPOKES send-community
  neighbor INET1-SPOKES next-hop-self all
  neighbor INET1-SPOKES weight 45000
  neighbor INET1-SPOKES soft-reconfiguration inbound
  neighbor INET1-SPOKES route-map WAN-IN in
  neighbor INET1-SPOKES route-map INET1-OUT out
  maximum-secondary-paths ibgp 1
  distance bgp 201 89 89
 exit-address-family
!
ip forward-protocol nd
!
ip bgp-community new-format
!
ip community-list standard POP1-SPOKES permit 10:10
ip community-list standard POP2-SPOKES permit 10:20
ip community-list standard POP3-SPOKES permit 10:30
!
!--------------------------------------------------------------------
! PREFIX LIST
!--------------------------------------------------------------------
!
ip prefix-list BRANCH-PREFIX seq 10 permit 10.5.0.0/16
ip prefix-list BRANCH-PREFIX seq 20 permit 10.6.0.0/16
ip prefix-list BRANCH-PREFIX seq 30 permit 10.7.0.0/16
ip prefix-list BRANCH-PREFIX seq 40 permit 10.9.0.0/16
!
ip prefix-list DEFAULT-ROUTE seq 10 permit 0.0.0.0/0
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list LOCALBRLOOPBACK seq 10 permit 10.8.88.20/32
!
ip prefix-list LOCALDC-PREFIX seq 10 permit 10.4.200.0/24
ip prefix-list LOCALDC-PREFIX seq 20 permit 10.4.201.0/24
ip prefix-list LOCALDC-PREFIX seq 30 permit 10.77.0.0/16
ip prefix-list LOCALDC-PREFIX seq 40 permit 10.8.0.0/16
!
ip prefix-list LOCALMCLOOPBACK seq 10 permit 10.8.88.10/32
!
ip prefix-list TUNNEL-DMVPN seq 10 permit 10.13.0.0/16
ip prefix-list TUNNEL-DMVPN seq 20 permit 10.23.0.0/16
ip prefix-list TUNNEL-DMVPN seq 30 permit 10.33.0.0/16
ip prefix-list TUNNEL-DMVPN seq 40 permit 10.43.0.0/16
ip prefix-list TUNNEL-DMVPN seq 50 permit 10.60.0.0/16
!
!--------------------------------------------------------------------
! ROUTE MAP
!--------------------------------------------------------------------
!
route-map INET1-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK ENTERPRISE-PREFIX
 set local-preference 780
 set community 10:300
!
route-map MPLS1-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK ENTERPRISE-PREFIX LOCALBRLOOPBACK
 set local-preference 800
 set community 10:100
!
route-map WAN-IN deny 10
 description All Blocked Prefixes to come IN on BGP
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK TUNNEL-DMVPN
!
route-map WAN-IN permit 1000
 description Allow Everything Else
!
route-map REDIST-BGP-TO-OSPF permit 10
 match community POP2-SPOKES
 set metric 2000
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF permit 20
 match community POP3-SPOKES
 set metric 3000
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF deny 30
 description Block Null routes to be distributed from BGP to OSPF
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX ENTERPRISE-PREFIX
!
route-map REDIST-BGP-TO-OSPF permit 1000
 set metric 1000
 set metric-type type-1
!
!

Notes:

  • All spokes are iBGP peers
  • R11 advertises datacenter prefix summaries, enterprise network summary and default route
  • An outbound route-map is used to tag BGP announcements to the spokes a local preference. R11 has the highest local preference and is the preferred hub.


R12 (DC-POP1-BR2) Overlay routing Configuration:

!--------------------------------------------------------------------
! OSPF
!--------------------------------------------------------------------
!
router ospf 100
 router-id 10.88.88.130
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface TenGigabitEthernet0/1/2
 network 10.0.0.0 0.255.255.255 area 0
!
!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.88.88.130
 bgp log-neighbor-changes
 bgp listen range 10.60.0.0/16 peer-group INET4G-SPOKES
 bgp listen range 10.43.0.0/16 peer-group INET2-SPOKES
 bgp listen range 10.23.0.0/16 peer-group MPLS2-SPOKES
 bgp listen limit 2100
 neighbor INET2-SPOKES peer-group
 neighbor INET2-SPOKES remote-as 10
 neighbor INET2-SPOKES description INET2 Spokes Route Reflector
 neighbor INET2-SPOKES update-source Tunnel400
 neighbor INET2-SPOKES timers 20 60
 neighbor MPLS2-SPOKES peer-group
 neighbor MPLS2-SPOKES remote-as 10
 neighbor MPLS2-SPOKES description MPLS2 Spoke Route Reflector
 neighbor MPLS2-SPOKES update-source Tunnel200
 neighbor MPLS2-SPOKES timers 20 60
 neighbor INET4G-SPOKES peer-group
 neighbor INET4G-SPOKES remote-as 10
 neighbor INET4G-SPOKES description INET4G Spokes Route Reflector
 neighbor INET4G-SPOKES update-source Tunnel500
 neighbor INET4G-SPOKES timers 20 60
 !
 address-family ipv4
  bgp redistribute-internal
  network 0.0.0.0
  network 10.0.0.0
  network 10.4.200.0 mask 255.255.255.0
  network 10.4.201.0 mask 255.255.255.0
  network 10.5.0.0 mask 255.255.0.0
  network 10.6.0.0 mask 255.255.0.0
  network 10.7.0.0 mask 255.255.0.0
  network 10.9.0.0 mask 255.255.0.0
  network 10.66.0.0 mask 255.255.0.0
  network 10.88.0.0 mask 255.255.0.0
  network 10.88.88.100 mask 255.255.255.255
  neighbor INET2-SPOKES activate
  neighbor INET2-SPOKES send-community
  neighbor INET2-SPOKES route-reflector-client
  neighbor INET2-SPOKES next-hop-self all
  neighbor INET2-SPOKES weight 45000
  neighbor INET2-SPOKES soft-reconfiguration inbound
  neighbor INET2-SPOKES route-map WAN-IN in
  neighbor INET2-SPOKES route-map INET2-OUT out
  neighbor MPLS2-SPOKES activate
  neighbor MPLS2-SPOKES send-community
  neighbor MPLS2-SPOKES route-reflector-client
  neighbor MPLS2-SPOKES next-hop-self all
  neighbor MPLS2-SPOKES weight 50000
  neighbor MPLS2-SPOKES soft-reconfiguration inbound
  neighbor MPLS2-SPOKES route-map WAN-IN in
  neighbor MPLS2-SPOKES route-map MPLS2-OUT out
  neighbor INET4G-SPOKES activate
  neighbor INET4G-SPOKES send-community
  neighbor INET4G-SPOKES route-reflector-client
  neighbor INET4G-SPOKES next-hop-self all
  neighbor INET4G-SPOKES weight 40000
  neighbor INET4G-SPOKES soft-reconfiguration inbound
  neighbor INET4G-SPOKES route-map WAN-IN in
  neighbor INET4G-SPOKES route-map INET4G-OUT out
  maximum-secondary-paths ibgp 2
  distance bgp 201 89 89
 exit-address-family
!
ip forward-protocol nd
!
ip bgp-community new-format
!
ip community-list standard POP1-SPOKES permit 10:10
ip community-list standard POP2-SPOKES permit 10:20
ip community-list standard POP3-SPOKES permit 10:30
!
!--------------------------------------------------------------------
! PREFIX-LIST
!--------------------------------------------------------------------
!

ip prefix-list BRANCH-PREFIX seq 10 permit 10.5.0.0/16
ip prefix-list BRANCH-PREFIX seq 20 permit 10.6.0.0/16
ip prefix-list BRANCH-PREFIX seq 30 permit 10.7.0.0/16
ip prefix-list BRANCH-PREFIX seq 40 permit 10.9.0.0/16
!
ip prefix-list DEFAULT-ROUTE seq 10 permit 0.0.0.0/0
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list LOCALDC-PREFIX seq 10 permit 10.4.200.0/24
ip prefix-list LOCALDC-PREFIX seq 20 permit 10.4.201.0/24
ip prefix-list LOCALDC-PREFIX seq 30 permit 10.77.0.0/16
ip prefix-list LOCALDC-PREFIX seq 40 permit 10.8.0.0/16
!
ip prefix-list LOCALMCLOOPBACK seq 10 permit 10.8.88.10/32
!
ip prefix-list TUNNEL-DMVPN seq 10 permit 10.13.0.0/16
ip prefix-list TUNNEL-DMVPN seq 20 permit 10.23.0.0/16
ip prefix-list TUNNEL-DMVPN seq 30 permit 10.33.0.0/16
ip prefix-list TUNNEL-DMVPN seq 40 permit 10.43.0.0/16
ip prefix-list TUNNEL-DMVPN seq 50 permit 10.60.0.0/16
!
!
!--------------------------------------------------------------------
! ROUTE MAP
!--------------------------------------------------------------------
!
route-map INET4G-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 560
 set community 10:501
!
route-map INET2-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 570
 set community 10:401
!
route-map MPLS2-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 590
 set community 10:201
!
route-map WAN-IN deny 10
 description All Blocked Prefixes to come IN on BGP
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK TUNNEL-DMVPN
!
route-map WAN-IN permit 1000
 description Allow Everything Else
!
route-map REDIST-BGP-TO-OSPF permit 10
 match community POP1-SPOKES
 set metric 2200
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF permit 20
 match community POP3-SPOKES
 set metric 3200
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF deny 30
 description Block Null routes to be distributed from BGP to OSPF
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX ENTERPRISE-PREFIX
!
route-map REDIST-BGP-TO-OSPF permit 1000
 set metric 1200
 set metric-type type-1
!

Notes:

  • All spokes are iBGP peers
  • An outbound route-map is used to tag BGP announcements to the spokes with a local preference


R21 (DC-POP2-BR1) Overlay routing Configuration:


!--------------------------------------------------------------------
! OSPF
!--------------------------------------------------------------------
!
router ospf 100
 router-id 10.88.88.120
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface TenGigabitEthernet0/1/0
 network 10.0.0.0 0.255.255.255 area 0
!
!
!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.88.88.120
 bgp log-neighbor-changes
 bgp listen range 10.33.0.0/16 peer-group INET1-SPOKES
 bgp listen range 10.13.0.0/16 peer-group MPLS1-SPOKES
 bgp listen limit 2100
 neighbor MPLS1-SPOKES peer-group
 neighbor MPLS1-SPOKES remote-as 10
 neighbor MPLS1-SPOKES description MPLS1 Spoke Route Reflector
 neighbor MPLS1-SPOKES update-source Tunnel100
 neighbor MPLS1-SPOKES timers 20 60
 neighbor INET1-SPOKES peer-group
 neighbor INET1-SPOKES remote-as 10
 neighbor INET1-SPOKES description INET1 Spokes Route Reflector
 neighbor INET1-SPOKES update-source Tunnel300
 neighbor INET1-SPOKES timers 20 60
 !
 address-family ipv4
  bgp redistribute-internal
  network 0.0.0.0
  network 10.0.0.0
  network 10.4.200.0 mask 255.255.255.0
  network 10.4.201.0 mask 255.255.255.0
  network 10.5.0.0 mask 255.255.0.0
  network 10.6.0.0 mask 255.255.0.0
  network 10.7.0.0 mask 255.255.0.0
  network 10.9.0.0 mask 255.255.0.0
  network 10.66.0.0 mask 255.255.0.0
  network 10.88.0.0 mask 255.255.0.0
  network 10.88.88.100 mask 255.255.255.255
  network 10.88.88.120 mask 255.255.255.255
  neighbor MPLS1-SPOKES activate
  neighbor MPLS1-SPOKES send-community
  neighbor MPLS1-SPOKES route-reflector-client
  neighbor MPLS1-SPOKES next-hop-self all
  neighbor MPLS1-SPOKES weight 50000
  neighbor MPLS1-SPOKES soft-reconfiguration inbound
  neighbor MPLS1-SPOKES route-map WAN-IN in
  neighbor MPLS1-SPOKES route-map MPLS1-OUT out
  neighbor INET1-SPOKES activate
  neighbor INET1-SPOKES send-community
  neighbor INET1-SPOKES route-reflector-client
  neighbor INET1-SPOKES next-hop-self all
  neighbor INET1-SPOKES weight 45000
  neighbor INET1-SPOKES soft-reconfiguration inbound
  neighbor INET1-SPOKES route-map WAN-IN in
  neighbor INET1-SPOKES route-map INET1-OUT out
  maximum-secondary-paths ibgp 1
  distance bgp 201 89 89
 exit-address-family
!
ip forward-protocol nd
!
ip bgp-community new-format
!
ip community-list standard POP1-SPOKES permit 10:10
ip community-list standard POP2-SPOKES permit 10:20
ip community-list standard POP3-SPOKES permit 10:30
!
!
!--------------------------------------------------------------------
! PREFIX-LIST
!--------------------------------------------------------------------
!
ip prefix-list BRANCH-PREFIX seq 10 permit 10.5.0.0/16
ip prefix-list BRANCH-PREFIX seq 20 permit 10.6.0.0/16
ip prefix-list BRANCH-PREFIX seq 30 permit 10.7.0.0/16
ip prefix-list BRANCH-PREFIX seq 40 permit 10.9.0.0/16
!
ip prefix-list DEFAULT-ROUTE seq 10 permit 0.0.0.0/0
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list LOCALDC-PREFIX seq 10 permit 10.4.200.0/24
ip prefix-list LOCALDC-PREFIX seq 20 permit 10.4.201.0/24
ip prefix-list LOCALDC-PREFIX seq 30 permit 10.66.0.0/16
ip prefix-list LOCALDC-PREFIX seq 40 permit 10.88.0.0/16
!
ip prefix-list LOCALMCLOOPBACK seq 10 permit 10.88.88.100/32
!
ip prefix-list LOCALBRLOOPBACK seq 10 permit 10.88.88.120/32
!
ip prefix-list TUNNEL-DMVPN seq 10 permit 10.13.0.0/16
ip prefix-list TUNNEL-DMVPN seq 20 permit 10.23.0.0/16
ip prefix-list TUNNEL-DMVPN seq 30 permit 10.33.0.0/16
ip prefix-list TUNNEL-DMVPN seq 40 permit 10.43.0.0/16
ip prefix-list TUNNEL-DMVPN seq 50 permit 10.60.0.0/16
!
!--------------------------------------------------------------------
! ROUTE MAP
!--------------------------------------------------------------------
!
route-map INET1-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 580
 set community 10:301
!
route-map MPLS1-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX LOCALBRLOOPBACK
 set local-preference 600
 set community 10:101
!
route-map WAN-IN deny 10
 description All Blocked Prefixes to come IN on BGP
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK TUNNEL-DMVPN
!
route-map WAN-IN permit 1000
 description Allow Everything Else
!
route-map REDIST-BGP-TO-OSPF permit 10
 match community POP1-SPOKES
 set metric 2000
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF permit 20
 match community POP3-SPOKES
 set metric 3000
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF deny 30
 description Block Null routes to be distributed from BGP to OSPF
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX ENTERPRISE-PREFIX
!
route-map REDIST-BGP-TO-OSPF permit 1000
 set metric 1000
 set metric-type type-1
!

Notes:

  • All spokes are iBGP peers
  • R21 advertises datacenter prefix summaries, enterprise network summary and default route
  • An outbound route-map is used to tag BGP announcements to the spokes a local preference. R11 has the highest local preference and is the preferred hub.


R22 (DC-POP2-BR2) Overlay routing Configuration:

!--------------------------------------------------------------------
! OSPF
!--------------------------------------------------------------------
!
router ospf 100
 router-id 10.88.88.130
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface TenGigabitEthernet0/1/2
 network 10.0.0.0 0.255.255.255 area 0
!
!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!

router bgp 10
 bgp router-id 10.88.88.130
 bgp log-neighbor-changes
 bgp listen range 10.60.0.0/16 peer-group INET4G-SPOKES
 bgp listen range 10.43.0.0/16 peer-group INET2-SPOKES
 bgp listen range 10.23.0.0/16 peer-group MPLS2-SPOKES
 bgp listen limit 2100
 neighbor INET2-SPOKES peer-group
 neighbor INET2-SPOKES remote-as 10
 neighbor INET2-SPOKES description INET2 Spokes Route Reflector
 neighbor INET2-SPOKES update-source Tunnel400
 neighbor INET2-SPOKES timers 20 60
 neighbor MPLS2-SPOKES peer-group
 neighbor MPLS2-SPOKES remote-as 10
 neighbor MPLS2-SPOKES description MPLS2 Spoke Route Reflector
 neighbor MPLS2-SPOKES update-source Tunnel200
 neighbor MPLS2-SPOKES timers 20 60
 neighbor INET4G-SPOKES peer-group
 neighbor INET4G-SPOKES remote-as 10
 neighbor INET4G-SPOKES description INET4G Spokes Route Reflector
 neighbor INET4G-SPOKES update-source Tunnel500
 neighbor INET4G-SPOKES timers 20 60
 !
 address-family ipv4
  bgp redistribute-internal
  network 0.0.0.0
  network 10.0.0.0
  network 10.4.200.0 mask 255.255.255.0
  network 10.4.201.0 mask 255.255.255.0
  network 10.5.0.0 mask 255.255.0.0
  network 10.6.0.0 mask 255.255.0.0
  network 10.7.0.0 mask 255.255.0.0
  network 10.9.0.0 mask 255.255.0.0
  network 10.66.0.0 mask 255.255.0.0
  network 10.88.0.0 mask 255.255.0.0
  network 10.88.88.100 mask 255.255.255.255
  neighbor INET2-SPOKES activate
  neighbor INET2-SPOKES send-community
  neighbor INET2-SPOKES route-reflector-client
  neighbor INET2-SPOKES next-hop-self all
  neighbor INET2-SPOKES weight 45000
  neighbor INET2-SPOKES soft-reconfiguration inbound
  neighbor INET2-SPOKES route-map WAN-IN in
  neighbor INET2-SPOKES route-map INET2-OUT out
  neighbor MPLS2-SPOKES activate
  neighbor MPLS2-SPOKES send-community
  neighbor MPLS2-SPOKES route-reflector-client
  neighbor MPLS2-SPOKES next-hop-self all
  neighbor MPLS2-SPOKES weight 50000
  neighbor MPLS2-SPOKES soft-reconfiguration inbound
  neighbor MPLS2-SPOKES route-map WAN-IN in
  neighbor MPLS2-SPOKES route-map MPLS2-OUT out
  neighbor INET4G-SPOKES activate
  neighbor INET4G-SPOKES send-community
  neighbor INET4G-SPOKES route-reflector-client
  neighbor INET4G-SPOKES next-hop-self all
  neighbor INET4G-SPOKES weight 40000
  neighbor INET4G-SPOKES soft-reconfiguration inbound
  neighbor INET4G-SPOKES route-map WAN-IN in
  neighbor INET4G-SPOKES route-map INET4G-OUT out
  maximum-secondary-paths ibgp 2
  distance bgp 201 89 89
 exit-address-family
!
ip forward-protocol nd
!
ip bgp-community new-format
!
ip community-list standard POP1-SPOKES permit 10:10
ip community-list standard POP2-SPOKES permit 10:20
ip community-list standard POP3-SPOKES permit 10:30
!
!--------------------------------------------------------------------
! PREFIX LIST
!--------------------------------------------------------------------
!
ip prefix-list BRANCH-PREFIX seq 10 permit 10.5.0.0/16
ip prefix-list BRANCH-PREFIX seq 20 permit 10.6.0.0/16
ip prefix-list BRANCH-PREFIX seq 30 permit 10.7.0.0/16
ip prefix-list BRANCH-PREFIX seq 40 permit 10.9.0.0/16
!
ip prefix-list DEFAULT-ROUTE seq 10 permit 0.0.0.0/0
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list LOCALDC-PREFIX seq 10 permit 10.4.200.0/24
ip prefix-list LOCALDC-PREFIX seq 20 permit 10.4.201.0/24
ip prefix-list LOCALDC-PREFIX seq 30 permit 10.66.0.0/16
ip prefix-list LOCALDC-PREFIX seq 40 permit 10.88.0.0/16
!
ip prefix-list LOCALMCLOOPBACK seq 10 permit 10.88.88.100/32
!
ip prefix-list TUNNEL-DMVPN seq 10 permit 10.13.0.0/16
ip prefix-list TUNNEL-DMVPN seq 20 permit 10.23.0.0/16
ip prefix-list TUNNEL-DMVPN seq 30 permit 10.33.0.0/16
ip prefix-list TUNNEL-DMVPN seq 40 permit 10.43.0.0/16
ip prefix-list TUNNEL-DMVPN seq 50 permit 10.60.0.0/16
!
!
!--------------------------------------------------------------------
! ROUTE MAP
!--------------------------------------------------------------------
!
route-map INET4G-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 560
 set community 10:501
!
route-map INET2-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 570
 set community 10:401
!
route-map MPLS2-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 590
 set community 10:201
!
route-map WAN-IN deny 10
 description All Blocked Prefixes to come IN on BGP
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK TUNNEL-DMVPN
!
route-map WAN-IN permit 1000
 description Allow Everything Else
!
route-map REDIST-BGP-TO-OSPF permit 10
 match community POP1-SPOKES
 set metric 2200
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF permit 20
 match community POP3-SPOKES
 set metric 3200
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF deny 30
 description Block Null routes to be distributed from BGP to OSPF
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX ENTERPRISE-PREFIX
!
route-map REDIST-BGP-TO-OSPF permit 1000
 set metric 1200
 set metric-type type-1
!

Notes:

  • All spokes are iBGP peers
  • An outbound route-map is used to tag BGP announcements to the spokes with a local preference


R31 (DC-POP3-BR1) Overlay routing Configuration:


!--------------------------------------------------------------------
! OSPF
!--------------------------------------------------------------------
!
router ospf 100
 router-id 10.188.88.220
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface GigabitEthernet0/0/0
 network 10.0.0.0 0.255.255.255 area 0
!
!
!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.188.88.220
 bgp log-neighbor-changes
 bgp listen range 10.33.0.0/16 peer-group INET1-SPOKES
 bgp listen range 10.13.0.0/16 peer-group MPLS1-SPOKES
 bgp listen limit 2100
 neighbor MPLS1-SPOKES peer-group
 neighbor MPLS1-SPOKES remote-as 10
 neighbor MPLS1-SPOKES description MPLS1 Spoke Route Reflector
 neighbor MPLS1-SPOKES update-source Tunnel100
 neighbor MPLS1-SPOKES timers 20 60
 neighbor INET1-SPOKES peer-group
 neighbor INET1-SPOKES remote-as 10
 neighbor INET1-SPOKES description INET1 Spoke Route Reflector
 neighbor INET1-SPOKES update-source Tunnel300
 neighbor INET1-SPOKES timers 20 60
 !
 address-family ipv4
  bgp redistribute-internal
  network 0.0.0.0
  network 10.0.0.0
  network 10.4.200.0 mask 255.255.255.0
  network 10.4.201.0 mask 255.255.255.0
  network 10.5.0.0 mask 255.255.0.0
  network 10.6.0.0 mask 255.255.0.0
  network 10.7.0.0 mask 255.255.0.0
  network 10.9.0.0 mask 255.255.0.0
  network 10.99.0.0 mask 255.255.0.0
  network 10.188.0.0 mask 255.255.0.0
  network 10.188.88.200 mask 255.255.255.255
  network 10.188.88.220 mask 255.255.255.255
  neighbor MPLS1-SPOKES activate
  neighbor MPLS1-SPOKES send-community
  neighbor MPLS1-SPOKES route-reflector-client
  neighbor MPLS1-SPOKES next-hop-self all
  neighbor MPLS1-SPOKES weight 50000
  neighbor MPLS1-SPOKES soft-reconfiguration inbound
  neighbor MPLS1-SPOKES route-map WAN-IN in
  neighbor MPLS1-SPOKES route-map MPLS1-OUT out
  neighbor INET1-SPOKES activate
  neighbor INET1-SPOKES send-community
  neighbor INET1-SPOKES route-reflector-client
  neighbor INET1-SPOKES next-hop-self all
  neighbor INET1-SPOKES weight 45000
  neighbor INET1-SPOKES soft-reconfiguration inbound
  neighbor INET1-SPOKES route-map WAN-IN in
  neighbor INET1-SPOKES route-map INET1-OUT out
  maximum-secondary-paths ibgp 1
  distance bgp 201 89 89
 exit-address-family
!
!
ip bgp-community new-format
!
ip community-list standard POP1-SPOKES permit 10:10
ip community-list standard POP2-SPOKES permit 10:20
ip community-list standard POP3-SPOKES permit 10:30
!
!--------------------------------------------------------------------
! PREFIX-LIST
!--------------------------------------------------------------------
!
ip prefix-list BRANCH-PREFIX seq 10 permit 10.5.0.0/16
ip prefix-list BRANCH-PREFIX seq 20 permit 10.6.0.0/16
ip prefix-list BRANCH-PREFIX seq 30 permit 10.7.0.0/16
ip prefix-list BRANCH-PREFIX seq 40 permit 10.9.0.0/16
!
ip prefix-list DEFAULT-ROUTE seq 10 permit 0.0.0.0/0
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list LOCALDC-PREFIX seq 10 permit 10.4.200.0/24
ip prefix-list LOCALDC-PREFIX seq 20 permit 10.4.201.0/24
ip prefix-list LOCALDC-PREFIX seq 30 permit 10.99.0.0/16
ip prefix-list LOCALDC-PREFIX seq 40 permit 10.188.0.0/16
!
ip prefix-list LOCALMCLOOPBACK seq 10 permit 10.188.88.200/32
!
ip prefix-list LOCALBRLOOPBACK seq 10 permit 10.188.88.220/32
!
ip prefix-list TUNNEL-DMVPN seq 10 permit 10.13.0.0/16
ip prefix-list TUNNEL-DMVPN seq 20 permit 10.23.0.0/16
ip prefix-list TUNNEL-DMVPN seq 30 permit 10.33.0.0/16
ip prefix-list TUNNEL-DMVPN seq 40 permit 10.43.0.0/16
ip prefix-list TUNNEL-DMVPN seq 50 permit 10.60.0.0/16
!
!--------------------------------------------------------------------
! ROUTE MAP
!--------------------------------------------------------------------
!
route-map INET1-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 380
 set community 10:302
!
route-map MPLS1-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX LOCALBRLOOPBACK
 set local-preference 400
 set community 10:102
!
route-map WAN-IN deny 10
 description All Blocked Prefixes to come IN on BGP
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK TUNNEL-DMVPN
!
route-map WAN-IN permit 1000
 description Allow Everything Else
!
route-map REDIST-BGP-TO-OSPF permit 10
 match community POP1-SPOKES
 set metric 3000
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF permit 20
 match community POP2-SPOKES
 set metric 3000
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF deny 30
 description Block Null routes to be distributed from BGP to OSPF
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX ENTERPRISE-PREFIX
!
route-map REDIST-BGP-TO-OSPF permit 1000
 set metric 1000
 set metric-type type-1
!

Notes:

  • All spokes are iBGP peers
  • R31 advertises datacenter prefix summaries, enterprise network summary and default route
  • An outbound route-map is used to tag BGP announcements to the spokes a local preference. R11 has the highest local preference and is the preferred hub.


R32 (DC-POP3-BR2) Overlay routing Configuration:

!--------------------------------------------------------------------
! OSPF
!--------------------------------------------------------------------
!
router ospf 100
 router-id 10.188.88.230
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface GigabitEthernet0/0/0
 network 10.0.0.0 0.255.255.255 area 0
!
!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!

router bgp 10
 bgp router-id 10.188.88.230
 bgp log-neighbor-changes
 bgp listen range 10.60.0.0/16 peer-group INET4G-SPOKES
 bgp listen range 10.43.0.0/16 peer-group INET2-SPOKES
 bgp listen range 10.23.0.0/16 peer-group MPLS2-SPOKES
 bgp listen limit 2100
 neighbor INET2-SPOKES peer-group
 neighbor INET2-SPOKES remote-as 10
 neighbor INET2-SPOKES description INET2 Spoke Route Reflector
 neighbor INET2-SPOKES update-source Tunnel400
 neighbor INET2-SPOKES timers 20 60
 neighbor MPLS2-SPOKES peer-group
 neighbor MPLS2-SPOKES remote-as 10
 neighbor MPLS2-SPOKES description MPLS2 Spoke Route Reflector
 neighbor MPLS2-SPOKES update-source Tunnel200
 neighbor MPLS2-SPOKES timers 20 60
 neighbor INET4G-SPOKES peer-group
 neighbor INET4G-SPOKES remote-as 10
 neighbor INET4G-SPOKES description INET4G Spoke Route Reflector
 neighbor INET4G-SPOKES update-source Tunnel500
 neighbor INET4G-SPOKES timers 20 60
 !
 address-family ipv4
  bgp redistribute-internal
  network 0.0.0.0
  network 10.0.0.0
  network 10.4.200.0 mask 255.255.255.0
  network 10.4.201.0 mask 255.255.255.0
  network 10.5.0.0 mask 255.255.0.0
  network 10.6.0.0 mask 255.255.0.0
  network 10.7.0.0 mask 255.255.0.0
  network 10.9.0.0 mask 255.255.0.0
  network 10.99.0.0 mask 255.255.0.0
  network 10.188.0.0 mask 255.255.0.0
  network 10.188.88.200 mask 255.255.255.255
  neighbor INET2-SPOKES activate
  neighbor INET2-SPOKES send-community
  neighbor INET2-SPOKES route-reflector-client
  neighbor INET2-SPOKES next-hop-self all
  neighbor INET2-SPOKES weight 45000
  neighbor INET2-SPOKES soft-reconfiguration inbound
  neighbor INET2-SPOKES route-map WAN-IN in
  neighbor INET2-SPOKES route-map INET2-OUT out
  neighbor MPLS2-SPOKES activate
  neighbor MPLS2-SPOKES send-community
  neighbor MPLS2-SPOKES route-reflector-client
  neighbor MPLS2-SPOKES next-hop-self all
  neighbor MPLS2-SPOKES weight 50000
  neighbor MPLS2-SPOKES soft-reconfiguration inbound
  neighbor MPLS2-SPOKES route-map WAN-IN in
  neighbor MPLS2-SPOKES route-map MPLS2-OUT out
  neighbor INET4G-SPOKES activate
  neighbor INET4G-SPOKES send-community
  neighbor INET4G-SPOKES route-reflector-client
  neighbor INET4G-SPOKES next-hop-self all
  neighbor INET4G-SPOKES weight 40000
  neighbor INET4G-SPOKES soft-reconfiguration inbound
  neighbor INET4G-SPOKES route-map WAN-IN in
  neighbor INET4G-SPOKES route-map INET4G-OUT out
  maximum-secondary-paths ibgp 2
  distance bgp 201 89 89
 exit-address-family
!
ip bgp-community new-format
!
ip community-list standard POP1-SPOKES permit 10:10
ip community-list standard POP2-SPOKES permit 10:20
ip community-list standard POP3-SPOKES permit 10:30
!
!--------------------------------------------------------------------
! PREFIX LIST
!--------------------------------------------------------------------
!
ip prefix-list BRANCH-PREFIX seq 10 permit 10.5.0.0/16
ip prefix-list BRANCH-PREFIX seq 20 permit 10.6.0.0/16
ip prefix-list BRANCH-PREFIX seq 30 permit 10.7.0.0/16
ip prefix-list BRANCH-PREFIX seq 40 permit 10.9.0.0/16
!
ip prefix-list DEFAULT-ROUTE seq 10 permit 0.0.0.0/0
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list LOCALDC-PREFIX seq 10 permit 10.4.200.0/24
ip prefix-list LOCALDC-PREFIX seq 20 permit 10.4.201.0/24
ip prefix-list LOCALDC-PREFIX seq 30 permit 10.99.0.0/16
ip prefix-list LOCALDC-PREFIX seq 40 permit 10.188.0.0/16
!
ip prefix-list LOCALMCLOOPBACK seq 10 permit 10.188.88.200/32
!
ip prefix-list TUNNEL-DMVPN seq 10 permit 10.13.0.0/16
ip prefix-list TUNNEL-DMVPN seq 20 permit 10.23.0.0/16
ip prefix-list TUNNEL-DMVPN seq 30 permit 10.33.0.0/16
ip prefix-list TUNNEL-DMVPN seq 40 permit 10.43.0.0/16
ip prefix-list TUNNEL-DMVPN seq 50 permit 10.60.0.0/16
!
!
!--------------------------------------------------------------------
! ROUTE MAP
!--------------------------------------------------------------------
!
route-map INET4G-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 360
 set community 10:502
!
route-map INET2-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 370
 set community 10:402
!
route-map MPLS2-OUT permit 10
 description All Allowed Prefixes to Go OUT on BGP to Spokes
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK BRANCH-PREFIX
 set local-preference 390
 set community 10:202
!
route-map WAN-IN deny 10
 description All Blocked Prefixes to come IN on BGP
 match ip address prefix-list DEFAULT-ROUTE ENTERPRISE-PREFIX LOCALDC-PREFIX LOCALMCLOOPBACK TUNNEL-DMVPN
!
route-map WAN-IN permit 1000
 description Allow Everything Else
!
route-map REDIST-BGP-TO-OSPF permit 10
 match community POP1-SPOKES
 set metric 3200
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF permit 20
 match community POP2-SPOKES
 set metric 3200
 set metric-type type-1
!
route-map REDIST-BGP-TO-OSPF deny 30
 description Block Null routes to be distributed from BGP to OSPF
 match ip address prefix-list DEFAULT-ROUTE BRANCH-PREFIX LOCALDC-PREFIX ENTERPRISE-PREFIX
!
route-map REDIST-BGP-TO-OSPF permit 1000
 set metric 1200
 set metric-type type-1
!

Notes:

  • All spokes are iBGP peers
  • An outbound route-map is used to tag BGP announcements to the spokes with a local preference


Overlay routing at Single CPE Branch - Configuration

The following example demonstrates a single router spoke site with one interface connected to Internet transport and the other to MPLS, each on different DMVPN clouds. The spoke router is a BGP Route Reflector client peering to a redundant pair of route-reflectors in each DMVPN cloud. We make sure to have a priority order for the next hop. To achieve this, Hub BGP peers set local-preference.


R41 Spoke Configuration:

!--------------------------------------------------------------------
! BGP
!--------------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.5.28.1
 bgp log-neighbor-changes
 neighbor MPLS1-HUB peer-group
 neighbor MPLS1-HUB remote-as 10
 neighbor MPLS1-HUB description To IWAN MPLS1 Hub Router
 neighbor MPLS1-HUB update-source Tunnel100
 neighbor MPLS1-HUB timers 20 60
 neighbor INET1-HUB peer-group
 neighbor INET1-HUB remote-as 10
 neighbor INET1-HUB description To IWAN INET1 Hub Router
 neighbor INET1-HUB update-source Tunnel300
 neighbor INET1-HUB timers 20 60
 neighbor INET4G-HUB peer-group
 neighbor INET4G-HUB remote-as 10
 neighbor INET4G-HUB description To IWAN INET4G Hub Router
 neighbor INET4G-HUB update-source Tunnel500
 neighbor INET4G-HUB timers 20 60
 neighbor 10.13.1.1 peer-group MPLS1-HUB
 neighbor 10.13.1.2 peer-group MPLS1-HUB
 neighbor 10.13.1.3 peer-group MPLS1-HUB
 neighbor 10.33.1.1 peer-group INET1-HUB
 neighbor 10.33.1.2 peer-group INET1-HUB
 neighbor 10.33.1.3 peer-group INET1-HUB
 neighbor 10.60.1.1 peer-group INET4G-HUB
 neighbor 10.60.1.2 peer-group INET4G-HUB
 neighbor 10.60.1.3 peer-group INET4G-HUB
 !
 address-family ipv4
  redistribute connected
  neighbor MPLS1-HUB send-community
  neighbor MPLS1-HUB next-hop-self all
  neighbor MPLS1-HUB weight 50000
  neighbor MPLS1-HUB soft-reconfiguration inbound
  neighbor MPLS1-HUB route-map POP-SELECT in
  neighbor MPLS1-HUB route-map SPOKE-OUT out
  neighbor INET1-HUB send-community
  neighbor INET1-HUB next-hop-self all
  neighbor INET1-HUB weight 50000
  neighbor INET1-HUB soft-reconfiguration inbound
  neighbor INET1-HUB route-map POP-SELECT in
  neighbor INET1-HUB route-map SPOKE-OUT out
  neighbor INET4G-HUB send-community
  neighbor INET4G-HUB next-hop-self all
  neighbor INET4G-HUB weight 50000
  neighbor INET4G-HUB soft-reconfiguration inbound
  neighbor INET4G-HUB route-map POP-SELECT in
  neighbor INET4G-HUB route-map SPOKE-OUT out
  neighbor 10.13.1.1 activate
  neighbor 10.13.1.2 activate
  neighbor 10.13.1.3 activate
  neighbor 10.33.1.1 activate
  neighbor 10.33.1.2 activate
  neighbor 10.33.1.3 activate
  neighbor 10.60.1.1 activate
  neighbor 10.60.1.2 activate
  neighbor 10.60.1.3 activate
  distance bgp 201 89 89
 exit-address-family
!
ip bgp-community new-format
!
ip community-list standard POP1-MPLS1 permit 10:100
ip community-list standard POP1-INET1 permit 10:300
ip community-list standard POP1-INET4G permit 10:500
ip community-list standard POP2-MPLS1 permit 10:101
ip community-list standard POP2-INET1 permit 10:301
ip community-list standard POP2-INET4G permit 10:501
ip community-list standard POP3-MPLS1 permit 10:102
ip community-list standard POP3-INET1 permit 10:302
ip community-list standard POP3-INET4G permit 10:502
!
!--------------------------------------------------------------------
! PREFIX LIST
!--------------------------------------------------------------------
!
ip prefix-list LOCAL-NETS seq 10 permit 10.5.28.1/32
ip prefix-list LOCAL-NETS seq 20 permit 10.5.29.0/24
ip prefix-list LOCAL-NETS seq 30 permit 10.5.30.0/24
!
ip prefix-list UTAH seq 10 permit 10.5.28.0/24
ip prefix-list UTAH seq 20 permit 10.5.29.0/24
ip prefix-list UTAH seq 30 permit 10.5.30.0/24
!
!
!--------------------------------------------------------------------
! ROUTE MAPS
!--------------------------------------------------------------------
!
route-map SPOKE-OUT permit 10
 match ip address prefix-list LOCAL-NETS
 set community 10:10
!
!
!
route-map POP-SELECT permit 100
 match community POP1-MPLS1
 set local-preference 800
!
route-map POP-SELECT permit 120
 match community POP1-INET1
 set local-preference 780
!
route-map POP-SELECT permit 140
 match community POP1-INET4G
 set local-preference 760
!
route-map POP-SELECT permit 200
 match community POP2-MPLS1
 set local-preference 600
!
route-map POP-SELECT permit 220
 match community POP2-INET1
 set local-preference 580
!
route-map POP-SELECT permit 240
 match community POP2-INET4G
 set local-preference 560
!
route-map POP-SELECT permit 300
 match community POP3-MPLS1
 set local-preference 400
!
route-map POP-SELECT permit 320
 match community POP3-INET1
 set local-preference 380
!
route-map POP-SELECT permit 340
 match community POP3-INET4G
 set local-preference 360
!
route-map POP-SELECT permit 1000
!


NOTE: The Overlay routing configs would be similar to "R41" for "R61" and "R71"


Overlay routing at Dual CPE Branch - Configuration

The following example demonstrates a dual router spoke site with one router connected to Internet transport and the other to MPLS, each on different DMVPN clouds. BGP peering to a redundant pair of route-reflectors in each DMVPN cloud is shown. Each spoke router runs OSPF to a Layer 3 LAN switch, where it learns the spoke site routes and sends redistributed WAN routes.

Spoke routers are required originate their local site routes into BGP which include connected router interfaces (LAN and loopback) and any site LAN prefixes that are learned by OSPF from a Layer 3 LAN switch or router. BGP route origination can be accomplished by redistribution of OSPF routes, using route-maps that permit only what is necessary. Note that the configuration is similar to the use we configured on single CPE branch. This has two main benefits:

  • This gives a more generic configuration and remove the specifics of the site, thus allowing an easier deployment.
  • configuration can be easily deployed with templates in Cisco Prime.
  • A single CPE branch can be easily extended to become a dual CPE branch without configuration changes.

The configuration on R51 is the following:

!------------------------------------------------------------
! ROUTER OSPF
!------------------------------------------------------------
!
router ospf 100
 router-id 10.5.8.1
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface GigabitEthernet0/0/1.99
 network 10.0.0.0 0.255.255.255 area 0
 default-information originate
!
!------------------------------------------------------------
! BGP
!------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.5.8.1
 bgp log-neighbor-changes
 neighbor MPLS1-HUB peer-group
 neighbor MPLS1-HUB remote-as 10
 neighbor MPLS1-HUB description To IWAN MPLS1 Hub Router
 neighbor MPLS1-HUB update-source Tunnel100
 neighbor MPLS1-HUB timers 20 60
 neighbor INET1-HUB peer-group
 neighbor INET1-HUB remote-as 10
 neighbor INET1-HUB description To IWAN INET1 Hub Router
 neighbor INET1-HUB update-source Tunnel300
 neighbor INET1-HUB timers 20 60
 neighbor 10.13.1.1 peer-group MPLS1-HUB
 neighbor 10.13.1.2 peer-group MPLS1-HUB
 neighbor 10.13.1.3 peer-group MPLS1-HUB
 neighbor 10.33.1.1 peer-group INET1-HUB
 neighbor 10.33.1.2 peer-group INET1-HUB
 neighbor 10.33.1.3 peer-group INET1-HUB
 !
 address-family ipv4
  bgp redistribute-internal
  redistribute connected
  redistribute ospf 100 route-map REDIST-OSPF-TO-BGP
  neighbor MPLS1-HUB send-community
  neighbor MPLS1-HUB next-hop-self all
  neighbor MPLS1-HUB weight 50000
  neighbor MPLS1-HUB soft-reconfiguration inbound
  neighbor MPLS1-HUB route-map POP-SELECT in
  neighbor MPLS1-HUB route-map SPOKE-OUT out
  neighbor INET1-HUB send-community
  neighbor INET1-HUB next-hop-self all
  neighbor INET1-HUB weight 50000
  neighbor INET1-HUB soft-reconfiguration inbound
  neighbor INET1-HUB route-map POP-SELECT in
  neighbor INET1-HUB route-map SPOKE-OUT out
  neighbor 10.13.1.1 activate
  neighbor 10.13.1.2 activate
  neighbor 10.13.1.3 activate
  neighbor 10.33.1.1 activate
  neighbor 10.33.1.2 activate
  neighbor 10.33.1.3 activate
  distance bgp 201 89 89
 exit-address-family
!
ip bgp-community new-format
!
ip community-list standard POP1-MPLS1 permit 10:100
ip community-list standard POP2-MPLS1 permit 10:101
ip community-list standard POP3-MPLS1 permit 10:102
ip community-list standard POP1-INET1 permit 10:300
ip community-list standard POP2-INET1 permit 10:301
ip community-list standard POP3-INET1 permit 10:302
!
!------------------------------------------------------------
! PREFIX LIST
!------------------------------------------------------------
!
ip prefix-list LOCAL-NETS seq 10 permit 10.5.8.1/32
ip prefix-list LOCAL-NETS seq 20 permit 10.5.9.0/24
ip prefix-list LOCAL-NETS seq 30 permit 10.5.10.0/24
ip prefix-list LOCAL-NETS seq 40 permit 10.5.8.129/32
!
ip prefix-list OVLD seq 10 permit 10.5.8.0/24
ip prefix-list OVLD seq 20 permit 10.5.9.0/24
ip prefix-list OVLD seq 30 permit 10.5.10.0/24
!
!
!------------------------------------------------------------
! ROUTE MAP
!------------------------------------------------------------
!
route-map SPOKE-OUT permit 10
 match ip address prefix-list LOCAL-NETS
 set community 10:10
!
route-map POP-SELECT permit 100
 match community POP1-MPLS1
 set local-preference 800
!
route-map POP-SELECT permit 110
 match community POP1-INET1
 set local-preference 780
!
route-map POP-SELECT permit 120
 match community POP2-MPLS1
 set local-preference 600
!
route-map POP-SELECT permit 130
 match community POP2-INET1
 set local-preference 580
!
route-map POP-SELECT permit 140
 match community POP3-MPLS1
 set local-preference 400
!
route-map POP-SELECT permit 170
 match community POP3-INET1
 set local-preference 380
!
route-map POP-SELECT permit 1000
!
route-map REDIST-BGP-TO-OSPF permit 10
 description Set a route tag to identify routes redistributed from BGP
 set tag 1
!
route-map REDIST-OSPF-TO-BGP deny 10
 description Block all routes redistributed from BGP
 match tag 1
!
route-map REDIST-OSPF-TO-BGP permit 20
 description Redistribute all other traffic
 match route-type internal
 match route-type external type-1
 match route-type external type-2
!


The configuration on R52 is the following:

!------------------------------------------------------------
! ROUTER OSPF
!------------------------------------------------------------
!
router ospf 100
 router-id 10.5.8.129
 redistribute bgp 10 subnets route-map REDIST-BGP-TO-OSPF
 passive-interface default
 no passive-interface GigabitEthernet0/0/1.99
 network 10.0.0.0 0.255.255.255 area 0
 default-information originate
!
!
!------------------------------------------------------------
! BGP
!------------------------------------------------------------
!
router bgp 10
 bgp router-id 10.5.8.129
 bgp log-neighbor-changes
 neighbor MPLS2-HUB peer-group
 neighbor MPLS2-HUB remote-as 10
 neighbor MPLS2-HUB description To IWAN MPLS2 Hub Router
 neighbor MPLS2-HUB update-source Tunnel200
 neighbor MPLS2-HUB timers 20 60
 neighbor INET4G-HUB peer-group
 neighbor INET4G-HUB remote-as 10
 neighbor INET4G-HUB description To IWAN INET4G Hub Router
 neighbor INET4G-HUB update-source Tunnel500
 neighbor INET4G-HUB timers 20 60
 neighbor INET2-HUB peer-group
 neighbor INET2-HUB remote-as 10
 neighbor INET2-HUB description To IWAN INET2 Hub Router
 neighbor INET2-HUB update-source Tunnel400
 neighbor INET2-HUB timers 20 60
 neighbor 10.23.1.1 peer-group MPLS2-HUB
 neighbor 10.23.1.2 peer-group MPLS2-HUB
 neighbor 10.23.1.3 peer-group MPLS2-HUB
 neighbor 10.43.1.1 peer-group INET2-HUB
 neighbor 10.43.1.2 peer-group INET2-HUB
 neighbor 10.43.1.3 peer-group INET2-HUB
 neighbor 10.60.1.1 peer-group INET4G-HUB
 neighbor 10.60.1.2 peer-group INET4G-HUB
 neighbor 10.60.1.3 peer-group INET4G-HUB
 !
 address-family ipv4
  bgp redistribute-internal
  redistribute connected
  redistribute ospf 100 route-map REDIST-OSPF-TO-BGP
  neighbor MPLS2-HUB send-community
  neighbor MPLS2-HUB next-hop-self all
  neighbor MPLS2-HUB weight 50000
  neighbor MPLS2-HUB soft-reconfiguration inbound
  neighbor MPLS2-HUB route-map POP-SELECT in
  neighbor MPLS2-HUB route-map SPOKE-OUT out
  neighbor INET4G-HUB send-community
  neighbor INET4G-HUB next-hop-self all
  neighbor INET4G-HUB weight 50000
  neighbor INET4G-HUB soft-reconfiguration inbound
  neighbor INET4G-HUB route-map POP-SELECT in
  neighbor INET4G-HUB route-map SPOKE-OUT out
  neighbor INET2-HUB send-community
  neighbor INET2-HUB next-hop-self all
  neighbor INET2-HUB weight 50000
  neighbor INET2-HUB soft-reconfiguration inbound
  neighbor INET2-HUB route-map POP-SELECT in
  neighbor INET2-HUB route-map SPOKE-OUT out
  neighbor 10.23.1.1 activate
  neighbor 10.23.1.2 activate
  neighbor 10.23.1.3 activate
  neighbor 10.43.1.1 activate
  neighbor 10.43.1.2 activate
  neighbor 10.43.1.3 activate
  neighbor 10.60.1.1 activate
  neighbor 10.60.1.2 activate
  neighbor 10.60.1.3 activate
  distance bgp 201 89 89
 exit-address-family
!
ip bgp-community new-format
!
ip community-list standard POP1-MPLS2 permit 10:200
ip community-list standard POP1-INET4G permit 10:500
ip community-list standard POP2-MPLS2 permit 10:201
ip community-list standard POP2-INET4G permit 10:501
ip community-list standard POP3-MPLS2 permit 10:202
ip community-list standard POP3-INET4G permit 10:502
ip community-list standard POP1-INET2 permit 10:400
ip community-list standard POP2-INET2 permit 10:401
ip community-list standard POP3-INET2 permit 10:402
!
!------------------------------------------------------------
! PREFIX LIST
!------------------------------------------------------------
!
ip prefix-list LOCAL-NETS seq 10 permit 10.5.8.1/32
ip prefix-list LOCAL-NETS seq 20 permit 10.5.9.0/24
ip prefix-list LOCAL-NETS seq 30 permit 10.5.10.0/24
ip prefix-list LOCAL-NETS seq 40 permit 10.5.8.129/32
!
!------------------------------------------------------------
! ROUTE MAP
!------------------------------------------------------------
!
route-map SPOKE-OUT permit 10
 match ip address prefix-list LOCAL-NETS
 set community 10:10
!
!
route-map POP-SELECT permit 100
 match community POP1-MPLS2
 set local-preference 790
!
route-map POP-SELECT permit 110
 match community POP1-INET2
 set local-preference 770
!
route-map POP-SELECT permit 120
 match community POP1-INET4G
 set local-preference 760
!
route-map POP-SELECT permit 130
 match community POP2-MPLS2
 set local-preference 590
!
route-map POP-SELECT permit 140
 match community POP2-INET2
 set local-preference 570
!
route-map POP-SELECT permit 150
 match community POP2-INET4G
 set local-preference 560
!
route-map POP-SELECT permit 160
 match community POP3-MPLS2
 set local-preference 390
!
route-map POP-SELECT permit 170
 match community POP3-INET2
 set local-preference 370
!
route-map POP-SELECT permit 180
 match community POP3-INET4G
 set local-preference 360
!
route-map POP-SELECT permit 1000
!
route-map REDIST-BGP-TO-OSPF permit 10
 description Set a route tag to identify routes redistributed from BGP
 set tag 1
!
route-map REDIST-OSPF-TO-BGP deny 10
 description Block all routes redistributed from BGP
 match tag 1
!
route-map REDIST-OSPF-TO-BGP permit 20
 description Redistribute all other traffic
 match route-type internal
 match route-type external type-1
 match route-type external type-2
!



PfR Configuration

IWAN Sites

An IWAN domain includes a mandatory Hub site, optional Transit sites, as well as Branch sites. Each site has a unique identifier called a Site-Id that is derived from the loopback address of the local MC.

Branch Sites

  • These will always be a DMVPN spoke, and are a stub site where traffic transit is not allowed.
  • The local MC peers with the logical domain controller (aka Hub MC) to get its policies, and monitoring guidelines.

Transit Sites

  • Located in an enterprise central site or headquarter location
  • Can act as a transit site to access servers in the datacenters or for spoke-to-spoke traffic.
  • Datacenters may or may not collocated with the transit site
  • A POP Identifier (POP-ID) is configured for each transit site. This POP-ID has to be unique in the domain.
  • The local MC peers with the Hub MC (aka Domain Controller) to get its policies, monitor configuration and timers.

Hub Site

  • The logical domain controller functionality resides on this site’s master controller (MC).
  • Only one Hub site exists per IWAN domain because of the uniqueness of the logical domain controller’s presence. The master controller (MC) for this site is known as the Hub master controller (Hub MC); thereby making this site the Hub site.
  • MCs from all other sites (transit or branch) connect to the Hub MC for PfR configuration and policies.
  • A POP Identifier (POP-ID) 0 is automatically assigned to a Hub site.
  • Can contain all other properties of a Transit site ad defined above.


Device Components and Role

PfR is comprised of two major Cisco IOS components, a Master Controller (MC) and a Border Router (BR). The MC is a policy decision point at which policies are defined and applied to various traffic classes that traverse the BR systems. The MC can be configured to learn and control traffic classes on the network:

  • Border Routers (BRs) are in the data forwarding path. BRs collect data from their Performance Monitor cache and smart probe results, provide a degree of aggregation of this information, and influence the packet forwarding path as directed by the site local MC to manage user traffic.
  • Master Controller (MC) is the policy decision maker. At a large site, such as data center or campus, the MC is a standalone chassis. For smaller branch locations, the MC is typically collocated (configured) on the same platform as the border router. As a general rule, the large locations manage more network prefixes and applications than a branch deployment, thus consuming more CPU and memory resources for the master controller function. Therefore, it makes a good design practice to dedicate a chassis for the master controller at large sites.

Each site in the PfR Domain must include a local MC and at least one BR.

There are five different roles a device can be in an IWAN domain:

  • Hub Master Controller (Hub MC) – This is the MC at the hub site that acts as MC for the site and makes optimization decision for that site and provides the Path Control policies for all the other MCs. The Hub MC contains the logical PfR domain controller role.
  • Transit Master Controller (Transit MC) – This is the MC at transit sites and makes optimization decision for that site. There is no policy configuration on Transit MCs because they receive their policy from the Hub MC.
  • Branch Master Controller (Branch MC) - The branch MC is the MC for branch sites, and makes optimization decisions for that branch site.. There is no policy configuration on branch MCs because they receive their policy from the Hub MC.
  • Transit Border Router (Transit BR) - The border controller at a Hub or Transit site. WAN interface terminates in the BRs. PfR is enabled on these interfaces. At the time of this writing, only one WAN interface is supported on a BR. This limitation is overcome by using multiple BRs devices.


Hub/DC Master Controller - "DC-POP1-MC" (R10)

The Hub Master Controller (Hub MC) is located at the hub site in the Intelligent WAN (IWAN) topology. R10 is the Hub MC for the guide’s sample topology. R10 controls two BRs in Site1: R11 and R12.

It is also essential to remember that the Hub MC actually supports two very different roles:

  • It is a Local MC for the site and as such peers with the local BRs and controls them. For that role, it is similar to a Transit MC.
  • It is a Global Domain Controller with the PfR policies for the entire IWAN domain and as such peers with all MCs in the domain.

The Hub MC should run as a standalone platform, on a physical device or as a virtual machine (CSR1000v)

Configuring Master Controller (MC) for hub includes the following:

  • Define the IWAN Domain Name
  • Define the VRF
  • Identify the Router as the Hub MC
  • Define the Source Interface for PfR Communication
  • Define Password (Optional)
  • Define PfR domain policies - this will be explained in the PfR policies section.

The example below illustrates the configuration of R10, Hub MC for the domain:

domain iwan
 vrf default
  master hub
   source-interface Loopback0
   site-prefixes prefix-list POP1-PREFIXES
   password 7 05080F1C2243
   load-balance advanced
    path-preference INET1 INET2 fallback MPLS1 next-fallback MPLS2
   enterprise-prefix  prefix-list ENTERPRISE-PREFIX
   collector 10.4.200.5 port 2055
!
ip prefix-list ENTERPRISE-PREFIX seq 10 permit 10.0.0.0/8
!
ip prefix-list POP1-PREFIXES seq 10 permit 10.4.200.0/24
ip prefix-list POP1-PREFIXES seq 20 permit 10.4.201.0/24
ip prefix-list POP1-PREFIXES seq 30 permit 10.77.0.0/16
!

Notes:

  • Enterprise-prefix : the main use of the enterprise prefix list is to determine the enterprise boundary.
    • With enterprise-prefix: if a prefix doesn't match any site-prefix but matches enterprise-prefix then the prefix belongs to a site that is not participating in PfRv3 but it does belong to the enterprise. PfR will not influence traffic towards sites that has NOT enabled PFR.
    • Without enterprise-prefix: all the traffic that would be going towards a spoke that is NOT PfR enabled will be learnt as internet traffic class and therefore can only be subjected to load balancing. (Monitoring of packet loss, delay or jitter is not possible)
  • Site-prefix : It is a set of prefixes that belongs to particular site. This allows configuring site-prefix manually instead of dynamic learning. This configuration should be used at the site if the site is used for transit (Hub and Transit sites). For example, Site A reaches Site B via Hub-Site, where Hub-Site is transit site. The configuration is used to prevent learning of Site A prefix as Hub-Site prefix when it is transiting from Hub.
  • Use of le/ge or "deny" is not supported with Site-prefix or Enterprise-prefix list
  • Netflow v9 records can be exported to a Netflow collector for reporting. Applications like Cisco Prime Infrastructure, LiveAction or LivingObjects can use this information. More information on Netflow export on the PfR Reporting Page


Hub/DC Border Routers - "DC-POP1-BR1" and "DC-POP1-BR2" (R11 and R12)

A Hub Border Router is a border controller at the hub-site. This is the device where WAN interface terminates. PfR is enabled on these interfaces. There can be one or more Hub BRs per DMVPN transport (Hub/Transit site only) for horizontal scaling. There is only one transport per Hub BR. The WAN tunnel interfaces are not automatically discovered and are manually configured.

On the Hub Border Routers, PfR must be configured with:

  • The address of the local MC
  • The path name on external interfaces
  • The path identifier on external interfaces. This identifier must be unique per site.

The border routers on Site1 register to their local MC (R10) with their external interface definition together with their path name and identifier. You can use the global routing table (default VRF) or define specific VRFs for hub border routers.

The example below illustrates the configuration of R11, Hub BR for MPLS:

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master 10.8.88.10
   password 7 094F471A1A0A
   advanced
    monitor-cache-percent 80
!
!
interface tunnel100
 domain iwan path MPLS1 path-id 1
!
interface tunnel300
 domain iwan path INET1 path-id 3
!


The example below illustrates the configuration of R12, Hub BR for INET:

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master 10.8.88.10
   password 7 0822455D0A16
   advanced
    monitor-cache-percent 80
!
interface tunnel200
 domain IWAN path MPLS2 path-id 2
!
interface tunnel400
 domain iwan path INET2 path-id 4
!
interface tunnel500
 domain iwan path INET4G path-id 5 path-last-resort
!

This is a one-time configuration. Once done, all changes will be centralized on the DC-POP1 MC.


DC - Transit Master Controllers: "DC-POP2-TMC" (R20) and "DC-POP3-TMC" (R30)

The Transit Master Controller is located at all of the Transit sites in an IWAN topology. The Transit MC should run as a standalone platform, on a physical device or as a virtual machine (CSR1000v).

R20 is the Transit MC for the guide’s sample topology. R20 controls two BRs in Site2: R21 and R22. A Transit MC needs to peer with the Domain Controller (Hub MC) to get the policies, monitor configurations, and global parameters

Configuring Master Controller (MC) for Transit includes the following:

  • Define the IWAN Domain Name
  • Define the VRF
  • Identify the router as the Transit MC and define the POP-ID
  • Define the Source Interface for PfR Communication
  • Configure the Hub MC
  • Define Password (Optional)


The example below illustrates the configuration of R20, Transit MC for Site2/DC-POP2:

!
domain iwan
 vrf default
  master transit 1
   source-interface Loopback0
   site-prefixes prefix-list POP2-PREFIXES
   password 7 094F471A1A0A
   hub 10.8.88.10
!
ip prefix-list POP2-PREFIXES seq 10 permit 10.4.200.0/24
ip prefix-list POP2-PREFIXES seq 20 permit 10.4.201.0/24
ip prefix-list POP2-PREFIXES seq 30 permit 10.66.0.0/16
!
!

Notes:

  • Transit Master Controller is configured with a POP-ID that must be unique per domain. R20 configured with POP-ID of 1.
  • Transit MC peers with the Hub MC to get the policies and monitor configurations.
  • Site-prefix : It is a set of prefixes that belong to particular site. This allows configuring site-prefix manually instead of learning. This configuration must at the site if the site is used for transit. For example, Site A reaches Site B via Site2. The configuration is used to prevent learning of Site A prefix as Site2 prefix when it is transiting across Site2.


Similarly the below example illustrates the configuration of R30, Transit MC for Site3/DC-POP3:

!
domain iwan
 vrf default
  master transit 2
   source-interface Loopback0
   site-prefixes prefix-list POP3-PREFIXES
   password 7 104D000A0618
   hub 10.8.88.10
!
ip prefix-list POP3-PREFIXES seq 10 permit 10.4.200.0/24
ip prefix-list POP3-PREFIXES seq 20 permit 10.4.201.0/24
ip prefix-list POP3-PREFIXES seq 30 permit 10.99.0.0/16
!


Transit Border Routers (at DC-POP2 and DC-POP3)

A Transit Border Router is a Border Router at a Transit site. A Transit BR has exactly the same role and behaviour than a Hub BR. This is the device where WAN interface terminates on a central site or datacenter. PfR is enabled on these interfaces. There can be one or more Transit BRs per DMVPN transport (Hub/Transit site only) for horizontal scaling. There is only one transport per Transit BR. The WAN tunnel interfaces are not automatically discovered and are manually configured.

On a Transit Border Routers, PfR must be configured with:

  • The address of the local MC
  • The path name on external interfaces
  • The path identifier on external interfaces. Must be unique per site.

The Border Routers on a Transit Site register to the local MC with their external interface definition together with their path name and identifier. You can use the global routing table (default VRF) or define specific VRFs for hub border routers.

The example below illustrates the configuration of R21 (DC-POP2-BR1):

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master 10.88.88.100
   password 7 104D000A0618
   advanced
    monitor-cache-percent 80
!
interface tunnel100
 domain iwan path MPLS1 path-id 1
!
interface tunnel300
 domain iwan path INET1 path-id 3
!


The example below illustrates the configuration of R22 (DC-POP2-BR2):

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master 10.88.88.100
   password 7 070C285F4D06
   advanced
    monitor-cache-percent 80
!
interface tunnel200
  domain iwan path MPLS2 path-id 2
!
interface tunnel400
 domain iwan path INET2 path-id 4
!
interface tunnel500
 domain iwan path INET4G path-id 5 path-last-resort
!

NOTE: The configs on R31(DC-POP3-BR1) and R32(DC-POP3-BR2) is identical to R21(DC-POP2-BR1) and R22(DC-POP2-BR2) respectively.

Branch Routers

The MC at a branch site typically does not require a dedicated router to act as a MC. This implies that at least one Branch site router will contain the MC and the BR roles on it. If a Branch site has two routers for resiliency, the MC is typically connected to the primary transport circuit (MPLS in the guide’s examples), and is the HSRP master when the branch routers provide the Layer 2 / Layer 3 delineation (I.E. all interfaces facing the LAN are layer 2 only).

A branch MC receives the PfR policies, performance monitors instances (PMI) and global parameters from the Hub MC. The branch MC acts as master controller for that site for making optimization decision.

Configuring Master Controller (MC) for branch includes the following:

  • Define the IWAN Domain Name
  • Define the VRF
  • Identify the router as the Branch MC
  • Define the Source Interface for PfR Communication
  • Configure the Hub MC
  • Define Password (Optional)


The example below illustrates the configuration of a single CPE branches (R41, R61 and R71 where MC and BR are collocated on the same platform:

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master local
   password 7 00071A150754
   advanced
    monitor-cache-percent 80
  master branch
   source-interface Loopback0
   site-prefixes prefix-list UTAH
   password 7 060506324F41
   hub 10.8.88.10
!



The example below illustrates the configuration of a dual CPE branch (R51, R52), where MC and BR are collocated on the primary router (R51) and another BR is defined on R52:

R51 Configuration - Includes the MC definition (with the IP address of the Hub MC, ie R10) and the BR definition (using local MC):

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master local
   password 7 070C285F4D06
   advanced
    monitor-cache-percent 80
  master branch
   source-interface Loopback0
   site-prefixes prefix-list OVLD
   password 7 030752180500
   hub 10.8.88.10
!

R52 Configuration - Includes the BR definition that points to the site MC, ie R51:

!
domain iwan
 vrf default
  border
   source-interface Loopback0
   master 10.5.8.1
   password 7 094F471A1A0A
   advanced
    monitor-cache-percent 80
!


PfR Policies

PfR policies are rules to dictate how PfR should monitor and control traffic. PfR policies are global to the IWAN domain and are defined in Hub MC and then distributed to Branch and Transit MCs using the SAF infrastructure.

PfR policies include:

  • Administrative Policies. Specify constraints such as path preference or Transit Site preference. Critical applications or media applications will be forwarded over a primary path which is MPLS and only failover to the secondary path INET if performance is out of policy.
  • Performance Policies. Specify constraints such as delay, jitter and loss threshold. That defines the boundaries in which an application is correctly supported and user experience is good.
  • Load Balancing policy. When load balancing is enabled, all the traffic that falls in the default class is load balanced
  • Monitor-interval. Configures interval time that defines monitoring interval on ingress monitors


Performance Policies PfR policies can be based on a per application basis or by Differentiated Services Code Point (DSCP) QoS marking. Policies are group in to groups called class. Class group are primarily used to define a classification order priority but also to group all traffic that has the same administrative policies. Each class group is assigned a sequence number and PfR will look at the policies by following the sequence numbers.

There are a couple of core principles that should be considered:

  • PfR does not support the mixing and matching of DSCP and application-based policies in the same class group.
  • PfR supports the use of predefined policies from the template or create custom policies.
  • Traffic that does not match any of the class group match statements fall into a default bucket called default class.


Load Balancing The Load-Balancing configuration is used to define the behavior of all the traffic that falls into the default class. When load balancing is enabled, Traffic Classes that fall in the default class are load balanced. When load balancing is disabled, PfRv3 deletes this default class and those Traffic Classes are forwarding based on the routing table information.


Configuring PfR policies for an IWAN domain includes the following:

  • Class name definition with their respective sequence numbers
  • Define policy based on DSCP or Application Name
  • Use custom or pre-defined policies
  • Define path preference


A very common example includes the definition of a class group for voice, interactive video and critical data. This is what we use in this solution guide for the sake of simplicity. More complex policies can be defined with the use of application names.

The example below illustrates a policy definition based on DSCP that uses custom thresholds. This is defined on R10, DC-POP1-MC for the domain:

!
domain iwan
 vrf default
  master hub
   source-interface Loopback0
   site-prefixes prefix-list POP1-PREFIXES
   password 7 05080F1C2243
   load-balance advanced
    path-preference INET1 INET2 fallback MPLS1 next-fallback MPLS2
   enterprise-prefix  prefix-list ENTERPRISE-PREFIX
   collector 10.4.200.5 port 2055
   class VOICE sequence 10
    match dscp ef policy voice
    path-preference MPLS1 MPLS2 fallback INET1 INET2
    path-last-resort INET4G
   class REAL_TIME_VIDEO sequence 20
    match dscp cs4 policy real-time-video
    match dscp af41 policy real-time-video
    match dscp af42 policy real-time-video
    match dscp af43 policy real-time-video
    match dscp af31 policy real-time-video
    match dscp af32 policy real-time-video
    match dscp af33 policy real-time-video
    path-preference MPLS1 MPLS2 fallback INET1 INET2
   class LOW_LATENCY_DATA sequence 30
    match dscp cs3 policy low-latency-data
    match dscp cs2 policy low-latency-data
    match dscp af21 policy low-latency-data
    match dscp af22 policy low-latency-data
    match dscp af23 policy low-latency-data
    path-preference MPLS1 MPLS2 fallback INET1 next-fallback INET2
    path-last-resort INET4G
   class BULK_DATA sequence 40
    match dscp af11 policy bulk-data
    match dscp af12 policy bulk-data
    match dscp af13 policy bulk-data
    path-preference MPLS1 MPLS2 fallback INET1 next-fallback INET2
   class SCAVENGER sequence 50
    match dscp cs1 policy scavenger
    path-preference INET1 fallback INET2 next-fallback routing
!

Notes:

  • You can either select an existing template as domain type for policy or a custom mode. The available templates for domain policy types are listed below:
    • best-effort
    • bulk-data
    • low-latency-data
    • real-time-video
    • scavenger
    • voice
    • custom - Defines customized user-defined policy values.
  • Configures policy on per DSCP basis only - The assumption is that DSCP marking is done on ingress (LAN interface of the BRs) or even within the site (access switch).
  • Path preference for MPLS for all voice/video and critical applications.
  • Predefined or custom policies can be used.
  • Unreachable timer: recommended to set to 4 seconds instead of the default 1 second to avoid unreachable channels.
  • Monitor interval set to 4 sec for critical applications. Default is 30 seconds. You can lower the monitor interval for a couple of critical applications in order to achieve a fast failover to the secondary path. This is called quick monitor.
  • load-balance: if this is enabled, then all traffic that falls in the default class will be load balanced. If this is NOT enabled then default class is un-controlled and traffic will follow the routing information.



Rating: 0.0/5 (0 votes cast)

Personal tools