Install and Setup of Cisco Cloud Services Router (CSR) for OpenStack VPN

From DocWiki

Revision as of 17:38, 26 March 2014 by Pcm (Talk | contribs)
Jump to: navigation, search



This guide provides instructions on how to set up a CSR to provide site-to-site IPSec VPNaaS for an OpenStack cloud computing environment. The CSR will operate as an out-of-band virtual router connected in parallel with a companion Neutron router to provide the ability to establish VPN connections for tenants of OpenStack.

For simplicity, the guide will use DevStack as a way to start up OpenStack. However, the same procedures can be applied to a traditional OpenStack deployment (albeit with some manual steps at this time).

Note that the OpenStack VPNaaS feature is an experimental release, and APIs may change in the future. These procedures describe a manual process to set up VPN using the CSR. The goal in the future is to integrate the CSR into OpenStack, so that use with VPNaaS (and other services) will be seamless.


There are several things needed to setup an OpenStack cloud for VPN operation with CSR:

  • A CSR 3.13 .iso image (TODO link to published image)
  • CSR license (TODO link...)
  • OpenStack Icehouse-3 or newer release
  • Helper scripts for using up the CSR under OpenStack -
  • Adequate hardware (CPU/memory) for running OpenStack with CSR images (TODO link to UCS...)
  • Connectivity to another OpenStack (e.g. DevStack) cloud or compatible IPSec VPN site-to-site device (virtual or physical).


  • You have a basic understanding of how to start up DevStack (
  • You have some familiarity with how to configure (from a user's perspective) OpenStack's VPN with either Horizon or the Neutron client.
  • You're using Ubuntu Server 12.04 LTS (64 bit, obviously) on your host. Using a different operating system is left as an exercise for the reader.

In this guide, we'll focus on the detailed steps needed to set up one of the two sites for the IPSec site-to-site VPN connectivity. The example uses the same method for the other site (with just brief info on the different IPs used), and you can follow the same instructions to create a complete site-to-site connection with two clouds using a CSR. Alternately, you can employ the reference OpenStack VPN implementation or a compatible virtual or physical VPN device for the other end of the site-to-site VPN connection.


In the example configuration, we have this physical topology:


There are two UCS systems, which will each host an OpenStack (using DevStack) cloud for the site-to-site VPN end-points. A physical connection between the two nodes will be used for the "public" network (, where traffic between the private networks ( and will be encrypted. There is an internal network on the node ( for management of the CSR during operation, and another network ( for setup of the clouds.

Logically, we have the following topology for the first cloud, running on the devstack-32 host:


You can see the IPs that will be used for public and private networks. For the other host, devstack-33, we have:


The key information in these drawings are the IP addresses used for the CSR (private and public), the public IP of the peer CSR, and the physical ethernet interface being used.

CSR Preparation

For this step, we have these attributes for the devstack-32 host:

  • Host IP is
  • Local (private) IP for CSR will be
  • Public IP for CSR will be

We've setup an account called openstack on our system, with the usual sudo permissions. Obtain the helper scripts that will be used to assist in installing, starting, and hooking the CSR to Neutron:

git clone csr
cd csr
mkdir 3.12

Download your ISO CSR image into the 3.12 directory (if you happen to have a OVA image, it can be converted to an ISO - procedures TODO).

Copy the config.ini.sample to config.ini file and set the filename listed in the ISO environment variable, to match the image placed in the 3.12 directory. Also, set the HOST environment variable to the IP of the host the CSR will run on and HOSTNAME to the hosts name.


NOTE: If you are using this same method on the far end's setup, be sure to change the UUID and MAC addresses, so that there are no conflicts on the public interfaces (do this on all I/Fs).

To do the install, run sudo ./install-csr as root. From another system (e.g. a Ubuntu desktop jump box) with access to this host, VNC into the host, on port zero (e.g. to access the console of the CSR and watch the boot process. It takes 15-20 minutes, so be patient. When asked "Would you like to enter the initial configuration dialog", select yes and the following selections can be made:

  • Basic management => yes
  • Defaults for settings for host
  • enable secret lab, enable password lab (ignore the warning about them being the same, and re-enter again)
  • virtual term password cisco, which will be used when accessing via REST
  • SNMP management "yes" and choose all defaults by pressing enter
  • Enter GigabitEthernet1 for management interface.
  • Enter "yes" to configure IP and set an IP address and subnet mask for management access (e.g.
  • Select to save the configuration (option 2).

You can then Telnet to the CSR using the management IP set up (e.g. with the password "cisco", enter enable mode "en" with password "lab", and provide the basic configuration:

config t
interface GigabitEthernet2
 ip address
 negotiation auto
 no shut
interface GigabitEthernet3
 ip address
 negotiation auto
 no shut
interface VirtualPortGroup0
 ip unnumbered GigabitEthernet1
transport-map type persistent webui http-restapi
virtual-service csr_mgmt
 vnic gateway VirtualPortGroup0
  guest ip address
transport type persistent webui input http-restapi
ip http server
ip http authentication local
ip http secure-server
ip route GigabitEthernet1
ip route VirtualPortGroup0
ip ssh version 2
username stack priv 15 secret cisco
license boot level premium

You must enter "yes" for the licensing, and when it is completed, "end" configuration and enter "write" to save the configuration. At this point, the basic CSR installation is completed, although the system must be rebooted to complete the licensing process. This will be done later, but for now, on the host, press control-C to shutdown the CSR VM. Use the following command to remove the temporary bridge that was created to install the CSR:

sudo ovs-vsctl del-br br-csr

The next step is to configure the CSR for REST access, but first OpenStack (DevStack) must be installed and started up...

OpenStack Startup

DevStack needs to be configured for the private and public networks for the topology. The VPN service needs to be enabled, gateway IP and floating IP range defined, and the ethernet interface for the public network defined. In addition, the CSR startup scripts are designed to work with the ML2 plugin, so this too must be specified in devstack.

For the devstack-32 host, we have these attributes to consider:

  • Local network (FIXED_RANGE) will be with router (GW) using
  • Public GW at on the public network.
  • Set of floating IPs reserved on the public network in the range -
  • Using eth1 for the public network interface.

Here is the localrc file for devstack-32, using the above information:

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service tempest
enable_service q-vpn




export OS_NO_CACHE=True
# Temp try ML2
# Q_PLUGIN=openvswitch



For the first stacking of DevStack, RECLONE is set to pull fresh sources down. Subsequent runs can set RECLONE to No, to keep any changes made in the /opt/stack area.

Prior to starting DevStack, the external bridge should be created and the ethernet interface added, and brought up. We'll create a script (prep.32) to run, as it will be used later.

cd /home/openstack/devstack
mkdir scripts
cat << EOF | tee scripts/prep
sudo ovs-vsctl del-br br-ex
sudo ifconfig eth1 down
sudo ifconfig eth1 up
sudo ovs-vsctl add-br br-ex
sudo ifconfig br-ex up
sudo ovs-vsctl add-port br-ex eth1

chmod 755 scripts/prep.32


Note that we are specifying the physical inteface, eth1, for this host so that Neutron can use this public network from br-ex.

At this point, you can start DevStack, by using ./ This will take some time, as all the components are downloaded, installed, and run.

Adapting OpenStack To Work With The CSR

Note: If not running Icehouse-3 with all the VPN functionality, this is the point where the Neutron (and Neutron client) code can be updated).

Now that Neutron code has been installed, you can modify localrc to turn off recloning and set offline operation:

# OFFLINE=false

Next, to prepare DevStack so that it starts up with the Cisco service driver and device driver, apply this change:

patch -p 1 < EOT
diff --git a/lib/neutron b/lib/neutron
index 81db2a7..8bd8d9e 100644
--- a/lib/neutron
+++ b/lib/neutron
@@ -481,7 +481,7 @@ function start_neutron_agents() {
         L3_CONF_FILES="$L3_CONF_FILES --config-file $Q_FWAAS_CONF_FILE"
     if is_service_enabled q-vpn; then
-        screen_it q-vpn "cd $NEUTRON_DIR && $AGENT_VPN_BINARY $L3_CONF_FILES"
+        screen_it q-vpn "cd $NEUTRON_DIR && $AGENT_VPN_BINARY $VPN_CONF_FILES"
         screen_it q-l3 "cd $NEUTRON_DIR && python $AGENT_L3_BINARY $L3_CONF_FILES"
@@ -646,6 +646,7 @@ function _configure_neutron_dhcp_agent() {
 function _configure_neutron_l3_agent() {
+    local cfg_file
     # for l3-agent, only use per tenant router if we have namespaces
@@ -657,6 +658,15 @@ function _configure_neutron_l3_agent() {
+    if is_service_enabled q-vpn; then
+       Q_VPN_CONF_FILE=$NEUTRON_CONF_DIR/vpn_agent.ini
+       cp $NEUTRON_DIR/etc/vpn_agent.ini $Q_VPN_CONF_FILE
+       VPN_CONF_FILES="--config-file $NEUTRON_CONF --config-file=$Q_L3_CONF_FILE --config-file=$Q_VPN_CONF_FILE"
+       for cfg_file in ${Q_VPN_EXTRA_CONF_FILES[@]}; do
+            VPN_CONF_FILES+=" --config-file $cfg_file"
+        done
+    fi
     cp $NEUTRON_DIR/etc/l3_agent.ini $Q_L3_CONF_FILE
     iniset $Q_L3_CONF_FILE DEFAULT verbose True
@@ -703,6 +713,7 @@ function _configure_neutron_vpn()
+    neutron_vpnaas_configure_driver
 # _configure_neutron_plugin_agent() - Set config files for neutron plugin agent
diff --git a/lib/neutron_plugins/services/vpn b/lib/neutron_plugins/services/vpn
index 02370e7..0e9a302 100644
--- a/lib/neutron_plugins/services/vpn
+++ b/lib/neutron_plugins/services/vpn
@@ -18,6 +18,10 @@ function neutron_vpn_configure_common() {
     _neutron_service_plugin_class_add $VPN_PLUGIN
+function neutron_vpnaas_configure_driver() {                                                                                                                                                
+    iniset_multiline $NEUTRON_CONF service_providers service_provider "" ""                                                                                                                                              
 function neutron_vpn_stop() {
     local ipsec_data_dir=$DATA_DIR/neutron/ipsec
     local pids

This will cause the service driver to use the reference OpenSwan and Cisco (provider) drivers. To enable the Cisco device driver as one of the available drivers, in /opt/stack/neutron/etc/vpn_agent.ini, uncommenting this line:

Finally, the CSR information must be placed in /opt/stack/neutron/etc/neutron/plugins/cisco/cisco_vpn_agent.ini, which the above patch will make this available when the agent starts up:

rest_mgmt =
tunnel_ip =
username = stack
password = cisco
timeout = 60

The IP on the [cisco_csr_rest:...] line is the public IP for the Neutron router that is associated with the CSR. Private network packets sent to the Neutron router, will be forwarded to the CSR, which will encrypt and send them over the tunnel established on the public network defined. The rest_mgmt IP is the predefined CSR IP address on the management network that will be used for REST messages. The tunnel_ip will be the public interface IP for the CSR, and will be on the same subnet as the router's public interface (typically, Neutron will assign the first floating IP to the router, and the third to the CSR). This must match the IP that is assigned by Neutron, when the CSR is later started (see below).

DevStack is then re-started with first ensuring that the OVS bridge is set up with port added (as can remove), and then stacking:

cd /home/openstack/devstack

Once this startup has completed, SSH and ping can be enabled on the private nework, and some VMs created (optionally, although this dictates the IP addresses that are assigned to Neutron for the CSR later on):

cat << EOT | tee scripts/build-vms
source openrc admin admin

# Allow admin tenant access to private net (needed?)
neutron net-update --shared=True private

nova secgroup-add-rule default icmp -1 -1
nova secgroup-add-rule default tcp 22 22
source openrc admin demo
PRIVATE_NET=\`neutron net-list | grep private | cut -f 2 -d'|' | cut -f 2 -d' '\`
nova boot --flavor 1 --image cirros-0.3.1-x86_64-uec --nic net-id=\$PRIVATE_NET peter
nova boot --flavor 1 --image cirros-0.3.1-x86_64-uec --nic net-id=\$PRIVATE_NET paul

chmod 755 scripts/build-vms


To setup a management network for the CSR and port in OVS to access the CSR, do the following:

cat << EOF | tee scripts/csr
source openrc admin admin

# Create a mamangement network for CSR
neutron net-create mgmt
neutron subnet-create --disable-dhcp --name=mgmt-subnet mgmt

sudo ovs-vsctl add-port br-int to_csr -- set interface to_csr type=internal 
sudo ifconfig to_csr up

chmod 755 scripts/csr


CSR Configuration

Now that the Neutron network is all set up (and the two VMs created), the CSR VM can be booted up. In a separate terminal (because this will need to remain running as long as the CSR is up), use the following commands:

cd /home/openstack/csr
sudo ./

The CSR will boot up (this takes 5-7 mins), after which you can then Telnet to the CSR using the management IP set up (e.g. The boot process can be monitored by VNC into the host at port 50 (e.g. The configuration done previously, should be there, and interfaces should all be up/up. For GigabitEthernet2 (on public net) and GigabitEthernet3 (on private net), you should verify the IP addresses and make sure they match what was created by the Neutron binding (and make sure the public IP matches what you placed in the cisco_vpn_agent.ini file). Use:

cd /home/openstack/devstack
source openrc admin demo
neutron port-list

and check public_p and private_p IPs for GigabitEthernet2 and GigabitEthernet3, respectively. The public network should be the same, but the private network IP may differ, depending on how many VMs you have started up on the network. We created two VMs, so the CSR should be at 10.x.0.6. If either of the addresses are wrong, then either they were entered incorrectly, a different number of VMs were created, or the CSR was stopped and restarted (as Neutron will assign the next available IP each time the CSR is started). If things don't match, alter the CSR configuration.

Note: In the future, each time you start DevStack, create VMs, and then the CSR, as long as you use the same sequence, the private IP address will be the same.


At this point, DevStack should be able to communicate with the CSR via the REST API. You'll need to do the same thing (with different IPs obviously) for a second cloud setup, either using another CSR with OpenStack, the reference OpenStack implementation, or a compatible VPN device (like a standalone CSR in a remote office). Once that is done, a VPN connection can be established using the Neutron client VPN commands or using Horizon.

Using the command line and the Neutron client, here are the steps...

Note: If you are behind a firewall, you may have to disable proxy for the CSR IPs:

export no_proxy=$no_proxy,,

You can verify access using:

curl  -X POST -H "Accept:application/json" -H "Content-Length: 0" -u "stack:cisco" -d "" -k -3 -v

To use the CSR, you can create a VPN service using --provider cisco and then connections created for that service will use the CSR to establish the tunnel. You'll need to have the Neutron router (e.g. router1) forward packets for the far end, to the CSR, so that client VM traffic destined for the remote end will be redirected to the CSR and sent over the tunnel. For example:

source openrc admin demo

CSR_PORT=`neutron port-list | grep private_p | cut -d' ' -f 11 | cut -f 2 -d'"'`
neutron router-update router1 --routes type=dict list=true destination=,nexthop=$CSR_PORT

Here is a sample of creating an IPSec site-to-site connection:

neutron vpn-ikepolicy-create ikepolicy1
neutron vpn-ipsecpolicy-create ipsecpolicy1
neutron vpn-service-create --name myvpn --description "My vpn service" --provider cisco router1 mysubnet

neutron ipsec-site-connection-create --name vpnconnection1 --vpnservice-id myvpn --ikepolicy-id ikepolicy1 \
        --ipsecpolicy-id ipsecpolicy1 --peer-address --peer-id \
        --peer-cidr --psk secret

Reference Information

TODO: OS link, VPN APIs, CSR pages,...

Rating: 5.0/5 (1 vote cast)

Personal tools