Install and Setup of Cisco Cloud Services Router (CSR) for OpenStack VPN

From DocWiki

Revision as of 13:34, 20 March 2014 by Pcm (Talk | contribs)
Jump to: navigation, search



This guide provides instructions on how to set up a CSR to provide site-to-site IPSec VPNaaS for an OpenStack cloud computing environment. The CSR will operate as an out-of-band virtual router connected in parallel with a companion Neutron router to provide the ability to establish VPN connections for tenants of OpenStack.

For simplicity, the guide will use DevStack as a way to start up OpenStack. However, the same procedures can be applied to a traditional OpenStack deployment (albeit with some manual steps at this time).

Note that the OpenStack VPNaaS feature is an experimental release, and APIs may change in the future. These procedures describe a manual process to set up VPN using the CSR. The goal in the future is to integrate the CSR into OpenStack, so that use with VPNaaS (and other services) will be seamless.


There are several things needed to make this happen:

  • A CSR 3.12 .iso image dated 3/21/2014 or newer
  • CSR license (TODO link...)
  • OpenStack Icehouse-3 or newer release
  • Helper scripts for using up the CSR under OpenStack (TODO location)
  • Adequate hardware (CPU/memory) for running OpenStack with CSR images (TODO link to UCS...)
  • Connectivity to another OpenStack (e.g. DevStack) cloud or compatible IPSec VPN site-to-site device.

This guide assumes your using Ubuntu Server 12.04 LTS (64 bit, obviously). Using a different operating system is left as an exercise for the reader.


For this guide, these networks will be defined:

  • - private network that OpenStack will use for client VMs.
  • - public network that will be used for the IPSec site to site connection.
  • - management network used to interact with the CSR via admin console and via REST API.

For the public network, a physical interface will be selected (eth1 in this example), and a gateway address defined ( for the VPN link. A floating IP range will be defined as well (

The CSR will have three interfaces, one for each network, set up with static IPs. For this example, will be used for the local network, for the public network (within the floating IP range), and and, for the management network. Note: The IP for the private net can vary in Neutron, based on how many VMs are brought up before the CSR (TODO: re-write procedure to start CSR first so it is static).

CSR Preparation

We've setup an account called openstack on our system, with the usual sudo permissions. Obtain the helper scripts that will be used to assist in installing, starting, and hooking the CSR to Neutron:

git clone csr
cd csr
mkdir 3.12

Download your ISO CSR image into the 3.12 directory (if you happen to have a OVA image, it can be converted to an ISO - procedures TODO).

Copy the config.ini.sample to config.ini file and set the filename listed in the ISO environment variable, to match the image placed in the 3.12 directory. Also, set the HOST environment variable to the IP of the host the CSR will run on and HOSTNAME to the hosts name.


NOTE: If you are using this same method on the far end's setup, be sure to change the UUID and MAC addresses, so that there are no conflicts on the public interfaces (do this on all I/Fs).

To do the install, run sudo ./install-csr as root. From another system (e.g. a Ubuntu desktop jump box) with access to this host, VNC into the host, on port zero (e.g. to access the console of the CSR and watch the boot process. It takes 15-20 minutes, so be patient. When asked "Would you like to enter the initial configuration dialog", select yes and the following selections can be made:

  • Basic management => yes
  • Defaults for settings for host
  • enable secret lab, enable password lab (ignore the warning about them being the same, and re-enter again)
  • virtual term password cisco, which will be used when accessing via REST
  • SNMP management "yes" and choose all defaults by pressing enter
  • Enter GigabitEthernet1 for management interface.
  • Enter "yes" to configure IP and set an IP address and subnet mask for management access (e.g.
  • Select to save the configuration (option 2).

At this point, the basic CSR installation is completed. On the host, press control-C to shutdown the CSR VM. Use the following command to remove the temporary bridge that was created to install the CSR:

sudo ovs-vsctl del-br br-csr

The next step is to configure the CSR for REST access, but first OpenStack (DevStack) must be installed and started up...

OpenStack Startup

DevStack needs to be configured for the private and public networks for the topology. The VPN service needs to be enabled, gateway IP and floating IP range defined, and the ethernet interface for the public network defined. In addition, the CSR startup scripts are designed to work with the ML2 plugin, so this too must be specified in devstack. Here is an example localrc file:

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service tempest
enable_service q-vpn




export OS_NO_CACHE=True
# Temp try ML2
# Q_PLUGIN=openvswitch



In the above, we have the local network (, public network (, floating IP range (, gateway, and public interface (eth1), defined. It is set to RECLONE and pull fresh sources down.

Prior to starting DevStack, the external bridge should be created and the ethernet interface added, and brought up. We'll create a script to run, as it will be used later.

cd /home/openstack/devstack
mkdir scripts
cat << EOF | tee scripts/prep
sudo ovs-vsctl del-br br-ex
sudo ifconfig eth1 down
sudo ifconfig eth1 up
sudo ovs-vsctl add-br br-ex
sudo ifconfig br-ex up
sudo ovs-vsctl add-port br-ex eth1

chmod 755 scripts/prep


To enable the CSR to be used with DevStack, several configuration changes are needed for Neutron. However, this is a chicken and the egg problem, because DevStack has not yet downloaded the Neutron code. This requires a two step process to start things up. First, DevStack needs to be started, by using ./ This will take some time, as all the components are downloaded, installed, and run. Once the process is completed, use ./ to shut things back down, to end the first stage.

Adapting OpenStack To Work With The CSR

Note: If not running Icehouse-3 with all the VPN functionality, this is the point where the Neutron (and Neutron client) code can be updated).

Now that Neutron code has been installed, you can modify localrc to turn off recloning and set offline operation:

# OFFLINE=false

Next, to prepare DevStack so that it starts up with the Cisco service driver and device driver, apply this change:

patch -p 1 < EOT
diff --git a/lib/neutron b/lib/neutron
index 81db2a7..8bd8d9e 100644
--- a/lib/neutron
+++ b/lib/neutron
@@ -481,7 +481,7 @@ function start_neutron_agents() {
         L3_CONF_FILES="$L3_CONF_FILES --config-file $Q_FWAAS_CONF_FILE"
     if is_service_enabled q-vpn; then
-        screen_it q-vpn "cd $NEUTRON_DIR && $AGENT_VPN_BINARY $L3_CONF_FILES"
+        screen_it q-vpn "cd $NEUTRON_DIR && $AGENT_VPN_BINARY $VPN_CONF_FILES"
         screen_it q-l3 "cd $NEUTRON_DIR && python $AGENT_L3_BINARY $L3_CONF_FILES"
@@ -646,6 +646,7 @@ function _configure_neutron_dhcp_agent() {
 function _configure_neutron_l3_agent() {
+    local cfg_file
     # for l3-agent, only use per tenant router if we have namespaces
@@ -657,6 +658,15 @@ function _configure_neutron_l3_agent() {
+    if is_service_enabled q-vpn; then
+       Q_VPN_CONF_FILE=$NEUTRON_CONF_DIR/vpn_agent.ini
+       cp $NEUTRON_DIR/etc/vpn_agent.ini $Q_VPN_CONF_FILE
+       VPN_CONF_FILES="--config-file $NEUTRON_CONF --config-file=$Q_L3_CONF_FILE --config-file=$Q_VPN_CONF_FILE"
+       for cfg_file in ${Q_VPN_EXTRA_CONF_FILES[@]}; do
+            VPN_CONF_FILES+=" --config-file $cfg_file"
+        done
+    fi
     cp $NEUTRON_DIR/etc/l3_agent.ini $Q_L3_CONF_FILE
     iniset $Q_L3_CONF_FILE DEFAULT verbose True
@@ -703,6 +713,7 @@ function _configure_neutron_vpn()
+    neutron_vpnaas_configure_driver
 # _configure_neutron_plugin_agent() - Set config files for neutron plugin agent
diff --git a/lib/neutron_plugins/services/vpn b/lib/neutron_plugins/services/vpn
index 02370e7..0e9a302 100644
--- a/lib/neutron_plugins/services/vpn
+++ b/lib/neutron_plugins/services/vpn
@@ -18,6 +18,10 @@ function neutron_vpn_configure_common() {
     _neutron_service_plugin_class_add $VPN_PLUGIN
+function neutron_vpnaas_configure_driver() {                                                                                                                                                
+    iniset_multiline $NEUTRON_CONF service_providers service_provider "" ""                                                                                                                                              
 function neutron_vpn_stop() {
     local ipsec_data_dir=$DATA_DIR/neutron/ipsec
     local pids

This will cause the service driver to use the reference OpenSwan and Cisco (provider) drivers. To enable the Cisco device driver as one of the available drivers, in /opt/stack/neutron/etc/vpn_agent.ini, uncommenting this line:

Finally, the CSR information must be placed in /opt/stack/neutron/etc/neutron/plugins/cisco/cisco_vpn_agent.ini, which the above patch will make this available when the agent starts up:

rest_mgmt =
tunnel_ip =
username = stack
password = cisco
timeout = 60

The IP on the [cisco_csr_rest:...] line is the public IP for the Neutron router that is associated with the CSR. Private network packets sent to the Neutron router, will be forwarded to the CSR, which will encrypt and send them over the tunnel established on the public network defined. The rest_mgmt IP is the predefined CSR IP address on the management network that will be used for REST messages. The tunnel_ip will be the public interface IP for the CSR, and will be on the same subnet as the router's public interface (typically, Neutron will assign the first floating IP to the router, and the third to the CSR). This must match the IP that is assigned by Neutron, when the CSR is later started (see below).

DevStack is then re-started with first ensuring that the OVS bridge is set up with port added (as can remove), and then stacking:

cd /home/openstack/devstack

Once this startup has completed, SSH and ping can be enabled on the private nework, and some VMs created (optionally, although this dictates the IP addresses that are assigned to Neutron for the CSR later on):

cat << EOT | tee scripts/build-vms
source openrc admin admin

# Allow admin tenant access to private net (needed?)
neutron net-update --shared=True private

nova secgroup-add-rule default icmp -1 -1
nova secgroup-add-rule default tcp 22 22
source openrc admin demo
PRIVATE_NET=\`neutron net-list | grep private | cut -f 2 -d'|' | cut -f 2 -d' '\`
nova boot --flavor 1 --image cirros-0.3.1-x86_64-uec --nic net-id=\$PRIVATE_NET peter
nova boot --flavor 1 --image cirros-0.3.1-x86_64-uec --nic net-id=\$PRIVATE_NET paul

chmod 755 scripts/build-vms


To setup a management network for the CSR and port in OVS to access the CSR, do the following:

cat << EOF | tee scripts/csr
source openrc admin admin

# Create a mamangement network for CSR
neutron net-create mgmt
neutron subnet-create --disable-dhcp --name=mgmt-subnet mgmt

sudo ovs-vsctl add-port br-int to_csr -- set interface to_csr type=internal 
sudo ifconfig to_csr up

chmod 755 scripts/csr


CSR Configuration

Now that the Neutron network is all set up, the CSR VM can be booted up. In a separate terminal (because this will need to remain running as long as the CSR is up), use the following commands:

cd /home/openstack/csr
sudo ./

The CSR will boot up (this takes 10-15 mins), after which you can then Telnet to the CSR using the management IP set up (e.g. The boot process can be monitored by VNC into the host at port 50 (e.g. Once logged into the CSR (password set up as cisco during the install process) and entered "enable" mode (password set up as lab), a basic configuration needs to be set up for the other two interfaces, and REST access and licensing need to be applied. Here are the configuration lines used for this guide's example system:

config t
interface GigabitEthernet2
 ip address
 negotiation auto
 no shut
interface GigabitEthernet3
 ip address
 negotiation auto
 no shut
interface VirtualPortGroup0
 ip unnumbered GigabitEthernet1
transport-map type persistent webui http-restapi
virtual-service csr_mgmt
 vnic gateway VirtualPortGroup0
  guest ip address
transport type persistent webui input http-restapi
ip http server
ip http authentication local
ip http secure-server
ip route GigabitEthernet1
ip route VirtualPortGroup0
ip ssh version 2
username stack priv 15 secret cisco
license boot level premium

For GigabitEthernet2 (on public net) and GigabitEthernet3 (on private net), you should verify the IP addresses and make sure they match what was created by the Neutron binding (and make sure the public IP matches what you placed in the cisco_vpn_agent.ini file). Use:

cd /home/openstack/devstack
source openrc admin demo
neutron port-list

and check public_p and private_p IPs for GigabitEthernet2 and GigabitEthernet3, respectively. The public network should be the same, but the private network IP may differ, depending on how many VMs you have started up on the network. If they are different, alter the above configuration. The license line will require that you accept the license agreement. When done, you must end configuration and write to save the configuration.

To complete the setup of the licensing, the CSR will need to be restarted. This requires restarting DevStack too, otherwise, if DevStack is still running, the CSR will get assigned different IP addresses by Neutron and they won't match that of what is statically configured in the CSR. The steps are:

  • In the terminal where the CSR was started, control-C to shut down the VM and disconnect from Neutron
  • In the terminal where DevStack was started, ./ to stop the running DevStack instance
  • Run to start up the Devstack
  • At this point you can start up VMs, with scripts/build-vms
  • Run scripts/csr to prepare for CSR operation.
  • Run as root again, to start the CSR back up. Again, it will take about 10-15 mins to boot, and you can monitor via VNC, if desired.

You can reverify connectivity by pinging other devices on the private, public, and management networks.

Note: In the future, each time you start DevStack, create VMs, and then the CSR, as long as you use the same sequence, the private IP address will be the same.


At this point, DevStack should be able to communicate with the CSR via the REST API.

Note: If you are behind a firewall, you may have to disable proxy for the CSR IPs:

export no_proxy=$no_proxy,,

You can verify access using:

curl  -X POST -H "Accept:application/json" -H "Content-Length: 0" -u "stack:cisco" -d "" -k -3 -v

To use the CSR, you can create a VPN service using --provider cisco and then connections created for that service will use the CSR to establish the tunnel. You'll need to have the Neutron router (e.g. router1) forward packets for the far end, to the CSR, so that client VM traffic destined for the remote end will be redirected to the CSR and sent over the tunnel. For example:

source openrc admin demo

CSR_PORT=`neutron port-list | grep private_p | cut -d' ' -f 11 | cut -f 2 -d'"'`
neutron router-update router1 --routes type=dict list=true destination=,nexthop=$CSR_PORT

Here is a sample of creating an IPSec site-to-site connection:

neutron vpn-ikepolicy-create ikepolicy1
neutron vpn-ipsecpolicy-create ipsecpolicy1
neutron vpn-service-create --name myvpn --description "My vpn service" --provider cisco router1 mysubnet

neutron ipsec-site-connection-create --name vpnconnection1 --vpnservice-id myvpn --ikepolicy-id ikepolicy1 \
        --ipsecpolicy-id ipsecpolicy1 --peer-address --peer-id \
        --peer-cidr --psk secret

Reference Information

TODO: OS link, VPN APIs, CSR pages,...

Rating: 5.0/5 (1 vote cast)

Personal tools