Openstack with LISP-enabled OpenVSwitch
From DocWiki
(→Bringing up a guest VM) |
|||
(26 intermediate revisions not shown) | |||
Line 1: | Line 1: | ||
- | This wiki describes all the steps necessary to setup a testbed of [http://www.openstack.org/software/folsom/ Openstack Folsom] nodes (All-in-one node and Compute nodes), which rely on a [http://lisp.cisco.com/ LISP-enabled] [http://openvswitch.org/ OpenVSwitch (OVS)] bridge. | + | <p>This wiki describes all the steps necessary to setup a testbed of [http://www.openstack.org/software/folsom/ Openstack Folsom] nodes (All-in-one node and Compute nodes), which rely on a [http://lisp.cisco.com/ LISP-enabled] [http://openvswitch.org/ OpenVSwitch (OVS)] bridge. |
+ | </p> | ||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | __TOC__ | + | <p>__TOC__ |
- | + | </p><p><br /> | |
- | + | </p> | |
- | + | <h1> Testbed Setup </h1> | |
- | + | <h2> Dependencies </h2> | |
- | + | <p>The Openstack nodes are considered hereafter to be <a href="http://releases.ubuntu.com/precise/">Ubuntu 12.04.1 LTS (Precise Pangolin)</a> machines (running the 3.2.0-29 kernel version). | |
- | + | ||
- | The Openstack nodes are considered hereafter to be | + | |
The following dependencies need to be met. | The following dependencies need to be met. | ||
- | + | </p> | |
<pre> | <pre> | ||
sudo apt-get update | sudo apt-get update | ||
Line 40: | Line 23: | ||
sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils virt-manager virtinst | sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils virt-manager virtinst | ||
</pre> | </pre> | ||
- | + | <p>In case you wish to also use Wireshark (with LISP dissector included): | |
- | In case you wish to also use Wireshark (with LISP dissector included): | + | </p> |
- | + | ||
<pre> | <pre> | ||
$ sudo apt-get build-dep wireshark | $ sudo apt-get build-dep wireshark | ||
Line 55: | Line 37: | ||
$ sudo make install | $ sudo make install | ||
$ sudo ldconfig | $ sudo ldconfig | ||
- | $ wireshark & | + | $ wireshark & |
</pre> | </pre> | ||
- | + | <p>Although this is simply a recommendation, the testbed has been successfully run with the following memory settings: | |
- | Although this is simply a recommendation, the testbed has been successfully run with the following memory settings: | + | </p> |
- | + | <ul><li> 1.8GB for the All-In-One Node | |
- | + | </li><li> 1.2GB for the Compute Node | |
- | + | </li></ul> | |
- | + | <p><br /> | |
- | + | </p> | |
+ | <h2> Network </h2> | ||
<pre> | <pre> | ||
Public + Management Network | Public + Management Network | ||
Line 69: | Line 52: | ||
| | | | | | ||
| | | | | | ||
- | |||
| | | | | | ||
+------|-------------+ +--------|-------+ | +------|-------------+ +--------|-------+ | ||
| (eth0) | | (eth0) | | | (eth0) | | (eth0) | | ||
| | | | | | | | | | | | | | ||
+ | | | | | | | | ||
+ | | 10.10.10.10 | | 10.10.11.10 | | ||
| | | | | | | | | | | | | | ||
| [ br-lisp ] | | [br-lisp] | | | [ br-lisp ] | | [br-lisp] | | ||
Line 89: | Line 73: | ||
() Network interface | () Network interface | ||
</pre> | </pre> | ||
- | + | ||
+ | or | ||
+ | |||
+ | [[Image:OpenstackandLISP_Testbed.png|630px|middle]] | ||
- | |||
- | |||
- | |||
+ | <p>If instead of physical machines you plan on setting this testbed up with <b>VMware Fusion</b>, make sure the network is configured as <i>Bridged (Ethernet).</i> | ||
+ | </p><p><br /> | ||
+ | </p> | ||
+ | <h2> Map Server (MS) </h2> | ||
+ | <p>Just to get an idea of how the EID addresses are supposed to be associated with their corresponding RLOCs, you can have a look at the following MS configuration, which makes static assignments. | ||
+ | </p><p>Under regular LISP operation you will not have to impose the following configuration. Rather, it is formed automatically and dynamically. | ||
+ | </p> | ||
<pre> | <pre> | ||
- | + | <?xml version="1.0" encoding="ISO-8859-1"?> | |
- | + | <db> | |
- | + | <eid prefix="0.0.0.0/3" act="2" ttl="5" /> | |
- | + | <eid prefix="128.0.0.0/8" act="2" ttl="5" /> | |
- | + | <eid prefix="171.0.0.0/8" act="2" ttl="5" /> | |
- | + | <eid prefix="224.0.0.0/4" act="2" ttl="5" /> | |
- | + | <eid prefix="192.168.127.3/32" ttl="1440" A="true"> | |
- | + | <rloc> | |
- | + | <address>10.10.10.10</address> | |
- | + | <priority>1</priority> | |
- | + | <weight>100</weight> | |
- | + | <m_priority>255</m_priority> | |
- | + | <m_weight>0</m_weight> | |
- | + | <reachable>true</reachable> | |
- | + | </rloc> | |
- | + | </eid> | |
- | + | <eid prefix="192.168.127.4/32" ttl="1440" A="true"> | |
- | + | <rloc> | |
- | + | <address>10.10.11.10</address> | |
- | + | <priority>1</priority> | |
- | + | <weight>100</weight> | |
- | + | <m_priority>255</m_priority> | |
- | + | <m_weight>0</m_weight> | |
- | + | <reachable>true</reachable> | |
- | + | </rloc> | |
- | + | </eid> | |
- | + | </db> | |
</pre> | </pre> | ||
- | + | <h1> Installing the LISP-enabled OVS bridge </h1> | |
- | + | <p>On both machines do the following. | |
- | + | ||
- | On both machines do the following. | + | |
First download the corresponding code: | First download the corresponding code: | ||
- | + | </p> | |
<pre> | <pre> | ||
- | cd / | + | cd root/src/ |
git clone https:// | git clone https:// | ||
</pre> | </pre> | ||
- | + | <p>First install lispmob: | |
- | First install lispmob: | + | </p> |
<pre> | <pre> | ||
- | cd / | + | cd root/src/lispmob-ovs/ |
make | make | ||
sudo make install | sudo make install | ||
sudo depmod -a | sudo depmod -a | ||
</pre> | </pre> | ||
- | + | <p>Afterward install the modified LISP-enabled OpenVswitch: | |
- | Afterward install the modified LISP-enabled OpenVswitch: | + | </p> |
<pre> | <pre> | ||
- | cd / | + | cd root/src/ovs-lisp/ |
./boot.sh | ./boot.sh | ||
./configure --with-linux=/lib/modules/`uname -r`/build | ./configure --with-linux=/lib/modules/`uname -r`/build | ||
make | make | ||
</pre> | </pre> | ||
- | |||
<pre> | <pre> | ||
- | cp / | + | cp root/src/lispmob-ovs/lisp_mod/Module.symvers root/src/ovs-lisp/datapath/linux/ |
- | sudo vi / | + | sudo vi root/src/ovs-lisp/datapath/linux/Makefile.main.in |
</pre> | </pre> | ||
- | ...and remove 'Module.symvers' from the 'clean' target (on line 33) | + | <p>...and remove 'Module.symvers' from the 'clean' target (on line 33) |
- | + | </p> | |
<pre> | <pre> | ||
sudo reboot | sudo reboot | ||
</pre> | </pre> | ||
- | + | <p>Then, you will need to REMAKE and REINSTALL lisp and reboot! | |
- | Then, you will need to REMAKE and REINSTALL lisp | + | </p> |
- | + | ||
<pre> | <pre> | ||
- | cd / | + | cd root/src/lispmob-ovs |
make | make | ||
sudo make install | sudo make install | ||
sudo depmod -a | sudo depmod -a | ||
- | cd / | + | cd root/src/ovs-lisp/ |
make | make | ||
sudo make install | sudo make install | ||
sudo depmod -a | sudo depmod -a | ||
- | sudo | + | sudo reboot |
</pre> | </pre> | ||
- | In order to bring up the OVS bridge: | + | Please retry the above in case you get the following message: |
+ | "WARNING: "eid_locator4" [/root/src/ovs-lisp/datapath/linux/openvswitch.ko] undefined!" | ||
+ | |||
+ | |||
+ | <p>In order to bring up the OVS bridge: | ||
+ | </p> | ||
<pre> | <pre> | ||
- | sudo / | + | sudo root/src/ovs-scripts/ovs-start |
</pre> | </pre> | ||
- | which includes: | + | <p>which includes: |
- | + | </p> | |
<pre> | <pre> | ||
#!/bin/bash | #!/bin/bash | ||
- | BUILD_DIR=/ | + | BUILD_DIR=root/src/ovs-lisp |
OVS_PATH=/usr/local | OVS_PATH=/usr/local | ||
Line 201: | Line 193: | ||
mkdir -p ${OVS_PATH}/etc/openvswitch | mkdir -p ${OVS_PATH}/etc/openvswitch | ||
rm ${OVS_PATH}/etc/openvswitch/conf.db | rm ${OVS_PATH}/etc/openvswitch/conf.db | ||
- | ovsdb-tool create /usr/local/etc/openvswitch/conf.db / | + | ovsdb-tool create /usr/local/etc/openvswitch/conf.db root/src/ovs-lisp/vswitchd/vswitch.ovsschema |
${OVS_PATH}/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ | ${OVS_PATH}/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ | ||
--remote=db:Open_vSwitch,manager_options \ | --remote=db:Open_vSwitch,manager_options \ | ||
Line 212: | Line 204: | ||
ps -ea | grep ovs; ps -ea | grep lisp; lsmod | grep lisp; lsmod | grep ovs | ps -ea | grep ovs; ps -ea | grep lisp; lsmod | grep lisp; lsmod | grep ovs | ||
</pre> | </pre> | ||
- | + | <p>You can verify everything is setup correctly by issuing: | |
- | You can verify everything is setup correctly by issuing: | + | </p> |
<pre> | <pre> | ||
ps -ea | grep ovs; ps -ea | grep lisp; lsmod | grep lisp; lsmod | grep ovs | ps -ea | grep ovs; ps -ea | grep lisp; lsmod | grep lisp; lsmod | grep ovs | ||
</pre> | </pre> | ||
- | which should yield an output similar to the following: | + | <p>which should yield an output similar to the following: |
+ | </p> | ||
<pre> | <pre> | ||
- | 1404 ? 00:00:00 ovs_workq 1406 ? 00:00:00 ovsdb-server | + | 1404 ? 00:00:00 ovs_workq |
- | 1409 ? 00:00:00 ovs-vswitchd | + | 1406 ? 00:00:00 ovsdb-server |
- | 1410 ? 00:00:00 ovs-vswitchd | + | 1409 ? 00:00:00 ovs-vswitchd |
+ | 1410 ? 00:00:00 ovs-vswitchd | ||
lisp 41286 1 openvswitch | lisp 41286 1 openvswitch | ||
</pre> | </pre> | ||
- | + | <h1> Openstack </h1> | |
- | + | <p>First create the stack user in order to install and run Openstack without any passwords requested. | |
- | + | </p> | |
- | First create the stack user in order to install and run Openstack without any passwords requested. | + | |
<pre> | <pre> | ||
su - | su - | ||
- | useradd -U -G sudo -s /bin/bash -m stack; echo | + | useradd -U -G sudo -s /bin/bash -m stack; echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers |
su stack | su stack | ||
cd | cd | ||
</pre> | </pre> | ||
- | + | <p>This Openstack installation is based on <a href="http://devstack.org/">Devstack</a>. | |
- | This Openstack installation is based on | + | </p><p>It is important you stick with the stable Folsom version: |
- | + | </p> | |
- | It is important you stick with the stable Folsom version: | + | |
<pre> | <pre> | ||
git clone -b stable/folsom https://github.com/openstack-dev/devstack.git | git clone -b stable/folsom https://github.com/openstack-dev/devstack.git | ||
</pre> | </pre> | ||
- | Alternatively, | + | <p>Alternatively, |
+ | </p> | ||
<pre> | <pre> | ||
git clone -b stable/folsom git://git.cisco.com/lakafosi/devstack.git | git clone -b stable/folsom git://git.cisco.com/lakafosi/devstack.git | ||
</pre> | </pre> | ||
- | + | <p>Additionally, you will have to download one more script for setting up the OVS-bridge and attaching the vnetX ports of the instantiated VMs, the lispd.conf control plane configuration file of LISP and a few cloud qcow2 (Ubuntu and Fedora) images (to use in addition to the preloaded tiny <a href="https://launchpad.net/cirros/+download">CirrOS cloud guest</a>). | |
- | Additionally, you will have to download one more script for setting up the OVS-bridge and attaching the vnetX ports of the instantiated VMs, the lispd.conf control plane configuration file of LISP and a few cloud qcow2 (Ubuntu and Fedora) images (to use in addition to the preloaded tiny | + | </p> |
<pre> | <pre> | ||
mkdir /home/stack/Downloads | mkdir /home/stack/Downloads | ||
mv /* /home/stack/devstack/Downloads/* /home/stack/Downloads | mv /* /home/stack/devstack/Downloads/* /home/stack/Downloads | ||
</pre> | </pre> | ||
- | + | <p><br /> | |
- | + | </p> | |
- | + | <h2> All-In-One Openstack Node </h2> | |
- | + | <h3> localrc </h3> | |
- | + | <p>Located in /home/stack/devstack/ | |
- | Located in /home/stack/devstack/ | + | </p> |
<pre> | <pre> | ||
#HOST_NAME=$(hostname) | #HOST_NAME=$(hostname) | ||
Line 293: | Line 286: | ||
#OFFLINE=True | #OFFLINE=True | ||
</pre> | </pre> | ||
- | + | <p><br /> | |
- | + | </p> | |
- | + | <h3> local.sh </h3> | |
- | This is the script run right after | + | <p>This is the script run right after <i>stack.sh</i>, i.e. the Openstack installation. It is located in /home/stack/devstack/ . |
+ | </p> | ||
<pre> | <pre> | ||
#!/usr/bin/env bash | #!/usr/bin/env bash | ||
Line 304: | Line 298: | ||
# Keep track of the devstack directory | # Keep track of the devstack directory | ||
- | TOP_DIR=$(cd $(dirname | + | TOP_DIR=$(cd $(dirname "$0") && pwd) |
# Import common functions | # Import common functions | ||
source $TOP_DIR/functions | source $TOP_DIR/functions | ||
Line 318: | Line 312: | ||
source $TOP_DIR/openrc admin admin | source $TOP_DIR/openrc admin admin | ||
# nova-manage instance_type create m1.cirrOS 256 1 0 0 0 0 | # nova-manage instance_type create m1.cirrOS 256 1 0 0 0 0 | ||
- | glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare | + | glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare < ~/Downloads/cirros-0.3.0-x86_64-disk.img |
- | glance add name=Ubuntu-12.04 is_public=true container_format=ovf disk_format=qcow2 | + | glance add name=Ubuntu-12.04 is_public=true container_format=ovf disk_format=qcow2 < ~/Downloads/precise-server-cloudimg-amd64-disk1.img |
- | glance add name=f16-jeos is_public=true disk_format=qcow2 container_format=bare | + | glance add name=f16-jeos is_public=true disk_format=qcow2 container_format=bare < ~/Downloads/f16-x86_64-openstack-sda.qcow2 |
# nova-manage floating create --ip_range=192.168.127.6/31 | # nova-manage floating create --ip_range=192.168.127.6/31 | ||
# Get OpenStack demo auth | # Get OpenStack demo auth | ||
source $TOP_DIR/openrc demo demo | source $TOP_DIR/openrc demo demo | ||
- | glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare | + | glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare < ~/Downloads/cirros-0.3.0-x86_64-disk.img |
- | glance add name=Ubuntu-12.04 is_public=true container_format=ovf disk_format=qcow2 | + | glance add name=Ubuntu-12.04 is_public=true container_format=ovf disk_format=qcow2 < ~/Downloads/precise-server-cloudimg-amd64-disk1.img |
- | glance add name=f16-jeos is_public=true disk_format=qcow2 container_format=bare | + | glance add name=f16-jeos is_public=true disk_format=qcow2 container_format=bare < ~/Downloads/f16-x86_64-openstack-sda.qcow2 |
Line 365: | Line 359: | ||
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0 | nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0 | ||
</pre> | </pre> | ||
- | + | <p><br /> | |
- | + | </p> | |
- | + | <h3> lispd.conf </h3> | |
- | LISP is run with | + | <p>LISP is run with |
+ | </p> | ||
<pre> | <pre> | ||
sudo lispd -f /etc/lispd.conf | sudo lispd -f /etc/lispd.conf | ||
</pre> | </pre> | ||
- | Its configuration file is located in /etc/ and contains: | + | <p>Its configuration file is located in /etc/ and contains: |
- | + | </p> | |
<pre> | <pre> | ||
# lispd example config file | # lispd example config file | ||
Line 388: | Line 383: | ||
key-type = 1 # SHA1 | key-type = 1 # SHA1 | ||
key = foo%bar | key = foo%bar | ||
- | verify = off # on -- | + | verify = off # on --> lig(self) |
proxy-reply = on # ask ms to proxy reply | proxy-reply = on # ask ms to proxy reply | ||
} | } | ||
Line 422: | Line 417: | ||
</pre> | </pre> | ||
- | + | <h3> Bringing up a guest VM </h3> | |
- | |||
- | |||
- | + | [[Image:Instances_-_OpenStack_Dashboard1.png|850px|middle]] | |
+ | [[Image:Instance_Detail_-_OpenStack_Dashboard.png|850px|middle]] | ||
+ | |||
+ | |||
+ | |||
+ | <h3> ovs-lisp-openstask-scenario.sh </h3> | ||
<pre> | <pre> | ||
su - | su - | ||
vi /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | vi /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | ||
</pre> | </pre> | ||
- | |||
<pre> | <pre> | ||
#!/bin/bash | #!/bin/bash | ||
- | echo | + | echo "=== lakafosi: Stopping firewall and allowing everyone... ===" |
iptables -F | iptables -F | ||
iptables -X | iptables -X | ||
Line 447: | Line 444: | ||
iptables -P FORWARD ACCEPT | iptables -P FORWARD ACCEPT | ||
iptables -P OUTPUT ACCEPT | iptables -P OUTPUT ACCEPT | ||
- | echo | + | echo "=== lakafosi: Fixing the bridges... ===" |
VM_mac=fa:16:$(ifconfig vnet0 | grep HWaddr | awk '{print $5}'i | awk '{print substr($0,7)}') | VM_mac=fa:16:$(ifconfig vnet0 | grep HWaddr | awk '{print $5}'i | awk '{print substr($0,7)}') | ||
#Alternatively: | #Alternatively: | ||
#VM_mac=$(brctl showmacs br100 | grep fa:16: |awk '{print $2}'i) | #VM_mac=$(brctl showmacs br100 | grep fa:16: |awk '{print $2}'i) | ||
- | echo | + | echo "MAC address of guest VM: $VM_mac" |
brctl delif br100 vnet0 | brctl delif br100 vnet0 | ||
brctl delif br100 eth0 | brctl delif br100 eth0 | ||
Line 477: | Line 474: | ||
killall farpd | killall farpd | ||
farpd -i br-lisp 192.168.127.0/24 | farpd -i br-lisp 192.168.127.0/24 | ||
- | echo | + | echo "=== lakafosi: Bridge Status: ===" |
brctl show | brctl show | ||
ovs-vsctl show | ovs-vsctl show | ||
Line 483: | Line 480: | ||
ovs-dpctl show | ovs-dpctl show | ||
ovs-ofctl dump-flows br-lisp | ovs-ofctl dump-flows br-lisp | ||
- | echo | + | echo "=== lakafosi: Go and START lisp...! ===" |
- | #echo | + | #echo "lakafosi: Starting lisp..." |
- | #lispd -f /etc/lispd.conf & | + | #lispd -f /etc/lispd.conf & |
- | echo | + | echo "=== lakafosi: lisp status: ===" |
ps -ea | grep lisp; lsmod | grep lisp | ps -ea | grep lisp; lsmod | grep lisp | ||
</pre> | </pre> | ||
- | + | <p>To run it: | |
- | To run it: | + | </p> |
<pre> | <pre> | ||
chmod +x /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | chmod +x /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | ||
/home/stack/Downloads/ovs-lisp-openstask-scenario.sh | /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | ||
</pre> | </pre> | ||
- | + | <h2> Compute Openstack Node </h2> | |
- | + | <h3> localrc </h3> | |
- | + | <p>Located in /home/stack/devstack/ | |
- | + | </p> | |
- | + | ||
- | + | ||
- | Located in /home/stack/devstack/ | + | |
<pre> | <pre> | ||
#SERVICE_HOST_NAME=lakafosi-server-1 | #SERVICE_HOST_NAME=lakafosi-server-1 | ||
Line 541: | Line 535: | ||
#OFFLINE=True | #OFFLINE=True | ||
</pre> | </pre> | ||
- | + | <p><br /> | |
- | + | </p> | |
- | + | <h3> local.sh </h3> | |
- | Make sure no local.sh exists under /home/stack/devstack | + | <p>Make sure no local.sh exists under /home/stack/devstack |
+ | </p> | ||
<pre> | <pre> | ||
rm /home/stack/devstack/local.sh | rm /home/stack/devstack/local.sh | ||
</pre> | </pre> | ||
- | + | <h3> lispd.conf </h3> | |
- | + | <p>LISP is run with | |
- | LISP is run with | + | </p> |
<pre> | <pre> | ||
sudo lispd -f /etc/lispd.conf | sudo lispd -f /etc/lispd.conf | ||
</pre> | </pre> | ||
- | Its configuration file is located in /etc/ and contains: | + | <p>Its configuration file is located in /etc/ and contains: |
- | + | </p> | |
<pre> | <pre> | ||
# lispd example config file | # lispd example config file | ||
Line 570: | Line 565: | ||
key-type = 1 # SHA1 | key-type = 1 # SHA1 | ||
key = foo%bar | key = foo%bar | ||
- | verify = off # on -- | + | verify = off # on --> lig(self) |
proxy-reply = on # ask ms to proxy reply | proxy-reply = on # ask ms to proxy reply | ||
} | } | ||
Line 603: | Line 598: | ||
} | } | ||
</pre> | </pre> | ||
- | + | <p><br /> | |
- | + | </p> | |
- | + | <h3> ovs-lisp-openstask-scenario.sh </h3> | |
- | + | ||
<pre> | <pre> | ||
su - | su - | ||
vi /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | vi /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | ||
</pre> | </pre> | ||
- | |||
<pre> | <pre> | ||
#!/bin/bash | #!/bin/bash | ||
- | echo | + | echo "=== lakafosi: Stopping firewall and allowing everyone... ===" |
iptables -F | iptables -F | ||
iptables -X | iptables -X | ||
Line 625: | Line 618: | ||
iptables -P FORWARD ACCEPT | iptables -P FORWARD ACCEPT | ||
iptables -P OUTPUT ACCEPT | iptables -P OUTPUT ACCEPT | ||
- | echo | + | echo "=== lakafosi: Fixing the bridges... ===" |
VM_mac=fa:16:$(ifconfig vnet0 | grep HWaddr | awk '{print $5}'i | awk '{print substr($0,7)}') | VM_mac=fa:16:$(ifconfig vnet0 | grep HWaddr | awk '{print $5}'i | awk '{print substr($0,7)}') | ||
#Alternatively: | #Alternatively: | ||
#VM_mac=$(brctl showmacs br100 | grep fa:16: |awk '{print $2}'i) | #VM_mac=$(brctl showmacs br100 | grep fa:16: |awk '{print $2}'i) | ||
- | echo | + | echo "MAC address of guest VM: $VM_mac" |
brctl delif br100 vnet0 | brctl delif br100 vnet0 | ||
brctl delif br100 eth0 | brctl delif br100 eth0 | ||
Line 655: | Line 648: | ||
killall farpd | killall farpd | ||
farpd -i br-lisp 192.168.127.0/24 | farpd -i br-lisp 192.168.127.0/24 | ||
- | echo | + | echo "=== lakafosi: Bridge Status: ===" |
brctl show | brctl show | ||
ovs-vsctl show | ovs-vsctl show | ||
Line 661: | Line 654: | ||
ovs-dpctl show | ovs-dpctl show | ||
ovs-ofctl dump-flows br-lisp | ovs-ofctl dump-flows br-lisp | ||
- | echo | + | echo "=== lakafosi: Go and START lisp...! ===" |
- | #echo | + | #echo "lakafosi: Starting lisp..." |
- | #lispd -f /etc/lispd.conf & | + | #lispd -f /etc/lispd.conf & |
- | echo | + | echo "=== lakafosi: lisp status: ===" |
ps -ea | grep lisp; lsmod | grep lisp | ps -ea | grep lisp; lsmod | grep lisp | ||
</pre> | </pre> | ||
- | + | <p>To run it: | |
- | To run it: | + | </p> |
<pre> | <pre> | ||
chmod +x /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | chmod +x /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | ||
/home/stack/Downloads/ovs-lisp-openstask-scenario.sh | /home/stack/Downloads/ovs-lisp-openstask-scenario.sh | ||
</pre> | </pre> | ||
+ | |||
+ | |||
+ | <h1> Demo video </h1> | ||
+ | |||
+ | For a copy of the video, please contact lakafosi@cisco.com. |
Revision as of 01:20, 6 December 2012
This wiki describes all the steps necessary to setup a testbed of Openstack Folsom nodes (All-in-one node and Compute nodes), which rely on a LISP-enabled OpenVSwitch (OVS) bridge.
Contents |
Testbed Setup
Dependencies
The Openstack nodes are considered hereafter to be <a href="http://releases.ubuntu.com/precise/">Ubuntu 12.04.1 LTS (Precise Pangolin)</a> machines (running the 3.2.0-29 kernel version). The following dependencies need to be met.
sudo apt-get update sudo apt-get upgrade (NOT dist-upgrade) sudo apt-get install openssh-server vim git openssh-server bridge-utils build-essential git-core libssl-dev libconfuse-dev pkg-config autoconf libtool pkg-config gengetopt \ clang mosh cscope vim-gtk htop subversion tmux iotop dpatch automake1.9 python-support uml-utilities farpd sudo apt-get install linux-headers-3.2.0-29-generic # For KVM sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils virt-manager virtinst
In case you wish to also use Wireshark (with LISP dissector included):
$ sudo apt-get build-dep wireshark $ sudo apt-get install autoconf bison flex libtool libgtk2.0-dev libpcap-dev libc-ares-dev libsmi2-dev libgnutls-dev libgcrypt11-dev libkrb5-dev libcap2-bin libgeoip-dev libortp-dev libportaudio-dev $ mkdir ~/Downloads; cd ~/Downloads; wget http://wiresharkdownloads.riverbed.com/wireshark/src/wireshark-1.8.3.tar.bz2 $ tar -xvf wireshark-1.8.3.tar.bz2 -C /tmp $ cd /tmp/wireshark-1.8.3/ $ ./autogen.sh $ ./configure --enable-setcap-install $ make $ sudo make install $ sudo ldconfig $ wireshark &
Although this is simply a recommendation, the testbed has been successfully run with the following memory settings:
- 1.8GB for the All-In-One Node
- 1.2GB for the Compute Node
Network
Public + Management Network --+--------------------------+----- | | | | | | +------|-------------+ +--------|-------+ | (eth0) | | (eth0) | | | | | | | | | | | | | | 10.10.10.10 | | 10.10.11.10 | | | | | | | | [ br-lisp ] | | [br-lisp] | | | | | | | | | (vnet0) (vnet1) | | (vnet0) | | | | | | | | | VM VM | | VM | | 192.168. 192.168. | | 192.168. | | 127.2 127.4 | | 127.3 | +--------------------+ +----------------+ All-In-One Node Compute Node [] OVS-LISP bridge () Network interface
or
If instead of physical machines you plan on setting this testbed up with VMware Fusion, make sure the network is configured as Bridged (Ethernet).
Map Server (MS)
Just to get an idea of how the EID addresses are supposed to be associated with their corresponding RLOCs, you can have a look at the following MS configuration, which makes static assignments.
Under regular LISP operation you will not have to impose the following configuration. Rather, it is formed automatically and dynamically.
<?xml version="1.0" encoding="ISO-8859-1"?> <db> <eid prefix="0.0.0.0/3" act="2" ttl="5" /> <eid prefix="128.0.0.0/8" act="2" ttl="5" /> <eid prefix="171.0.0.0/8" act="2" ttl="5" /> <eid prefix="224.0.0.0/4" act="2" ttl="5" /> <eid prefix="192.168.127.3/32" ttl="1440" A="true"> <rloc> <address>10.10.10.10</address> <priority>1</priority> <weight>100</weight> <m_priority>255</m_priority> <m_weight>0</m_weight> <reachable>true</reachable> </rloc> </eid> <eid prefix="192.168.127.4/32" ttl="1440" A="true"> <rloc> <address>10.10.11.10</address> <priority>1</priority> <weight>100</weight> <m_priority>255</m_priority> <m_weight>0</m_weight> <reachable>true</reachable> </rloc> </eid> </db>
Installing the LISP-enabled OVS bridge
On both machines do the following. First download the corresponding code:
cd root/src/ git clone https://
First install lispmob:
cd root/src/lispmob-ovs/ make sudo make install sudo depmod -a
Afterward install the modified LISP-enabled OpenVswitch:
cd root/src/ovs-lisp/ ./boot.sh ./configure --with-linux=/lib/modules/`uname -r`/build make
cp root/src/lispmob-ovs/lisp_mod/Module.symvers root/src/ovs-lisp/datapath/linux/ sudo vi root/src/ovs-lisp/datapath/linux/Makefile.main.in
...and remove 'Module.symvers' from the 'clean' target (on line 33)
sudo reboot
Then, you will need to REMAKE and REINSTALL lisp and reboot!
cd root/src/lispmob-ovs make sudo make install sudo depmod -a cd root/src/ovs-lisp/ make sudo make install sudo depmod -a sudo reboot
Please retry the above in case you get the following message: "WARNING: "eid_locator4" [/root/src/ovs-lisp/datapath/linux/openvswitch.ko] undefined!"
In order to bring up the OVS bridge:
sudo root/src/ovs-scripts/ovs-start
which includes:
#!/bin/bash BUILD_DIR=root/src/ovs-lisp OVS_PATH=/usr/local rmmod -s openvswitch modprobe lisp insmod ${BUILD_DIR}/datapath/linux/openvswitch.ko mkdir -p ${OVS_PATH}/etc/openvswitch rm ${OVS_PATH}/etc/openvswitch/conf.db ovsdb-tool create /usr/local/etc/openvswitch/conf.db root/src/ovs-lisp/vswitchd/vswitch.ovsschema ${OVS_PATH}/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,manager_options \ --private-key=db:SSL,private_key \ --certificate=db:SSL,certificate \ --bootstrap-ca-cert=db:SSL,ca_cert \ --pidfile --detach ${OVS_PATH}/bin/ovs-vsctl --no-wait init ${OVS_PATH}/sbin/ovs-vswitchd --pidfile --detach ps -ea | grep ovs; ps -ea | grep lisp; lsmod | grep lisp; lsmod | grep ovs
You can verify everything is setup correctly by issuing:
ps -ea | grep ovs; ps -ea | grep lisp; lsmod | grep lisp; lsmod | grep ovs
which should yield an output similar to the following:
1404 ? 00:00:00 ovs_workq 1406 ? 00:00:00 ovsdb-server 1409 ? 00:00:00 ovs-vswitchd 1410 ? 00:00:00 ovs-vswitchd lisp 41286 1 openvswitch
Openstack
First create the stack user in order to install and run Openstack without any passwords requested.
su - useradd -U -G sudo -s /bin/bash -m stack; echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers su stack cd
This Openstack installation is based on <a href="http://devstack.org/">Devstack</a>.
It is important you stick with the stable Folsom version:
git clone -b stable/folsom https://github.com/openstack-dev/devstack.git
Alternatively,
git clone -b stable/folsom git://git.cisco.com/lakafosi/devstack.git
Additionally, you will have to download one more script for setting up the OVS-bridge and attaching the vnetX ports of the instantiated VMs, the lispd.conf control plane configuration file of LISP and a few cloud qcow2 (Ubuntu and Fedora) images (to use in addition to the preloaded tiny <a href="https://launchpad.net/cirros/+download">CirrOS cloud guest</a>).
mkdir /home/stack/Downloads mv /* /home/stack/devstack/Downloads/* /home/stack/Downloads
All-In-One Openstack Node
localrc
Located in /home/stack/devstack/
#HOST_NAME=$(hostname) #SERVICE_HOST_NAME=lakafosi-server-1 SERVICE_HOST=10.10.10.10 HOST_IP=10.10.10.10 FIXED_RANGE=192.168.127.0/24 MULTI_HOST=True SCHEDULER=nova.scheduler.simple.SimpleScheduler Q_HOST=$SERVICE_HOST MYSQL_HOST=$SERVICE_HOST RABBIT_HOST=$SERVICE_HOST GLANCE_HOSTPORT=$SERVICE_HOST:9292 KEYSTONE_AUTH_HOST=$SERVICE_HOST KEYSTONE_SERVICE_HOST=$SERVICE_HOST CINDER_BRANCH=stable/folsom NOVA_BRANCH=stable/folsom SWIFT_BRANCH=stable/folsom GLANCE_BRANCH=stable/folsom KEYSTONE_BRANCH=stable/folsom HORIZON_BRANCH=stable/folsom #QUANTUM_BRANCH=stable/folsom # password MYSQL_PASSWORD=mysql RABBIT_PASSWORD=rabbit SERVICE_TOKEN=service SERVICE_PASSWORD=admin ADMIN_PASSWORD=admin # Uncomment the following line to save time by skipping checking of packages that need to be installed #OFFLINE=True
local.sh
This is the script run right after stack.sh, i.e. the Openstack installation. It is located in /home/stack/devstack/ .
#!/usr/bin/env bash # Sample ``local.sh`` for user-configurable tasks to run automatically # at the sucessful conclusion of ``stack.sh``. # Keep track of the devstack directory TOP_DIR=$(cd $(dirname "$0") && pwd) # Import common functions source $TOP_DIR/functions # Use openrc + stackrc + localrc for settings source $TOP_DIR/stackrc # Destination path for installation ``DEST`` DEST=${DEST:-/opt/stack} # lakafosi for i in `seq 2 5`; do /opt/stack/nova/bin/nova-manage fixed reserve 192.168.127.$i; done # Get OpenStack admin auth source $TOP_DIR/openrc admin admin # nova-manage instance_type create m1.cirrOS 256 1 0 0 0 0 glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare < ~/Downloads/cirros-0.3.0-x86_64-disk.img glance add name=Ubuntu-12.04 is_public=true container_format=ovf disk_format=qcow2 < ~/Downloads/precise-server-cloudimg-amd64-disk1.img glance add name=f16-jeos is_public=true disk_format=qcow2 container_format=bare < ~/Downloads/f16-x86_64-openstack-sda.qcow2 # nova-manage floating create --ip_range=192.168.127.6/31 # Get OpenStack demo auth source $TOP_DIR/openrc demo demo glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare < ~/Downloads/cirros-0.3.0-x86_64-disk.img glance add name=Ubuntu-12.04 is_public=true container_format=ovf disk_format=qcow2 < ~/Downloads/precise-server-cloudimg-amd64-disk1.img glance add name=f16-jeos is_public=true disk_format=qcow2 container_format=bare < ~/Downloads/f16-x86_64-openstack-sda.qcow2 # Import keys from the current user into the default OpenStack user (usually # ``demo``) # Get OpenStack auth source $TOP_DIR/openrc # Add first keypair found in localhost:$HOME/.ssh for i in $HOME/.ssh/id_rsa.pub $HOME/.ssh/id_dsa.pub; do if [[ -r $i ]]; then nova keypair-add --pub_key=$i `hostname` break fi done # Create my Flavor # --------------- # Get OpenStack admin auth source $TOP_DIR/openrc admin admin # Name of new flavor # set in ``localrc`` with ``DEFAULT_INSTANCE_TYPE=m1.micro`` MI_NAME=m1.cirrOS # Create micro flavor if not present if [[ -z $(nova flavor-list | grep $MI_NAME) ]]; then nova-manage instance_type create m1.cirrOS 256 1 0 0 0 0 # nova flavor-create $MI_NAME 6 128 0 1 fi # Other Uses # ---------- # Add tcp/22 and icmp to default security group nova secgroup-add-rule default tcp 22 22 0.0.0.0/0 nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
lispd.conf
LISP is run with
sudo lispd -f /etc/lispd.conf
Its configuration file is located in /etc/ and contains:
# lispd example config file debug = on map-request-retries = 2 # send 2 before giving up # Encapsulated Map-Requests are sent to this map resolver map-resolver = 10.32.164.141 # Register to this map server map-server { address = 10.32.164.141 key-type = 1 # SHA1 key = foo%bar verify = off # on --> lig(self) proxy-reply = on # ask ms to proxy reply } # Encapsulate packets for non-LISP sites to this Proxy-ETR proxy-etr { address = 149.20.48.60 priority = 1 weight = 100 } # IPv4 EID of the mobile node database-mapping { eid-prefix = 192.168.127.3/32 # for Openstack AllInOne Node # iid = 0 interface = br-lisp priority = 1 weight = 100 } # List of PITRs to SMR on handover proxy-itrs = { 69.31.31.98, # eqx-ash-pxtr 149.20.48.60, # isc-pxtr 198.6.255.37, # asp-pxtr 129.250.1.63, # ntt-amer-pxtr 217.8.98.33, # intouch-pxtr-1 217.8.98.35, # intouch-pxtr-2 193.162.145.46, # tdc-pxtr 158.38.1.92, # uninett-pxtr 203.181.249.172 # apan-pxtr }
Bringing up a guest VM
ovs-lisp-openstask-scenario.sh
su - vi /home/stack/Downloads/ovs-lisp-openstask-scenario.sh
#!/bin/bash echo "=== lakafosi: Stopping firewall and allowing everyone... ===" iptables -F iptables -X iptables -t nat -F iptables -t nat -X iptables -t mangle -F iptables -t mangle -X iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT echo "=== lakafosi: Fixing the bridges... ===" VM_mac=fa:16:$(ifconfig vnet0 | grep HWaddr | awk '{print $5}'i | awk '{print substr($0,7)}') #Alternatively: #VM_mac=$(brctl showmacs br100 | grep fa:16: |awk '{print $2}'i) echo "MAC address of guest VM: $VM_mac" brctl delif br100 vnet0 brctl delif br100 eth0 #ifconfig br100 0.0.0.0 #ifconfig br100 0.0.0.0 #ifconfig br100 192.168.127.1 netmask 255.255.255.0 ifconfig br100 down brctl delbr br100 ovs-vsctl -- --if-exists del-br br-lisp ovs-vsctl add-br br-lisp ovs-vsctl add-port br-lisp vnet0 #AllInOne ovs-vsctl add-port br-lisp lisp0 -- set Interface lisp0 type=lisp options:remote_ip=10.10.11.10 ovs-vsctl add-port br-lisp eth0 ovs-ofctl del-flows br-lisp ovs-ofctl add-flow br-lisp priority=3,dl_dst=00:11:22:ee:ee:ee,action=mod_dl_dst:$VM_mac,NORMAL ovs-ofctl add-flow br-lisp priority=2,in_port=1,dl_type=0x0806,action=NORMAL ovs-ofctl add-flow br-lisp priority=1,in_port=1,dl_type=0x0800,vlan_tci=0,nw_src=192.168.127.0/24,action=output:2 ovs-ofctl add-flow br-lisp priority=0,action=NORMAL ifconfig vnet0 up ifconfig eth0 0.0.0.0 #AllInOne ifconfig br-lisp 10.10.10.10 netmask 255.255.255.192 route add default gw 10.10.1.1 killall farpd farpd -i br-lisp 192.168.127.0/24 echo "=== lakafosi: Bridge Status: ===" brctl show ovs-vsctl show ovs-appctl fdb/show br-lisp ovs-dpctl show ovs-ofctl dump-flows br-lisp echo "=== lakafosi: Go and START lisp...! ===" #echo "lakafosi: Starting lisp..." #lispd -f /etc/lispd.conf & echo "=== lakafosi: lisp status: ===" ps -ea | grep lisp; lsmod | grep lisp
To run it:
chmod +x /home/stack/Downloads/ovs-lisp-openstask-scenario.sh /home/stack/Downloads/ovs-lisp-openstask-scenario.sh
Compute Openstack Node
localrc
Located in /home/stack/devstack/
#SERVICE_HOST_NAME=lakafosi-server-1 SERVICE_HOST=10.10.10.10 HOST_IP=10.10.11.10 MULTI_HOST=True #FIXED_RANGE=192.168.127.0/24 # compute node ENABLED_SERVICES=n-cpu,n-net,n-api,n-vol,rabbit #disable_all_services #enable_service rabbit n-cpu quantum q-agt n-vol n-api Q_HOST=$SERVICE_HOST MYSQL_HOST=$SERVICE_HOST RABBIT_HOST=$SERVICE_HOST GLANCE_HOSTPORT=$SERVICE_HOST:9292 KEYSTONE_AUTH_HOST=$SERVICE_HOST KEYSTONE_SERVICE_HOST=$SERVICE_HOST CINDER_BRANCH=stable/folsom NOVA_BRANCH=stable/folsom SWIFT_BRANCH=stable/folsom GLANCE_BRANCH=stable/folsom KEYSTONE_BRANCH=stable/folsom HORIZON_BRANCH=stable/folsom #QUANTUM_BRANCH=stable/folsom # password MYSQL_PASSWORD=mysql RABBIT_PASSWORD=rabbit SERVICE_TOKEN=service SERVICE_PASSWORD=admin ADMIN_PASSWORD=admin # Uncomment the following line to save time by skipping checking of packages that need to be installed #OFFLINE=True
local.sh
Make sure no local.sh exists under /home/stack/devstack
rm /home/stack/devstack/local.sh
lispd.conf
LISP is run with
sudo lispd -f /etc/lispd.conf
Its configuration file is located in /etc/ and contains:
# lispd example config file debug = on map-request-retries = 2 # send 2 before giving up # Encapsulated Map-Requests are sent to this map resolver map-resolver = 10.32.164.141 # Register to this map server map-server { address = 10.32.164.141 key-type = 1 # SHA1 key = foo%bar verify = off # on --> lig(self) proxy-reply = on # ask ms to proxy reply } # Encapsulate packets for non-LISP sites to this Proxy-ETR proxy-etr { address = 149.20.48.60 priority = 1 weight = 100 } # IPv4 EID of the mobile node database-mapping { eid-prefix = 192.168.127.4/32 # for Openstack Compute Node # iid = 0 interface = br-lisp priority = 1 weight = 100 } # List of PITRs to SMR on handover proxy-itrs = { 69.31.31.98, # eqx-ash-pxtr 149.20.48.60, # isc-pxtr 198.6.255.37, # asp-pxtr 129.250.1.63, # ntt-amer-pxtr 217.8.98.33, # intouch-pxtr-1 217.8.98.35, # intouch-pxtr-2 193.162.145.46, # tdc-pxtr 158.38.1.92, # uninett-pxtr 203.181.249.172 # apan-pxtr }
ovs-lisp-openstask-scenario.sh
su - vi /home/stack/Downloads/ovs-lisp-openstask-scenario.sh
#!/bin/bash echo "=== lakafosi: Stopping firewall and allowing everyone... ===" iptables -F iptables -X iptables -t nat -F iptables -t nat -X iptables -t mangle -F iptables -t mangle -X iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT echo "=== lakafosi: Fixing the bridges... ===" VM_mac=fa:16:$(ifconfig vnet0 | grep HWaddr | awk '{print $5}'i | awk '{print substr($0,7)}') #Alternatively: #VM_mac=$(brctl showmacs br100 | grep fa:16: |awk '{print $2}'i) echo "MAC address of guest VM: $VM_mac" brctl delif br100 vnet0 brctl delif br100 eth0 #ifconfig br100 0.0.0.0 #ifconfig br100 0.0.0.0 #ifconfig br100 192.168.127.1 netmask 255.255.255.0 ifconfig br100 down brctl delbr br100 ovs-vsctl -- --if-exists del-br br-lisp ovs-vsctl add-br br-lisp ovs-vsctl add-port br-lisp vnet0 #Compute ovs-vsctl add-port br-lisp lisp0 -- set Interface lisp0 type=lisp options:remote_ip=10.10.10.10 ovs-vsctl add-port br-lisp eth0 ovs-ofctl del-flows br-lisp ovs-ofctl add-flow br-lisp priority=3,dl_dst=00:11:22:ee:ee:ee,action=mod_dl_dst:$VM_mac,NORMAL ovs-ofctl add-flow br-lisp priority=2,in_port=1,dl_type=0x0806,action=NORMAL ovs-ofctl add-flow br-lisp priority=1,in_port=1,dl_type=0x0800,vlan_tci=0,nw_src=192.168.127.0/24,action=output:2 ovs-ofctl add-flow br-lisp priority=0,action=NORMAL ifconfig vnet0 up ifconfig eth0 0.0.0.0 #Compute ifconfig br-lisp 10.10.11.10 netmask 255.255.255.192 route add default gw 10.10.1.1 killall farpd farpd -i br-lisp 192.168.127.0/24 echo "=== lakafosi: Bridge Status: ===" brctl show ovs-vsctl show ovs-appctl fdb/show br-lisp ovs-dpctl show ovs-ofctl dump-flows br-lisp echo "=== lakafosi: Go and START lisp...! ===" #echo "lakafosi: Starting lisp..." #lispd -f /etc/lispd.conf & echo "=== lakafosi: lisp status: ===" ps -ea | grep lisp; lsmod | grep lisp
To run it:
chmod +x /home/stack/Downloads/ovs-lisp-openstask-scenario.sh /home/stack/Downloads/ovs-lisp-openstask-scenario.sh
Demo video
For a copy of the video, please contact lakafosi@cisco.com.