OpenStack: Icehouse: full HA

From DocWiki

Jump to: navigation, search

Contents

Background

High-Availability Introduction

Most OpenStack deployments are maturing from evaluation-level environments to highly available and highly scalable environments to support production applications and services. The Cisco OpenStack High-Availability Guide provides users instructions for deploying an active/active, highly scalable OpenStack environment. The architecture consists of the following components used to provide high-availability to OpenStack services:

  • MySQL Galera is synchronous multi-master cluster technology for MySQL/InnoDB databases that includes features such as:
    • Synchronous replication
    • Active/active multi-master topology
    • Read and write to any cluster node
    • True parallel replication, on row level
    • Direct client connections, native MySQL look & feel
    • No slave lag or integrity issues
  • Several OpenStack services utilize a message queuing system to send and receive messages between software components. The Cisco reference architecture leverages RabbitMQ as the messaging system since it is most commonly used within the OpenStack community. RabbitMQ Clustering and RabbitMQ Mirrored Queues provide active/active and highly scalable message queuing for OpenStack services.
    • RabbitMQ Clustering: If your RabbitMQ broker consists of a single node, then a failure of that node will cause downtime, temporary unavailability of service, and potentially loss of messages. A cluster of RabbitMQ nodes can be used to construct your RabbitMQ broker. Clustering RabbitMQ nodes are resilient to the loss of individual nodes in terms of the overall availability of service. All data/state required for the operation of a RabbitMQ broker is replicated across all nodes, for reliability and scaling. An exception to this are message queues, which by default reside on the node that created them, though they are visible and reachable from all nodes.
    • RabbitMQ Mirrored Queues: While exchanges and bindings survive the loss of individual nodes, message queues and their messages do not. This is because a queue and its contents reside on exactly one node, thus the loss of a node will render its queues unavailable. To solve these various problems, RabbitMQ has developed active/active high-availability for message queues. This works by allowing queues to be mirrored on other nodes within a RabbitMQ cluster. The result is that should one node of a cluster fail, the queue can automatically switch to one of the mirrors and continue to operate, with no unavailability of service. This solution still requires a RabbitMQ Cluster, which means that it will not cope seamlessly with network partitions within the cluster and, for that reason, is not recommended for use across a WAN (though of course, clients can still connect from as near and as far as needed).
  • HAProxy and Keepalived provide load-balancing betwwen clients and OpenStack API Endpoints.
    • HAProxy is a free, very fast and reliable software solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. HAProxy implements an event-driven, single-process model which enables support for a high number of simultaneous connections.
    • Keepalived is a routing software written in C. The main goal of the Keepalived project is to provide simple and robust facilities for load-balancing and high-availability to Linux systems and Linux-based infrastructures. Load-balancing frameworks rely on the well-known and widely used Linux Virtual Server (IPVS) kernel module providing Layer4 load-balancing. Keepalived implements a set of checkers to dynamically and adaptively maintain and manage load-balanced server pool according their health. On the other hand high-availability is achieved by VRRP protocol. VRRP is a fundamental brick for router fail-over.
  • Multiple Neutron L3 and DHCP Agents Blueprint, as it states, allows multiple Neutron Layer-3 and DHCP Agents to be deployed for high-availability and scalability purposes. At this time, multiple DHCP Agents can service a Neutron network, however, only a single L3 Agent can service one Neutron network at a time. Therefore, the L3 Agent is a single point of failure and is not included in the Cisco High-Availability Deployment Guide. Neutron Provider Network Extensions are used to map physical data center networks to Neutron networks. In this deployment model, Neutron relies on the physical data center to provide Layer-3 high-availability instead of the L3 Agent.
  • Glance uses Swift as the back-end to storage OpenStack images. Just as with the rest of the OpenStack API's, HAProxy and Keepalived provide high-availability to the Glance API and Registry endpoints.
  • Swift: Multiple Swift Proxy nodes are used to provide high-availability to the Swift proxy service. Replication provides high-availability to data stored within a Swift object-storage system. The replication processes compare local data with each remote copy to ensure they all contain the latest version. Object replication uses a hash list to quickly compare subsections of each partition, and container and account replication use a combination of hashes and shared high water marks.

Critical Reminders

The most common OpenStack HA deployment issues are either incorrect configuration files or not deploying the nodes in the proper order. To save you from future troubleshooting steps, ENSURE that you deploy the nodes in the order described within the document and verify the accuracy of all configuration files. You will likely be using your own IP addressing and passwords in your setup and it is critical to ensure any variations from this guide are fully understood.

Do not configure RAID on the hard disks of Swift Storage Nodes. Swift performs better without RAID and disk redundancy is unneeded since Swift protects the data through replication. Therefore, if a RAID Controller manages the hard disks, ensure you present each of the hard disks independently. Our example uses disk /dev/sda for the Operating System installation and disks /dev/sdb-/dev/sdf for Swift storage. Please remember to modify these definitions based on your specific deployment environment. Additional Swift considerations and tuning information can be found here.

Compute Nodes run Cinder Volume to provide block storage services to Instances. The default Cinder driver (volume_driver=nova.volume.driver.ISCSIDriver) is an iSCSI solution that employs the use of Linux Logical Volume Manager (LVM). Therefore, you must create an LVM Volume Group either during the Ubuntu Precise installation or afterwards. The name of the LVM Volume Group must match the volume_group definition in cinder.conf. Our example uses the name cinder-volumes (default value) for the LVM Volume Group and associated cinder.conf volume_group name.

The password used in our examples is keystone_admin. Every account, service and configuration file uses this one password. You will want to change this in your setup and you certainly want to use a strong password and a different password for each account/service if this system is going into production.

Please read the release notes for the release you are installing. The release notes contain important information about limitations, features, and how to use snapshot repositories if you're installing an older release.


Operating System

The operating system used for this installation is Ubuntu 14.04 LTS (Trusty).

Server Requirements

Our deployment uses 13 Cisco UCS C-series servers to serve the roles of Controller, Compute, Load-Balancer and Swift Proxy/Storage. The environment scales linearly, therefore individual nodes can be added to increase capacity for any particular OpenStack service. The five distinct node types used in this document are:

3 Controller Nodes- Runs Nova API, Nova Conductor, Nova Consoleauth, Nova Scheduler, Neutron Server, Neutron Plugin OVS, Neutron DHCP Agent, Glance API/Registry, Keystone, Cinder API, Cinder Scheduler, RabbitMQ Server, MySQL Server WSREP and Galera.

  • Provides management functionality of the OpenStack environment.

3 Compute Nodes- Runs Nova Compute, Neutron OVS Agent, Cinder Volume and TGT services.

  • Provides the hypervisor role for running Nova instances (Virtual Machines) and presents LVM volumes for Cinder block storage.

2 Load-Balancer Nodes- Runs HAProxy and Keepalived to load-balance traffic across Controller and Swift Proxy clusters.

2 Swift Proxy Nodes- The Proxy Node is responsible for tying together users and their data within the the Swift object storage system. For each request, it will look up the location of the account, container or object in the Swift ring and route the request accordingly. The public API is also exposed by Proxy Node.

3 Swift Storage Nodes- Each Storage Nodes contains Swift object, container, and account services. At a very high-level, these are the servers that contain the user data and perform replication among one another to keep the system in a consistent state.

Networking Requirements

The OpenStack HA environment uses five separate networks. Three of the five networks are used by Tenants. Three tenant networks are being used as an example, and thus the tenant networks can be increased or decreased based on your deployment needs. Connectivity within Tenants uses Neutron with the Open vSwitch (OVS) plugin and Provider Network Extensions. Provider Network Extensions allow cloud administrators to create OpenStack networks that map directly to physical networks in the data center and support local, VLAN and GRE deployment models. Our example uses the Provider VLAN networking model. The network details are as follows:

1 Management Network

  • This network is used to perform management functions against the node. For example, SSH'ing to the nodes to change a configuration setting. The network is also used for lights-out management using the CIMC interface of the UCS servers. Lastly, OpenStack API's and the Horizon web dashboard is associated to this network.
  • An IP address for each node is required for this network. If using lights-out management such as CIMC, each node will require 2 addresses from this network.
  • This network typically employs private (RFC1918).

3 Tenant Networks

  • These networks are used to provide connectivity to Instances. Since Neutron Provider Networking Extensions are being used, it is common to give tenants direct access to a "public" network that can be used to reach the Internet.
  • Compute Nodes will have an interface attached to this network. Since the Compute Node interfaces that attach to this network are managed by OVS, they should not contain an IP address.
  • This network typically employs publicly routable IP addressing if external NAT'ing is not used upstream towards the Internet edge (Note: in this document all IP addressing for all interfaces comes out of various private addressing blocks).

1 Storage Network

  • This network is used for providing separate connectivity between Swift Proxy and Storage Nodes. This ensures storage traffic is not interfering with Instance traffic.
  • This network typically employs private (RFC1918) IP addressing.

Reference Physical Network Configuration

Our reference architecture uses one Nexus 5000 per rack as a layer-2 top-of-rack switch. Each top of rack switch is connected to a Nexus 7000 switch that acts as a layer 3 gateway. It is recommended to use multiple layer-3 devices (i.e. Nexus 7000) and VRRP for redundancy.

Create VLANs on TOR L2 switch and L3 GW:

vlan 220
  name pod1_mgt
vlan 222
  name pod1_swift_storage
vlan 223
  name pod1-qtm--net1
vlan 224
  name pod1-qtm--net2
vlan 225
  name pod1-qtm--net3

Configure physical interfaces on the Nexus 5000 and 7000 switches. Note: This configuration applies to every interface within the path between the gateway interface and OpenStack nodes. Allowed VLANs can be further restricted based on OpenStack node type if needed:

interface <NAME/NUMBER>
  switchport mode trunk
  switchport trunk native vlan 220
  switchport trunk allowed vlan 220-230
  spanning-tree port type edge

Configure layer-3 gateway interfaces. VRRP configuration should be added to these interfaces if and additional physical layer-3 gateway is used. Notice that no layer-3 gateway interface has been created for the Swift storage network:

interface Vlan220
  ip address 192.168.220.1/24
  no shutdown

interface Vlan223
  ip address 192.168.223.1/24
  description ### Daneyon- Neutron Provider VLAN Deployment ###
  no shutdown

interface Vlan224
  ip address 192.168.224.1/24
  description ### Daneyon- Neutron Provider VLAN Deployment ###
  no shutdown

interface Vlan225
  ip address 192.168.225.1/24
  description ### Daneyon- Neutron Provider VLAN Deployment ###
  no shutdown

Figure 1 is used to help visualize the network deployment and to act as a reference for configuration steps within the document. It is highly recommend to print the diagram so it can easily be referenced throughout the installation process.


Figure 1:OpenStack HA Network Design Details

Havana manual ha network design details v1 0.png







Other Network Services

  • DNS: In this setup an external DNS server (192.168.26.186) is used for name resolution of OpenStack nodes and external name resolution. If DNS is not being used, the /etc/hosts file should include the following for all nodes:
127.0.0.1	localhost
192.168.220.40  control.dmz-pod2.lab		control
192.168.220.41  control01.dmz-pod2.lab	        control01
192.168.220.42  control02.dmz-pod2.lab  	control02
192.168.220.43  control03.dmz-pod2.lab  	control03
192.168.220.60  swiftproxy.dmz-pod2.lab	        swiftproxy
192.168.220.61  swiftproxy01.dmz-pod2.lab	swiftproxy01
192.168.220.62  swiftproxy02.dmz-pod2.lab	swiftproxy02
192.168.220.51  compute01.dmz-pod2.lab          compute01
192.168.220.52  compute02.dmz-pod2.lab          compute02
192.168.220.53  compute03.dmz-pod2.lab          compute03
  • NTP: In this setup an external NTP server(s) is used for time synchronization.
  • Physical Network Switches: Each node in the reference deployment is physically connected to two Cisco Nexus switches that are configured for layer 2 only. eth0 of each node connects to Nexus switch 1 and eth1 of each node is connected to Nexus switch 2. Trunking is configured on each interface connecting to the eth0 and eth1 NICs of each node. Trunking is also configured between the switches and to the upstream physical routers.
  • Physical Network Routers: Each Nexus switch is connected to a pair of upstream ASR 9000 routers. The routers provide layer 3 functionality, including first-hop redundancy (using VRRP) for all OpenStack networks. Keep in mind that the HA architecture uses Neutron Provider Networking Extensions, so tenant networks will also use ASR 9000 VRRP addresses as gateways.


Note: Upstream routers/aggregation layer switches will most likely be terminating the Layer-3 VLAN interfaces. If these interfaces are deployed in a redundant fashion with a First Hop Redundancy Protocol such as HSRP or VRRP, then you should be careful of the IP addresses assigned to the physical L3 switches/routers as they may conflict with the IP address of the Neutron router's public subnet (.3 by default). For example, if you are using HSRP and you have .1 as the standby IP address, .2 as the first L3 switch IP and .3 as the second L3 switch IP, you will receive a duplicate IP address error on the second L3 switch. This can be worked around by using high-order IPs on your upstream L3 device or altering the Neutron subnet configuration at the time of creation to have an IP starting range higher than the physical switches/routers are using (i.e. .4 and higher). Our example uses an IP allocation range that starts with .10 to avoid this issue.

Installation

The installation of the nodes should be in the following order:

  1. Load-Balancer Nodes- slb01 and slb02
  2. Swift Storage Nodes- swift01, swift02 and swift03
  3. Swift Proxy Nodes- swiftproxy01 and swiftproxy02
  4. Controller Nodes- control01, control02 and control03
  5. Compute Nodes- compute01, compute02 and compute03

General Installation Steps for All Nodes

Build Node Configuration

  • Follow the build node setup instructions from the [1]. After install.sh(Step 10) completes you will need to configure the data model to reflect your topology.
  • Update /etc/puppet/data/role_mappings.yaml to reflect your short hostname for build, control, compute swift-proxy, swift-storage and load-balancer roles.
build-server: build

compute-server01: compute_without_osd
compute-server02: compute_without_osd
compute-server03: compute_without_osd

control01: controller_without_mon
control02: controller_without_mon
control03: controller_without_mon

load-balancer01: load_balancer
load-balancer02: load_balancer

swift-proxy01: swift_proxy
swift-proxy02: swift_proxy

swift-storage01: swift_storage
swift-storage02: swift_storage
swift-storage03: swift_storage


Configuration and Customization

At this point, the data model has been installed and any required customizations should be made to the data model. Listed below are specific changes required for setting up the network.

In /etc/puppet/data/hiera_data/user.common.yaml

Since this is a full_ha setup, all the controller address variables listed below should be pointing to the the Virtual Ip of the controller and not the real IP of any of the control nodes.

Required Config

######### Node Addresses ##############
# Change the following to the short host name you have given your build node.
# This name should be in all lower case letters due to a Puppet limitation
# (refer to http://projects.puppetlabs.com/issues/1168).
build_node_name: build-server

# Change the following to the host name you have given to your control
# node.  This name should be in all lower case letters due to a Puppet
# limitation (refer to http://projects.puppetlabs.com/issues/1168).
coe::base::controller_hostname: control-server

# The IP address to be used to connect to Horizon and external
# services on the control node.  In the compressed_ha or full_ha scenarios,
# this will be an address to be configured as a VIP on the HAProxy
# load balancers, not the address of the control node itself.
controller_public_address: 192.168.242.10

# The IP address used for management functions (such as monitoring)
# on the control node.  In the compressed_ha or full_ha scenarios, this will
# be an address to be configured as a VIP on the HAProxy
# load balancers, not the address of the control node itself.
controller_admin_address: 192.168.242.10

# Controller public url
controller_public_url: "http://192.168.242.10:5000"

# Controller admin url
controller_admin_url: "http://192.168.242.10:35357"

# Controller admin url
controller_internal_url: "http://192.168.242.10:35357"


Connectivity Config

# This domain name will be the name your build and compute nodes use for the
# local DNS.  It doesn't have to be the name of your corporate DNS - a local
# DNS server on the build node will serve addresses in this domain - but if
# it is, you can also add entries for the nodes in your corporate DNS
# environment they will be usable *if* the above addresses are routeable
# from elsewhere in your network.
domain_name: domain.name

########### NTP Configuration ############
# Change this to the location of a time server or servers in your
# organization accessible to the build server.  The build server will
# synchronize with this time server, and will in turn function as the time
# server for your OpenStack nodes.
ntp_servers:
  - 1.pool.ntp.org

# Control node interfaces.
# internal_ip be used for the ovs local_ip setting for GRE tunnels.

# This sets the IP for the private(internal) interface of controller nodes
# (which is predefined already in $controller_node_internal, and the internal
# interface for compute nodes.  It is generally also the IP address
# used in Cobbler node definitions.
internal_ip: "%{ipaddress_eth3}"

# The external_interface is used to provide a Layer2 path for
# the l3_agent external router interface.  It is expected that
# this interface be attached to an upstream device that provides
# a L3 router interface, with the default router configuration
# assuming that the first non "network" address in the external
# network IP subnet will be used as the default forwarding path
# if no more specific host routes are added.
external_interface: eth2

# The public_interface will have an IP address reachable by
# all other nodes in the openstack cluster.  This address will
# be used for API Access, for the Horizon UI, and as an endpoint
# for the default GRE tunnel mechanism used in the OVS network
# configuration.
public_interface: eth1

# The interface used for VM networking connectivity.  This will usually
# be set to the same interface as public_interface.
private_interface: eth1

Cobbler Configuration

If the build node runs a cobbler server, then modify these parameters as per the testbed.

### Cobbler config
# The IP address of the node on which Cobbler will be installed and
# on which it will listen.
cobbler_node_ip: 192.168.242.10

# The subnet address of the subnet on which Cobbler should serve DHCP
# addresses.
node_subnet: '192.168.242.0'

# The netmask of the subnet on which Cobbler should serve DHCP addresses.
node_netmask: '255.255.255.0'

# The default gateway that should be provided to DHCP clients that acquire
# an address from Cobbler.
node_gateway: '192.168.242.1'

# The admin username and crypted password used to authenticate to Cobbler.
admin_user: localadmin
password_crypted: $6$UfgWxrIv$k4KfzAEMqMg.fppmSOTd0usI4j6gfjs0962.JXsoJRWa5wMz8yQk4SfInn4.WZ3L/MCt5u.62tHDGB36EhiKF1

# Cobbler can instruct nodes being provisioned to start a Puppet agent
# immediately upon bootup.  This is generally desirable as it allows
# the node to immediately begin configuring itself upon bootup without
# further human intervention.  However, it may be useful for debugging
# purposes to prevent Puppet from starting automatically upon bootup.
# If you want Puppet to run automatically on bootup, set this to true.
# Otherwise, set it to false.
autostart_puppet: true

# Cobbler installs a minimal install of Ubuntu. Specify any additional
# packages which should be part of the base install on top of the minimal
# install set of packages
packages: 'openssh-server vim vlan lvm2 ntp'

# If you are using Cisco UCS servers managed by UCSM, set the port on
# which Cobbler should connect to UCSM in order to power nodes off and on.
# If set to 443, the connection will use SSL, which is generally
# desirable and is usually enabled on UCS systems.
ucsm_port: 443

# The name of the hard drive on which Cobbler should install the operating
# system.
install_drive: /dev/sda

Passwords

### The following are passwords and usernames used for
### individual services.  You may wish to change the passwords below
### in order to better secure your installation.
cinder_db_password: cinder_pass
glance_db_password: glance_pass
keystone_db_password: key_pass
nova_db_password: nova_pass
network_db_password:   quantum_pass
database_root_password: mysql_pass
cinder_service_password: cinder_pass
glance_service_password: glance_pass
nova_service_password: nova_pass
ceilometer_service_password: ceilometer_pass
admin_password: Cisco123
admin_token: keystone_admin_token
network_service_password: quantum_pass
rpc_password: openstack_rabbit_password
metadata_shared_secret: metadata_shared_secret
horizon_secret_key: horizon_secret_key
ceilometer_metering_secret: ceilometer_metering_secret
ceilometer_db_password: ceilometer
heat_db_password: heat
heat_service_password: heat_pass
heat::engine::auth_encryption_key: 'notgood but just long enough i think'

# Set this parameter to use a single secret for the Horizon secret
# key, neutron agents, Nova API metadata proxies, swift hashes,etc.
# This prevents you from needing to specify individual secrets above,
# but has some security implications in that all services are using
# the same secret (creating more vulnerable services if it should be
# compromised).
secret_key: secret

# Set this parameter to use a single password for all the services above.
# This prevents you from needing to specify individual passwords above,
# but has some security implications in that all services are using
# the same password (creating more vulnerable services if it should be
# compromised).
password: password123

Disk Partitioning

When deploying nodes with Cobbler, you can control some aspects of disk partitioning by placing the following directives in /etc/puppet/data/hiera_data/user.common.yaml. If you do not want a separate /var from /, set enable_var to false. If you do not want extra disk space set aside in an LVM volume to use for Cinder volumes via iSCSI, set enable_vol_space to false (you likely want this true if you want to use iscsi volumes on compute nodes). You can specify the minimum sizing of /var and / partitions using the var_part_size and root_part_size directives, respectively. The values should be specified in units of megabytes.

#### Disk partitioning options ###########
# expert_disk is needed for fine-grained configuration of drive layout
# during Cobbler-driven installs. It is also required when installing
# on large drives that require GPT
expert_disk: true

# The /var directory is where logfiles and instance data are stored
# on disk.  If you wish to have /var on it's own partition (considered
# a best practice), set enable_var to true.
enable_var: true

# The Cinder volume service can make use of unallocated space within
# the "cinder-volumes" volume group to create iscsi volumes for 
# export to instances.  If you wish to leave free space for volumes
# and not preallocate the entire install drive, set enable_vol_space
# to true.
enable_vol_space: true

# Use the following two directives to set the size of the / and /var
# partitions, respectively.  The var_part_size directive will be ignored
# if enable_var is not set to true above.
root_part_size: 65536
var_part_size: 432000

Swift Related Config Changes

Make these changes only if you are going to deploy swift. Here again the swift address variables should point to the Virtual IP for the swift proxy service, and not the real iP of either of the swift-proxy servers.

### Swift configuration
# The username used to authenticate to the Swift cluster.
swift_service_password: swift_pass

# The password hash used to authenticate to the Swift cluster.
swift_hash: super_secret_swift_hash

# The IP address used by Swift on the control node to communicate with
# other members of the Swift cluster.  In the compressed_ha or full_ha
# scenarios, this will be the address to be configured as a VIP on 
# the HAProxy load balancers, not the address of an individual Swift node.
swift_internal_address: 192.168.242.41

# The IP address which external entities will use to connect to Swift,
# including clients wishing to upload or retrieve objects.  In the
# compressed_ha or full_ha scenarios, this will be the address to
# be configured as a VIP on the HAProxy load balancers, not the address
# of an individual Swift node.
swift_public_address: 192.168.242.41

# The IP address over which administrative traffic for the Swift
# cluster will flow.  In the compressed_ha or full_ha
# scenarios, this will be the address to be configured as a VIP on 
# the HAProxy load balancers, not the address of an individual Swift node.
swift_admin_address: 192.168.242.41

Swift needs a different neetwork for allowing local data traffic between storage nodes. This can either be set on a different network interface on the swift nodes, or can be configured on the same interface by using vlans(Eg eth0.XXX)

# The interface on which Swift will run data storage traffic.
# This should generally be a different interface than is used for
# management traffic to avoid congestion.
swift_storage_interface: eth0.222

# The IP address to be configured on the Swift storage interface.
swift_local_net_ip: "%{ipaddress_eth0_222}"

# The netmask to be configured on the Swift storage interface.
swift_storage_netmask: 255.255.255.0

# The IP address of the Swift proxy server. This is the address which
# is used for management, and is often on a separate network from
# swift_local_net_ip.
swift_proxy_net_ip: "%{ipaddress_eth0}"

### The following three parameters are only used if you are configuring
### Swift to serve as a backend for the Glance image service.

# Enable Glance to use Ceph RBD as it's backend storage by uncommenting
# the line below.  It can also be set to "file" or "swift".
#glance_backend: rbd

# The key used by Glance to connect to Swift.
glance::backend::swift::swift_store_key: secret_key

# The IP address to which Glance should connect in order to talk
# to the Swift cluster.
glance::backend::swift::swift_store_auth_address: '127.0.0.1'

ML2 Related Config Changes

We are using the Vlan mode with the ml2 plugin for creating provider networks in this setup. Here are the required configurations

# An array of Neutron service plugins to enable.
service_plugins:
# For ML2, uncomment this line
  - 'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin'
  - 'neutron.services.loadbalancer.plugin.LoadBalancerPlugin'
  - 'neutron.services.firewall.fwaas_plugin.FirewallPlugin'
  - 'neutron.services.vpn.plugin.VPNDriverPlugin'

# ml2 hack. Current packages have ml2 enabled, even if you do not
# use the driver. This param corrects the configuration files.
# This is enabled by default. If you are using ml2, change to false.
disableml2: false

# For ML2, uncomment this line
neutron::core_plugin: 'ml2'

In /etc/puppet/data/hiera_data/user.full_ha.yaml, the following changes need to be made

# This is the short hostname (not FQDN) used to refer to the VIP which
# sits in front of all clustered OpenStack control services.
coe::base::controller_hostname: control

# Specify the URI to be used by horizon to access keystone. For full_ha
# this should use the controller VIP to access keystone.
horizon::keystone_url: 'http://192.168.220.40:5000/v2.0/'


#
# HA connections
#
# controller_names sets the short hostnames (not FQDN) of the
# controller nodes.
controller_names:
  - control01
  - control02
  - control03
# controller_ipaddresses lists the real IP addresses of the controller
# nodes which are being clustered behind the controller VIP.
openstack-ha::load-balancer::controller_ipaddresses:
  - 192.168.220.41
  - 192.168.220.42
  - 192.168.220.43

# swift_proxy_names sets the short hostnames (not FQDN) of the Swift
# proxy nodes.
openstack-ha::load-balancer::swift_proxy_names:
  - swift-proxy01
  - swift-proxy02
# swift_proxy_ipaddresses lists the real IP addresses of the Swift
# proxy nodes which are being clustered behind the Swift VIP.
openstack-ha::load-balancer::swift_proxy_ipaddresses:
  - 192.168.220.61
  - 192.168.220.62
# swift_proxy_net_ip lists the VIP used in front of swift_proxy_ipaddresses
openstack::swift::proxy::swift_proxy_net_ip: "%{ipaddress_eth0}%

# memcached_servers lists the real IP addresses and ports of the
# memcached services on the controller nodes.
nova::memcached_servers:
  - 192.168.220.41:11211
  - 192.168.220.42:11211
  - 192.168.220.43:11211
# swift_memcache_servers lists the real IP addresses and ports of
# the memcached services on the Swift proxy nodes.
openstack::swift::proxy::swift_memcache_servers:
  - 192.168.222.61:11211
  - 192.168.222.62:11211
# rabbit_hosts lists the short hostnames (not FQDN) and ports of the
# RabbitMQ services on the control nodes.
rabbit_hosts:
  - control01:5672
  - control02:5672
  - control03:5672
# galera_servers lists the IP addresses of the nodes which comprise
# the HA MySQL database
galera::galera_servers:
  - 192.168.220.41
  - 192.168.220.42
  - 192.168.220.43
# galera_master defines which node of the galera_servers is the 'master'
# node for bootstrapping purposes; once the cluster is functional this
# distinction has no meaning. This should be the FQDN for that node.
galera::galera_master: control01.local.domain

# Galera which gets writes by default
galera_master_ipaddress: 192.168.220.41
# galera_backup_ipaddresses lists the IP addresses of the other nodes in
# the Galera cluster. HAProxy will direct writes to them if
# galera_master_ipaddress is not reachable
galera_backup_ipaddresses:
  - 192.168.220.42
  - 192.168.220.43
# galera_master_name is the bare hostname (not FQDN) of galera_master_ipaddress
galera_master_name: control01
# galera_backup_names is the bare hostname (not FQDN) of the nodes listed in
# galera_backup_ipaddresses
galera_backup_names:
  - control02
  - control03

# bind_address specifies the IP address on each control node to which
# the OpenStack APIs should bind.
bind_address: "%{ipaddress_eth0}"


#Set the ringserver to the real Ip of one of the proxy servers 
openstack::swift::storage-node::ring_server: 192.168.222.61

In data/scenarios/full_ha, Make these changes. In this HA setup, we do not have a ceph deployment(we have a swift instead), so we pick the roles controller_without_mon and compute_without_osd. In case we have a ceph deployment with mon and osds, the default deployment combines these roles with the controller and compute nodes respectively. So the same changes here should be made in the controller and compute roles instead.

   controller_without_mon:
    classes:
      - coe::base
      - coe::network::interface
      - mongodb::replset
      - "nova::%{rpc_type}"
      - "%{network_service}"
#      - "%{network_service}::plugins::%{network_plugin}"
      - "%{network_service}::server"
    class_groups:
      - glance_all
      - keystone_all
      - cinder_controller
      - nova_controller
      - horizon
      - ceilometer_controller
      - heat_all
      - "%{db_type}_database"
#      - provider_network_controller
# For ML2, Uncomment this and comment above line
      - provider_network_controller_ml2
      - test_file
  compute_without_osd:
    classes:
      - coe::base
      - coe::network::interface
      - cinder::setup_test_volume
    class_groups:
#      - nova_compute
# For ML2, Uncomment this and comment above line
      - nova_compute_ml2
      - cinder_volume
      - ceilometer_compute

In /etc/puppet/data/global_hiera_params/common.yaml

# supports gre or vlan
tenant_network_type: vlan

In /etc/puppet/data/hiera_data/tenant_network_type/vlan.yaml, set the network_vlan_ranges to the vlan range you want to use. Make sure the vlans set are indeed free to use, are forwarded by the physical switch in your testbed. There are multiple network_vlan_ranges and bridge_mappings variable in the file, make sure you set the one for ml2 as shown belown.

neutron::plugins::ml2::network_vlan_ranges:
  - 'physnet1:500:600'
# ML2 Agent
neutron::agents::ml2::ovs::bridge_mappings:
  - "physnet1:br-ex"

The sample config files can be found here https://github.com/CiscoSystems/coi-sample-configs/tree/icehouse/full_ha_swift

Specific Hostname Yaml config files

In puppet/data/hiera_data/hostname/load-balancer01.yaml

openstack-ha::load-balancer::controller_state: MASTER
openstack-ha::load-balancer::swift_proxy_state: BACKUP

In puppet/data/hiera_data/hostname/load-balancer02.yaml

openstack-ha::load-balancer::controller_state: BACKUP
openstack-ha::load-balancer::swift_proxy_state: MASTER

In puppet/data/hiera_data/hostname/control01.yaml

galera::server::master_ip: false

The hostname.yaml files for the other two control nodes should look like this

(for the second control node)

    galera::server::master_ip: '$first_control_node's IP'

(for the third control node)

    galera::server::master_ip: '$second_node's IP'

For each of the two swift-proxy and the three swift storage nodes create a hostname.yaml file with the following information. The ipaddress and the netmask should be of the local swift storage data network, and the interface should be the local interface this network uses. For the swift storage nodes the swift_zone value will be 1, 2 and 3 respectively.

openstack::swift::storage-node::swift_zone: 1
coe::network::interface::interface_name: "eth0.222"
coe::network::interface::ipaddress: "192.168.222.XXX"
coe::network::interface::netmask: "255.255.255.0"

Cobbler Configuration

If you are using cobbler to bring up all the nodes in your deployment, Create profiles for all the 13 nodes in /etc/puppet/data/cobbler/cobbler.yaml. You can follow the instructions here to add profiles for each node. http://docwiki.cisco.com/wiki/OpenStack:_Icehouse:_2-Role#Cobbler_Configuration_2

If you are not using cobbler, then you should have already installed Ubuntu 14.04 on all the 13 nodes in the deployment.

Networking

Our implementation uses VLANs for segmentation of certain networks. Make sure the VLAN package is installed and your network switches have been configured for VLANs. Otherwise, replicate the network setup using only physical interfaces. Additionally, the examples below use two separate physical interfaces on Control and Compute Nodes. For deployments requiring a single physical NIC on Control/Compute Nodes, reference the Single Physical NIC Section for configuration examples and additional deployment details.

apt-get install vlan -y

Load-Balancer Node slb01 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.81
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

Load-Balancer Node slb02 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.82
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

Storage Node swift01 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# Management Network
auto eth0
iface eth0 inet static
        address 192.168.220.71
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	dns-search dmz-pod2.lab
	dns-nameservers 192.168.220.254

# Storage Network
auto eth0.222
iface eth0.222 inet static
        address 192.168.222.71
	netmask 255.255.255.0

Storage Node swift02 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# Management Network
auto eth0
iface eth0 inet static
        address 192.168.220.72
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	dns-search dmz-pod2.lab
	dns-nameservers 192.168.220.254

# Storage Network
auto eth0.222
iface eth0.222 inet static
        address 192.168.222.72
	netmask 255.255.255.0

Storage Node swift03 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# Management Network
auto eth0
iface eth0 inet static
        address 192.168.220.73
	netmask 255.255.255.0
	network 192.168.220.0
        broadcast 192.168.220.255
	gateway 192.168.220.1
	dns-search dmz-pod2.lab
	dns-nameservers 192.168.220.254
	
# Storage Network
auto eth0.222
iface eth0.222 inet static
        address 192.168.222.73
	netmask 255.255.255.0
  • Proxy Node swiftproxy01 /etc/network/interfaces:
# The loopback network interface
auto lo
iface lo inet loopback

# Management Network
auto eth0
iface eth0 inet static
        address 192.168.220.61
	netmask 255.255.255.0
	network 192.168.220.0
        broadcast 192.168.220.255
	gateway 192.168.220.1
	dns-search dmz-pod2.lab
	dns-nameservers 192.168.220.254

# Storage Network
auto eth0.222
iface eth0.222 inet static
        address 192.168.222.61
	netmask 255.255.255.0

Proxy Node swiftproxy02 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# Management Network
auto eth0
iface eth0 inet static
        address 192.168.220.62
	network 192.168.220.0
	netmask 255.255.255.0
        broadcast 192.168.220.255
	gateway 192.168.220.1
	dns-search dmz-pod2.lab
	dns-nameservers 192.168.220.254

# Storage Network
auto eth0.222
iface eth0.222 inet static
        address 192.168.222.62
	netmask 255.255.255.0

Control Node control01 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.41
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

# Public Network: Bridged Interface
auto eth1
iface eth1 inet manual
	up ifconfig $IFACE 0.0.0.0 up
	up ip link set $IFACE promisc on
	down ifconfig $IFACE 0.0.0.0 down

Control Node control02 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.42
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

# Public Network: Bridged Interface
auto eth1
iface eth1 inet manual
	up ifconfig $IFACE 0.0.0.0 up
	up ip link set $IFACE promisc on
	down ifconfig $IFACE 0.0.0.0 down

Control Node control03 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.43
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

# Public Network: Bridged Interface
auto eth1
iface eth1 inet manual
	up ifconfig $IFACE 0.0.0.0 up
	up ip link set $IFACE promisc on
	down ifconfig $IFACE 0.0.0.0 down

Compute Node compute01 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.51
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

# Public Network: Bridged Interface
auto eth1
iface eth1 inet manual
	up ifconfig $IFACE 0.0.0.0 up
	up ip link set $IFACE promisc on
	down ifconfig $IFACE 0.0.0.0 down

Compute Node compute02 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.52
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

# Public Network: Bridged Interface
auto eth1
iface eth1 inet manual
	up ifconfig $IFACE 0.0.0.0 up
	up ip link set $IFACE promisc on
	down ifconfig $IFACE 0.0.0.0 down

Compute Node compute03 /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.220.53
	netmask 255.255.255.0
	network 192.168.220.0
	broadcast 192.168.220.255
	gateway 192.168.220.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 192.168.220.254
	dns-search dmz-pod2.lab

# Public Network: Bridged Interface
auto eth1
iface eth1 inet manual
	up ifconfig $IFACE 0.0.0.0 up
	up ip link set $IFACE promisc on
	down ifconfig $IFACE 0.0.0.0 down

Restart networking:

/etc/init.d/networking restart

Genreal Deployment Instructions on the openstack nodes

It would be good to run the puppet agent manually if you are trying a new full_ha setup. In /etc/puppet/data/hiera_data/user.common.yaml set autostart_puppet to false

  # Cobbler can instruct nodes being provisioned to start a Puppet agent
  # immediately upon bootup.  This is generally desirable as it allows
  # the node to immediately begin configuring itself upon bootup without
  # further human intervention.  However, it may be useful for debugging
  # purposes to prevent Puppet from starting automatically upon bootup.
  # If you want Puppet to run automatically on bootup, set this to true.
  # Otherwise, set it to false.
  autostart_puppet: false

Once the nodes are installed with Ubuntu, login to the server and stop the puppet agent. If the build-server domain name is not recognized on the node then add the build-server hostname entry in /etc/hosts

service puppet stop
puppet agent --enable
puppet agent -td --server=build-server.domainname --pluginsync

Follow the steps below for specific node deployments. Refer this link for more HA related service configs http://docwiki.cisco.com/wiki/OpenStack_Havana_Release:_High-Availability_Manual_Deployment_Guide

Load Balancer Node Installation

Preferably run the puppet agent one by one on the load-balancer nodes. First run the puppet agent on load-balancer01. After the puppet agent finishes check for the status of haproxy service, it should be up and running.

service haproxy status

Also check for the control and the swift-proxy VIPs to be associated with the network interfaces on this loadbalancer node.

ip a

By default the control VIP gets associated to the primary interface on the load-balancer. The swift-proxy gets associated with the interface chosen for the local swift network in the user.common.yaml file

swift_local_net_ip: "%{ipaddress_eth0_222}"

Make sure this interface exists on the load-balancer and is up.

Once the fist load-balancer is up, run the puppet agent on the second load-balancer02.

If the haproxy & keepalive services are not up, or the VIP's do not get bound to expected interfaces. Check if the configs are set as expected. http://docwiki.cisco.com/wiki/OpenStack_Havana_Release:_High-Availability_Manual_Deployment_Guide#Load_Balancer_Node_Installation

General Installation Steps for All Swift Nodes

The deployment does not automatically assoicate the swift storage network ip in the swift nodes. So once each swift node is up, make sure you manually configure the ip on the second network interface

# Storage Network
auto eth0.222
iface eth0.222 inet static
        address 192.168.222.XX
	netmask 255.255.255.0

Swift Storage Node Installation Steps

We run the agent on the swift nodes one by one.

We start with the swift-storage01 and run the puppet agent on that node.

At this point, the ring files will not be present on the storage nodes. This will cause the Swift *-replicator and container-sync services to not start properly. After you create the ring files on the first proxy node (in the next section) and distribute them to the storage nodes, restarting the object-expirer, *-replicator and container-sync services will allow the remaining Swift storage node services to start properly. The following swift services should be started:

swift-init object-updater start
swift-init object-server start
swift-init object-auditor start
swift-init container-server start
swift-init container-updater start
swift-init container-auditor start
swift-init account-server start
swift-init account-auditor start
swift-init account-reaper start

swift-init all status
No container-updater running
account-auditor running (4416 - /etc/swift/account-server.conf)
object-replicator running (19143 - /etc/swift/object-server.conf)
No proxy-server running
container-replicator running (19037 - /etc/swift/container-server.conf)
object-auditor running (5021 - /etc/swift/object-server.conf)
No object-expirer running
container-auditor running (4733 - /etc/swift/container-server.conf)
container-server running (5084 - /etc/swift/container-server.conf)
account-server running (5156 - /etc/swift/account-server.conf)
account-reaper running (4400 - /etc/swift/account-server.conf)
No container-sync running
account-replicator running (19105 - /etc/swift/account-server.conf)
No object-updater running
object-server running (5608 - /etc/swift/object-server.conf)

Repeat these steps for every Storage Node.

Swift Proxy Node Installation Steps

Perform these steps on swiftproxy01.

Run the puppet agent. The puppet agent should complete without any errors. The swift proxy service should be running on swift-proxy01.

Once this is up we can run the puppet agent on each of the storage nodes. This puppet agent run(on the storage nodes) should copy over the account.ring.gz, container.ring.gz, and object.ring.gz files to the 3 storage nodes in /etc/swift. This should enable the start of the replicator services on the storage nodes that were not starting previously.

Run the puppet agent on swift-proxy02 next.

If you continue to see errors in any of the swift nodes, run the puppet agent a couple of times on it.

Verify all the expected services are running in the nodes, if not try starting some manually like this.

swift-init status all
swift-init restart all
swift-init status all

Verify the Swift Installation

You can run verification commands from the proxy server or any server with access to Keystone. Keep in mind that proxy nodes are configured to use Keystone for user authentication. As a result, you MUST complete the Controller Node Installation steps and ensure Keystone is operational before proceeding with Swift verification.

Verify that you can successfully authenticate against Keystone using the Swift authentication credentials:

apt-get install -y curl

curl -s -d "{\"auth\":{\"passwordCredentials\": {\"username\": \"swift\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

You should receive output similar to the following:

{"access": {"token": {"issued_at": "2013-04-02T14:55:31.149327", "expires": "2013-04-03T14:55:31Z", "id": "bb29ef5439ce4a75bf85332bbadf6538", "tenant": {"description": 
null, "enabled": true, "id": "b38d88aad6314870b746e7d60808e59a", "name": "services"}}, "serviceCatalog": [{"endpoints": [{"adminURL": 
"http://192.168.220.40:8774/v2/b38d88aad6314870b746e7d60808e59a", "region": "RegionOne", "internalURL": "http://192.168.220.40:8774/v2/b38d88aad6314870b746e7d60808e59a"
, "id": "45a336cb74e04e11ab95c0ea28b699d6", "publicURL": "http://192.168.220.40:8774/v2/b38d88aad6314870b746e7d60808e59a"}], "endpoints_links": [], "type": "compute", 
"name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.220.40:9696/", "region": "RegionOne", "internalURL": "http://192.168.220.40:9696/", "id": 
"259fef5e66814f47ac1934d3cf522a3d", "publicURL": "http://192.168.220.40:9696/"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [
{"adminURL": "http://192.168.220.40:9292/v2", "region": "RegionOne", "internalURL": "http://192.168.220.40:9292/v2", "id": "166de3790eb54c31a58237fe9ea3d301", 
"publicURL": "http://192.168.220.40:9292/v2"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": 
"http://192.168.220.40:8776/v1/b38d88aad6314870b746e7d60808e59a", "region": "RegionOne", "internalURL": "http://192.168.220.40:8776/v1/b38d88aad6314870b746e7d60808e59a"
, "id": "0a2c69157d5948a9ae8ecee5c65a6d2b", "publicURL": "http://192.168.220.40:8776/v1/b38d88aad6314870b746e7d60808e59a"}], "endpoints_links": [], "type": "volume", 
"name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.220.40:8773/services/Admin", "region": "RegionOne", 
"internalURL": "http://192.168.220.40:8773/services/Cloud", "id": "05f85b8aacbd4c87b680dcc2fb6da539", "publicURL": "http://192.168.220.40:8773/services/Cloud"}], 
"endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://192.168.220.60:8080/v1", "region": "RegionOne", "internalURL": 
"http://192.168.220.60:8080/v1/AUTH_b38d88aad6314870b746e7d60808e59a", "id": "4a1af526137341c0a682eb573101ddde", "publicURL": 
"http://192.168.220.60:8080/v1/AUTH_b38d88aad6314870b746e7d60808e59a"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": 
"http://192.168.220.40:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.220.40:5000/v2.0", "id": "3e3f7b50b5bd44b7a15b3e4ae55086bf", "publicURL": 
"http://192.168.220.40:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "swift", "roles_links": [], "id": 
"ed69664ac78a4b65a36d63da6b760863", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "swift"}, "metadata": {"is_admin": 0, "roles": [
"9fe2ff9ee4384b1894a90878d3e92bab", "6a553ae3be3c4f8c8fe079830d4102a5"]}}}

Use the swift client stat command to make sure you can view the contents of the ring. You can run these commands from the proxy server or any server with the swift client and access to Keystone.

swift -V 2 -A http://192.168.220.40:5000/v2.0/ -V 2 -U services:swift -K keystone_admin stat
   Account: AUTH_3eccdb2a9331419c96ac9ff336110b65
Containers: 1
   Objects: 2
     Bytes: 0
Accept-Ranges: bytes
X-Timestamp: 1363989109.30329
X-Trans-Id: tx147dd9983ac54af1b71c5a561ae2aa9a
Content-Type: text/plain; charset=utf-8

You can see that 1 container exists.  Now, lets find out the name of the container:

swift -V 2 -A http://192.168.220.40:5000/v2.0/ -V 2 -U services:swift -K keystone_admin list
glance

Note: The glance container is created after the Controller cluster is built and an image has been uploaded to Glance.

List the contents of the Glance container:

swift -V 2 -A http://192.168.220.40:5000/v2.0/ -V 2 -U services:swift -K keystone_admin list glance
24164630-ba2f-436a-8bc6-43975717d5e5
858a11dc-ed61-4a18-a778-eabcb454ae45

Controller Node Installation

As the steps followed for the other nodes, the control nodes need to be deployed one after the other. Start with control01. The initial value in /etc/puppet/data/hiera_data/hostname/control01.yaml should be set as shown below

galera::server::master_ip: false

The first puppet run on the control run will return some errors that can be ignored. After the run on the first control01 node, run the puppet agent on the second control02 node, folowed by the third. The hostname.yaml files for the other two control nodes should look like this

(for the second control node)

    galera::server::master_ip: '$first_control_node's IP'

(for the third control node)

    galera::server::master_ip: '$second_node's IP'


After completing the puppet run on the third control node, change the control01.yaml to

 galera::server::master_ip: '$third_node's IP'

Run the puppet agent a couple of times in each of the control nodes, till you see no errors in the run.

To check if all the openstack services are up, use the following commands

MySQL WSREP and Galera Installation

Verify the Galera cluster has been established. The value should show 4 for all nodes in the cluster:

mysql -e "show global status where variable_name='wsrep_local_state';"
  +------------------------+---+
  | Variable_name      | Value |
  +------------------------+---+
  | wsrep_local_state  |    4  |
  +------------------------+---+

MySQL WSREP and Galera Monitoring

Verify that you can access the MySQL database by using the Virtual IP address (VIP) of the Galera cluster:

mysql -umonitor_user -pmonitor_password -h192.168.220.40

For informational purposes, this is the command used by the health check script. This example is for control01:

/usr/bin/mysql -N -q -A --host=192.168.220.41 --user=monitor_user --password=monitor_password -e "show global status where variable_name='wsrep_local_state';"

RabbitMQ Installation

Clustering can be configured using rabbitmqctl commands or by modifying the RabbitMQ configuration file. Our example uses the rabbitmqctl commands. You can see both approaches to configuring RabbitMQ clustering here. Note: Joining a cluster implicitly resets the node, thus removing all resources and data that were previously present on that node.

From control02:

rabbitmqctl stop_app
rabbitmqctl cluster rabbit@control01
rabbitmqctl start_app

Verify that control02 is now clustered with control01:

rabbitmqctl cluster_status

Cluster status of node rabbit@control02 ...
[{nodes,[{disc,[rabbit@control01]},{ram,[rabbit@control02]}]},
 {running_nodes,[rabbit@control01,rabbit@control02]}]
...done.

From control03:

rabbitmqctl stop_app
rabbitmqctl cluster rabbit@control02
rabbitmqctl start_app

Verify that control03 has joined the cluster:

rabbitmqctl cluster_status

Cluster status of node rabbit@control03 ...
[{nodes,[{disc,[rabbit@control01]},{ram,[rabbit@control03,rabbit@control02]}]},
 {running_nodes,[rabbit@control02,rabbit@control01,rabbit@control03]}]
...done.

Now that clustering is complete, secure RabbitMQ by removing the default (guest) user from only one of the nodes in the cluster:

rabbitmqctl delete_user guest

From only one of the nodes in the cluster, create a RabbitMQ user account that will be used by OpenStack services:

rabbitmqctl add_user openstack_rabbit_user openstack_rabbit_password

From only one of the nodes in the cluster, set the permissions for the new RabbitMQ user account:

rabbitmqctl set_permissions -p / openstack_rabbit_user ".*" ".*" ".*"

Verify the user settings:

rabbitmqctl list_users
rabbitmqctl list_user_permissions openstack_rabbit_user

Keystone Installation

Test connectivity to Keystone by using a curl request :

apt-get install curl openssl -y

curl -d '{"auth": {"tenantName": "admin", "passwordCredentials":{"username": "admin", "password": "keystone_admin"}}}' -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens | python -mjson.tool

If the above command is successful, you will receive output that includes a token and a list of service endpoints. You may also want to verify the other service account credentials:

Glance

curl -s -d  "{\"auth\":{\"passwordCredentials\": {\"username\": \"glance\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

Nova

curl -s -d  "{\"auth\":{\"passwordCredentials\": {\"username\": \"nova\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

Swift

curl -s -d  "{\"auth\":{\"passwordCredentials\": {\"username\": \"swift\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

Neutron

curl -s -d  "{\"auth\":{\"passwordCredentials\": {\"username\": \"neutron\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

Cinder

curl -s -d  "{\"auth\":{\"passwordCredentials\": {\"username\": \"cinder\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

Heat

curl -s -d  "{\"auth\":{\"passwordCredentials\": {\"username\": \"heat\", \"password\": \"keystone_admin\"}, \"tenantName\": \"services\"}}" -H "Content-type: application/json" http://192.168.220.40:35357/v2.0/tokens

You can also use the Keystone client to verify the configuration:

keystone tenant-list
keystone user-list
keystone role-list
keystone service-list
keystone endpoint-list

Now that Keystone is operational, you may want to go back to the Verify the Swift Installation section to ensure that Swift is fully operational. If Swift is inoperable, you will be unable to add images to Glance in the Glance_Installation next section.

Glance Installation

Synchronize the glance database on one control node (You may get a message about deprecation - you can ignore):

glance-manage db_sync

Download the Cirros 0.3.1 cloud image to a controller node and then upload it to Glance:

wget http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img

glance image-create --name cirros --is-public=true --disk-format=qcow2 --container-format=ovf < cirros-0.3.1-x86_64-disk.img

Verify that Glance is serving the image:

glance image-list

Neutron Installation

Check the status of the Open vSwitch services on each control node:

service openvswitch-switch status

Start the Open vSwitch services on each control node if they are not running:

service openvswitch-switch start

Control Nodes require OVS bridges named "br-int" and "br-ex", and that "br-ex" is associated with the Public Network interface (eth1 in our example):

ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex
ovs-vsctl add-port br-ex eth1


Edit the OVS plugin configuration file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini on all control nodes. Note: 223:225 signifies the VLAN ID range used for tenant VLANs. Modify this range based on your deployment needs. These VLANs should be trunked to eth1 of Control Nodes and you must create a gateway address (i.e. 192.168.223.1 for VLAN 223) on your upstream Layer-3 device.

[ovs]
tenant_network_type = vlan
network_vlan_ranges = physnet1:223:225
bridge_mappings = physnet1:br-ex

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Make sure the OVS interface_driver is set in /etc/neutron/dhcp_agent.ini:

interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

Restart Neutron services:

service neutron-server restart; service neutron-dhcp-agent restart; service neutron-plugin-openvswitch-agent restart

Nova Installation

Synchronize the Nova database on only one control node (You may get a DEBUG message - You can ignore this):

nova-manage db sync

Restart nova-* services on all control nodes:

cd /etc/init.d/; for i in $( ls nova-* ); do sudo service $i restart; done

Check for the smiling faces on nova services to confirm your installation:

nova-manage service list

Also check that nova-api is running:

service nova-api status

Use the following steps if you would like to add support for migrating instances. The details of migration is outside the scope of this document. Use the official OpenStack documentation to understand the details of migration.

  • Uncomment the following in /etc/libvirt/qemu.conf
cgroup_device_acl = [
"/dev/null", "/dev/full", "/dev/zero",
"/dev/random", "/dev/urandom",
"/dev/ptmx", "/dev/kvm", "/dev/kqemu",
"/dev/rtc", "/dev/hpet"
]
  • Edit /etc/libvirt/libvirtd.conf file as follows:
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
  • Modify libvirtd_opts variable in /etc/init/libvirt-bin.conf file :
env libvirtd_opts="-d -l" 
  • Edit /etc/default/libvirt-bin file :
libvirtd_opts="-d -l"
  • Restart libvirt :
service libvirt-bin restart

Cinder Installation

Initialize the Cinder database on only one control node:

cinder-manage db sync

Restart Cinder services on all control nodes:

service cinder-api restart; service cinder-scheduler restart

Heat Installation

Initialize the Heat database on only one control node:

heat-manage db_sync

Restart Heat services on all control nodes:

service heat-api restart; service heat-api-cfn restart; service heat-api-cloudwatch restart; service heat-engine restart

After you complete the HA deployment, use the official OpenStack guide to create Heat stacks.

Horizon Installation

Access Horizon by using the following URL in your web browser. Use admin/keystone_admin for your login credentials.  If you have problems accessing Horizon by using the VIP (192.168.220.40), then try using a real IP address of a control node (i.e. control01 = 192.168.220.41):

http://192.168.220.40/horizon

Optionally, if you would like to remove the Ubuntu theme:

apt-get purge -y openstack-dashboard-ubuntu-theme

Compute Node Installation

Follow these steps for compute01, compute02 and compute03 compute nodes.

Neutron Installation

Check the status of the Open vSwitch services on each compute node:

service openvswitch-switch status

Start the Open vSwitch services on each compute node if they are not running:

service openvswitch-switch start

Compute Nodes require OVS bridges named "br-int" and "br-ex", and that "br-ex" is associated with the Public Network interface (eth1 in our example):

ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex
ovs-vsctl add-port br-ex eth1

Edit the OVS plugin configuration file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini to include the following. Note: 223:225 signifies the VLAN ID range used for tenant VLANs. Modify this range based on your deployment needs. These VLANs should be trunked to eth1 of Compute Nodes and you must create a gateway address (i.e. 192.168.223.1 for VLAN 223) on your upstream Layer-3 device.

[ovs]
tenant_network_type = vlan
network_vlan_ranges = physnet1:223:225
bridge_mappings = physnet1:br-ex

# Using Neutron Security Groups instead of Nova Security Groups
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Restart the Neutron OVS Plugin Agent service on each compute node:

service neutron-plugin-openvswitch-agent restart

Nova Installation

Create a credentials file so you can issue OpenStack client commands from the Compute Nodes:

vi /root/openrc

export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=keystone_admin
export OS_AUTH_URL="http://192.168.220.40:5000/v2.0/"
export OS_AUTH_STRATEGY=keystone

source /root/openrc

Check for the smiling faces on nova services to confirm your installation:

nova-manage service list

Use the following steps if you would like to add support for migrating instances. The details of migration is outside the scope of this document. Use the official OpenStack documntation to understand the details of migration.

  • Edit /etc/libvirt/libvirtd.conf file as follows:
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
  • Modify libvirtd_opts variable in /etc/init/libvirt-bin.conf file :
env libvirtd_opts="-d -l" 
  • Edit /etc/default/libvirt-bin file :
libvirtd_opts="-d -l"
  • Restart libvirt :
service libvirt-bin restart

Cinder Installation

Restart the Cinder services on all compute nodes:

service cinder-volume restart; service tgt restart

Configuring OpenStack Networking (Neutron) and Deploying the First VM

Run the following commands from either a Compute Node or Controller Node. If something has to be done on a specific node it will be called out.

Create your first tenant network. In our example, we use the admin tenant.  Create additional networks as needed. Note: The --tenant_id flag is not specified in the following commands because we previously sourced our credential file.

neutron net-create public223 --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 223

Create your first tenant subnet and associate it to the network you created in the previous step. The example below uses .10-.250 for Instance IP addresses. Modify the allocation-pool and dns_nameservers based on your deployment needs. Create additional networks as needed.

neutron subnet-create --name 223-subnet --allocation-pool start=192.168.223.10,end=192.168.223.250 public223 192.168.223.0/24 --dns_nameservers list=true 192.168.26.186

Edit the default Neutron Security Group to allow ingress traffic to Instances. Our example allows all ingress ICMP and SSH traffic. Note: Security Group rules are associated to a specific tenant.

neutron security-group-rule-create default --direction ingress --ethertype IPv4 --protocol icmp --remote-ip-prefix 0.0.0.0/0
neutron security-group-rule-create default --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 22 --port-range-max 22 --remote-ip-prefix 0.0.0.0/0

Note: If you receive the following message "Multiple security_group matches found for name 'default', use an ID to be more specific.", then replace the security-group name default with the security-group ID. The ID can be found by using the neutron security-group-list command.

If you skipped the earlier step of downloading an image and uploading it to glance, do that now. Note: The HA architecture uses Config Drive instead of Nova Metadata, so make sure you use an image that supports Cloud Init.

wget http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img

glance image-create --name cirros --is-public=true --disk-format=qcow2 --container-format=ovf < cirros-0.3.1-x86_64-disk.img

Before booting the instance, check for the ID of the network we created earlier. Note: the <neutron_net_id> value will come from the output of the "neutron net-list" command:

neutron net-list
nova boot --image cirros --flavor m1.tiny --nic net-id=<neutron_net_id> <instance_name>

Example:

nova boot --image cirros --flavor m1.tiny --nic net-id=f9035744-72a9-42cf-bd46-73d54c0cea06 vm1

Watch the status of the instance:

nova show <instance_name>

Example:

nova show vm1

The instance is booted completely when the OS-EXT-STS:vm_state is "active". Make note of the IP address of the VM. Alternatively, you can watch the complete log of the VM booting by running:

nova console-log --length=25 vm1

Note: If you see the instance log stuck in the following state, restart the neutron-dhcp-agent service on compute nodes and re-deploy your instance:

Starting network...
udhcpc (v1.20.1) started
Sending discover…

From a host with connectivity to your Neutron public subnet (192.168.223.0/24 in our example), you should now be able to ping and SSH to the VM. Note: The cirros image uses cirros/cubswin:) as the default SSH login credentials:

# ssh cirros@192.168.223.17
The authenticity of host '192.168.223.17 (192.168.223.17)' can't be established.
RSA key fingerprint is 42:49:28:17:67:ed:b2:63:f8:66:a6:3a:e8:0a:1a:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.223.17' (RSA) to the list of known hosts.
cirros@192.168.223.17's password: 
$ 

Configuring SSH Key Injection for Accessing Instances

Config Drive and Cloud Init provide the ability to inject user and system information into an Instance during the boot-up process. SSH keys can be injected into Instances to provide password-less SSH access. A high-level breakdown of the process is as follows:

  1. Create an SSH key-pair that will be used for injection.
  2. Add the public key from the key-pair to Nova.
  3. Reference the Nova key-pair name within the Nova boot command or Horizon GUI.
  4. The public key will be injected into the instance by Config Drive and cloud-init during the boot-up.
  5. The SSH client references the associated private key during the connection request.

From a host with connectivity to the Neutron public subnet (192.168.223.0/24 in our example), generate an SSH key-pair. Note: Leave the passphrase empty when creating the key-pair.

# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
fd:d1:d4:6b:ab:df:03:9b:9d:64:6f:2b:2b:47:ae:d3 root@ubuntu-mgmt
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|               . |
|              . .|
|         .   o  .|
|        S . . .o |
|           . +.o.|
|            = B.o|
|           o E.++|
|           .=o+++|
+-----------------+

Add the SSH public (.pub) key to Nova. If the host being used to generate the SSH key-pair does not have the Nova client installed and openrc credential file, you will need to copy the public key (i.e. id_rsa.pub) to an OpenStack Control Node. The following command is from a host that contains the Nova client, openrc credential file and SSH public key:

nova keypair-add --pub-key id_rsa.pub <key_name>

Example:

nova keypair-add --pub-key /root/.ssh/id_rsa.pub test-key

This creates a new key-pair called test-key. The private key (id_rsa) is saved locally in ~/.ssh which can be used to connect to an instance launched using test-key as the Nova keypair name. You can see the available key-pairs with nova keypair-list command.

nova keypair-list
+-------+-------------------------------------------------+
|  Name |                   Fingerprint                   |
+-------+-------------------------------------------------+
| test-key  | b0:18:32:fa:4e:d4:3c:1b:c4:6c:dd:cb:53:29:13:82 |
+-------+-------------------------------------------------+

Boot an Instance and include the newly creatd Nova keypair:

nova boot --image cirros --flavor m1.tiny --key_name test-key --nic net-id=<neutron_net_id> <instance_name>

The Instance should boot successfully. You can use the nova show <instance_name> command to obtain the IP address used to ping and SSH to the instance in the following command.
SSH to the Instance by specifing the -i and path to the SSH private key:

ssh -i /root/.ssh/id_rsa cirros@192.168.223.14

The authenticity of host '192.168.223.14 (192.168.223.14)' can't be established.
RSA key fingerprint is ef:62:fa:09:24:bd:67:b8:42:a5:2e:ce:c7:b1:65:41.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.223.14' (RSA) to the list of known hosts.
$ 

If you receive a Permissions are too open… private key will be ignored error, change the permissions of your private key so that it is only readable and writeable by the owner:

chmod 600 /root/.ssh/id_rsa

Configuring OpenStack Networking (Neutron) DHCP Agent High-Availability

First, verify the status of your Neutron Agents. You should see :-) under alive and True under admin_state_up for all agents. Do not proceed with DHCP Agent fail-over testing if this is not the case:

neutron agent-list
+--------------------------------------+--------------------+------------------------+-------+----------------+
| id                                   | agent_type         | host                   | alive | admin_state_up |
+--------------------------------------+--------------------+------------------------+-------+----------------+
| 17538649-c80b-4c82-aedf-67ffca608a5d | DHCP agent         | compute03.dmz-pod2.lab | :-)   | True           |
| 4bc8dac3-ec4a-4369-b1b9-3c0111211d63 | DHCP agent         | compute02.dmz-pod2.lab | :-)   | True           |
| 4f574568-1342-4eea-94ae-96ce7b6af3f1 | Open vSwitch agent | compute01.dmz-pod2.lab | :-)   | True           |
| 53a44eae-025d-40a5-9505-10d33b3f9779 | DHCP agent         | compute01.dmz-pod2.lab | :-)   | True           |
| 7394b48c-5b1a-459f-95c0-1522c443fa88 | Open vSwitch agent | compute02.dmz-pod2.lab | :-)   | True           |
| a2f55922-b230-4e8f-8535-30ce59dbbb98 | Open vSwitch agent | compute03.dmz-pod2.lab | :-)   | True           |
+--------------------------------------+--------------------+------------------------+-------+----------------+

Verify what DHCP Agent is servicing the Neutron public223 network:

neutron dhcp-agent-list-hosting-net public223
+--------------------------------------+------------------------+----------------+-------+
| id                                   | host                   | admin_state_up | alive |
+--------------------------------------+------------------------+----------------+-------+
| 4bc8dac3-ec4a-4369-b1b9-3c0111211d63 | compute02.dmz-pod2.lab | True           | :-)   |
+--------------------------------------+------------------------+----------------+-------+

Have another DHCP Agent service the public223 network. The DHCP Agent on Compute01 is being added to the public223 in our example. Note: The DHCP Agent ID can found in the output of the neutron agent-list command.

neutron dhcp-agent-network-add 53a44eae-025d-40a5-9505-10d33b3f9779 public223
Added network public223 to DHCP agent

Verify the DHCP Agent on Compute01 has been added to the public223 network:

root@control01:~# neutron dhcp-agent-list-hosting-net public223
+--------------------------------------+------------------------+----------------+-------+
| id                                   | host                   | admin_state_up | alive |
+--------------------------------------+------------------------+----------------+-------+
| 4bc8dac3-ec4a-4369-b1b9-3c0111211d63 | compute02.dmz-pod2.lab | True           | :-)   |
| 53a44eae-025d-40a5-9505-10d33b3f9779 | compute01.dmz-pod2.lab | True           | :-)   |
+--------------------------------------+------------------------+----------------+-------+

Verify that you can successfully boot a few Instances:

nova boot --image precise --flavor m1.small --key_name <key_name> --nic net-id=<neutron_net_id> <instance_name>

Example:

nova boot --image precise --flavor m1.small --key_name net-key --nic net-id=f9035744-72a9-42cf-bd46-73d54c0cea06 vm1

Watch the status of the instance:

nova show <instance_name>

Example:

nova show vm1

The instance is booted completely when the OS-EXT-STS:vm_state is "active". Make note of the IP address of the VM. Alternatively, you can watch the complete log of the VM booting by running:

nova console-log --length=25 vm1

Note: Ensure that your Instance has successfully received an IP address from the nova console-log command. Test fail-over by disabling a DHCP Agent on one of the Compute Nodes. Our example disables the DHCP Agent on Compute02. Note: The DHCP Agent ID can found in the output of the neutron agent-list command.

neutron agent-update 4bc8dac3-ec4a-4369-b1b9-3c0111211d63 --admin_state_up=false
Updated agent: 4bc8dac3-ec4a-4369-b1b9-3c0111211d63

Verify the agent is disabled. You should see admin_state_up False associated to the DHCP Agent that you disabled:

root@control01:~# neutron agent-list
+--------------------------------------+--------------------+------------------------+-------+----------------+
| id                                   | agent_type         | host                   | alive | admin_state_up |
+--------------------------------------+--------------------+------------------------+-------+----------------+
| 17538649-c80b-4c82-aedf-67ffca608a5d | DHCP agent         | compute03.dmz-pod2.lab | :-)   | True           |
| 4bc8dac3-ec4a-4369-b1b9-3c0111211d63 | DHCP agent         | compute02.dmz-pod2.lab | :-)   | False          |
| 4f574568-1342-4eea-94ae-96ce7b6af3f1 | Open vSwitch agent | compute01.dmz-pod2.lab | :-)   | True           |
| 53a44eae-025d-40a5-9505-10d33b3f9779 | DHCP agent         | compute01.dmz-pod2.lab | :-)   | True           |
| 7394b48c-5b1a-459f-95c0-1522c443fa88 | Open vSwitch agent | compute02.dmz-pod2.lab | :-)   | True           |
| a2f55922-b230-4e8f-8535-30ce59dbbb98 | Open vSwitch agent | compute03.dmz-pod2.lab | :-)   | True           |
+--------------------------------------+--------------------+------------------------+-------+----------------+

Verify that you can successfully boot a few Instances:

nova boot --image precise --flavor m1.small --key_name <key_name> --nic net-id=<neutron_net_id> <instance_name>

Example:

nova boot --image precise --flavor m1.small --key_name net-key --nic net-id=f9035744-72a9-42cf-bd46-73d54c0cea06 vm1

Watch the status of the instance:

nova show <instance_name>

Example:

nova show vm1

The instance is booted completely when the OS-EXT-STS:vm_state is "active". Make note of the IP address of the VM. Alternatively, you can watch the complete log of the VM booting by running:

nova console-log --length=25 vm1

Note: Ensure that your Instance has successfully received an IP address from the nova console-log command.

Test fail-over by disabling a DHCP Agent on one of the Compute Nodes. Repeat the same process with other DHCP agents.

When fail-over testing is complete, enable all DHCP Agents and repeat the steps above for fail-back testing.

Create and Attach a Cinder Volume

You can use Horizon to create and attach Cinder volumes if you prefer. Since using Horizon to manage volumes is self-explanatory, this section covers the CLI for managing volumes. First, verify the status of your Cinder services from a controller. You should see :-) under State and enabled under Status for all services. Do not proceed with creating and attaching volumes if this is not the case:

cinder-manage service list
Binary           Host                                 Zone             Status     State Updated At
cinder-scheduler control01                            nova             enabled    :-)   2013-11-01 17:13:57
cinder-volume    compute01                            nova             enabled    :-)   2013-11-01 17:13:57
cinder-scheduler control02                            nova             enabled    :-)   2013-11-01 17:13:57
cinder-volume    compute02                            nova             enabled    :-)   2013-11-01 17:13:57
cinder-scheduler control03                            nova             enabled    :-)   2013-11-01 17:13:57
cinder-volume    compute03                            nova             enabled    :-)   2013-11-01 17:13:57

Create a Cinder volume. Our example uses v2 for the volume name and is 2GB in size:

cinder create --display-name v2 2
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-11-01T17:12:01.058239      |
| display_description |                 None                 |
|     display_name    |                  v2                  |
|          id         | 8b858ebe-c7aa-4674-af0d-7031bb5cc7a1 |
|       metadata      |                  {}                  |
|         size        |                  2                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

Attach the volume to a running instance. Our examples attaches the 8b858ebe-c7aa-4674-af0d-7031bb5cc7a1 volume (UUID of the v2 volume) to an Instance named c2.

nova volume-attach c2 8b858ebe-c7aa-4674-af0d-7031bb5cc7a1 /dev/vdb
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdvc                            |
| serverId | 91f1b1a3-f1b8-4efc-a3a2-8708447e9377 |
| id       | 8b858ebe-c7aa-4674-af0d-7031bb5cc7a1 |
| volumeId | 8b858ebe-c7aa-4674-af0d-7031bb5cc7a1 |
+----------+--------------------------------------+

Log into the c2 Instance and verify the volume attachment. You should observe a 2GB disk named /dev/vdb:

sudo -i
fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux

Disk /dev/vdb: 2147 MB, 2147483648 bytes
16 heads, 63 sectors/track, 4161 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn't contain a valid partition table

Partition the /dev/vdb disk:

fdisk /dev/vdb

Once in fdisk, perform the following commands:

  1. Press n to create a new disk partition,
  2. Press p to create a primary disk partition,
  3. Press 1 to denote it as 1st disk partition,
  4. Either press ENTER twice to accept the default of 1st and last cylinder – to convert the remainder of hard disk to a single disk partition -OR- press ENTER once to accept the default of the 1st, and then choose how big you want the partition to be by specifying +size[K,M,G] e.g. +5G or +6700M.
  5. Press t and select the new partition that you have created.
  6. Press 83 change your new partition to 8e, i.e. Linux LVM partition type.
  7. Press p to display the hard disk partition setup. Please take note that the first partition is denoted as /dev/sdb1 in Linux.
  8. Press w to write the partition table and exit fdisk upon completion.

You should see your new partition named /dev/vdb1 in this listing.

fdisk -l

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux

Disk /dev/vdb: 2147 MB, 2147483648 bytes
1 heads, 16 sectors/track, 262144 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x93bf0c66

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1            2048     4194303     2096128   83  Linux

Make a file system on the partition and mount it.

mkfs.ext3 /dev/vdb1
mkdir /cinder-test
mount /dev/vdb1 /cinder-test

In our experience, in order for the contents of a Cinder volume to become persistent, you must first reboot the Instance before writing data to the volume.

reboot now

Log back into the Instance, create a test file and reboot to test data persistence.

touch /cinder-test/test-file
reboot now

Single NIC Configurations

In the Networking Section, multiple NICs are used on Control and Compute Nodes. Eth0 is used for node management traffic and for associating API endpoints on Control Nodes. Eth1, which does not use an IP address, is used to bridge OVS traffic between instances and the provider's physical network. Below is an /etc/network/interfaces example (control01) for deployments requiring a single NIC on Control/Compute Nodes:

# The loopback network interface
auto lo
iface lo inet loopback

# Public Network: Bridged Interface
auto eth0
iface eth0 inet static
  address 192.168.220.41
  netmask 255.255.255.0
  gateway 192.168.220.1
  dns-nameservers 192.168.220.254
  dns-search dmz-pod2.lab
  post-down ip addr flush dev eth0

# Public Network: Bridged Interface
auto br-ex
iface br-ex inet manual
  pre-up ip link set eth0 down
  up ifconfig br-ex 192.168.220.41 netmask 255.255.255.0 up
  dns-search dmz-pod2.lab
  dns-nameservers 192.168.220.254
  up ip route add default via 192.168.220.1 dev br-ex
  up ifconfig eth0 0.0.0.0 up
  up ip link set eth0 up
  post-up service openvswitch-switch restart
  post-up service neutron-plugin-openvswitch-agent restart
  post-up service neutron-dhcp-agent restart
  post-up service nova-compute restart
  post-down ip link set br-ex down

Cisco UCS Servers support NIC virtualization. With NIC virtualization, you can create multiple vNIC's per physical NIC. Therefore, you can create 2 vNIC's associated to physical NIC 0 and the Ubuntu Operating System will see eth0 and eth1. This will allow you to follow the same steps identified in the Networking Section, while only using a single physical NIC.

Support

Email: openstack-support@cisco.com

Credits

This work has been based on:

  • OpenStack Havana Installation Guide for Ubuntu 12.04 (LTS) Documentation [2]

Authors

Shweta Padubidri

Rating: 0.0/5 (0 votes cast)

Personal tools