Getting through OpenStack Quantum with Open vSwitch

Allan Feid — Wed, 13 Feb 2013 05:55:59 +0000

Over the last couple days I've been working on setting up OpenStack Quantum on my Folsom cluster. I'm using CentOS 6.3, which is by far not the best supported platform to run OpenStack on. I figure I'll try to cover the problems I went through getting this running, and hopefully explain a bit about how Quantum works. I'll assume you have installed an OpenStack cluster from the EPEL provided packages already, and are familiar with the basics of OpenStack's services architecture.

Open vSwitch

Before I go on to actually configuring Quantum, you will want to build the openvswitch and kmod-openvswitch RPMs. I've opened a bug with RedHat to get this package in EPEL, and it also gives you an idea of how to build the package for yourself. For reference, something as simple as this should work for you:

$ wget https://bugzilla.redhat.com/attachment.cgi?id=696698 -O $HOME/rpmbuild/SOURCES/openvswitch-1.7.3-el6.patch
$ wget http://openvswitch.org/releases/openvswitch-1.7.3.tar.gz -O $HOME/rpmbuild/SOURCES/openvswitch-1.7.3.tar.gz
$ wget https://bugzilla.redhat.com/attachment.cgi?id=696699 -O $HOME/rpmbuild/SPECS/openvswitch.spec
$ wget \
  http://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=rhel/openvswitch-kmod-rhel6.spec.in;hb=branch-1.7 \
    -O $HOME/rpmbuild/SPECS/openvswitch-kmod.spec
$ rpmbuild -vs -D "kversion `uname -r`" $HOME/rpmbuild/SPECS/openvswitch.spec
$ rpmbuild -vs -D "kversion `uname -r`" $HOME/rpmbuild/SPECS/openvswitch-kmod.spec
$ mock $HOME/SRPMS/openvswitch-1.7.3-1.src.rpm
$ mock $HOME/SRPMS/openvswitch-kmod-1.7.3-1.el6.src.rpm

I won't go over the intricacies of RPM building since there is plenty of information available on the Fedora project's wiki. Once you have the package ready, you'll want to install it on your controller and all of your compute nodes (hypervisors). The Open vSwitch project will replace a lot of the networking functionality in the kernel, especially the stuff provided by the bridge module. The openvswitch deamon actually won't start up properly unless you reboot after the install.

When you're up and running again and the openvswitch service is running, you will need to create two bridge. One bridge is for external access and the other provides internal networking for your VMs. The cool thing is that OVS supports the standard network scripts (to an extent) available on EL based distros. An example being:

# ifcfg-br-ext
DEVICE=br-ext
ONBOOT=yes
BOOTPROTO=static
DEVICETYPE=ovs
IPADDR=10.10.20.30
NETMASK=255.255.254.0
TYPE=OVSBridge
GATEWAY=10.10.20.1

# ifcfg-em1
DEVICE=em1
ONBOOT=yes
BOOTPROTO=none
DEVICETYPE=ovs
HWADDR=FF:FF:FF:FF:FF:FF
TYPE=OVSPort
OVS_BRIDGE=br-ext

# ifcfg-br-int
DEVICE=br-int
ONBOOT=yes
BOOTPROTO=none
DEVICETYPE=ovs
TYPE=OVSBridge

Perform a network service restart, and you should be in a pretty good position. You should now see a network similar to the following:

# ovs-vsctl show
1b0f63cf-d024-488e-872e-09a4389a067c
    Bridge br-int
  Port br-int
      Interface br-int
    type: internal
    Bridge br-ext
  Port br-ext
      Interface br-ext
    type: internal
  Port "em1"
      Interface "em1"
    ovs_version: "1.7.3"

You should get your controller and compute nodes to this point before continuing, since this is probably the most difficult part.

Quantum configuration

Getting quantum going takes some thought and configuration on both your network gear and your compute/controller nodes. In my case, I'm trunking a set of VLANs down to my compute nodes and adding the gateway IPs to my core network. I'll be using the LibvirtOpenVswitchDriver driver for nova, which does not support security groups or other security filtering features. This is fine for my internal cluster, but if you're a service provider you probably want something else. I can mention now that you will need the OVS bridge compatibility daemon if you plan to use CentOS 6.3 and the LibvirtHybridOVSBridgeDriver.

You will want to follow the Quantum Admin Guide to get a bare minimal configuration going.

I'll point out a couple things that were not obvious for me. The first being you want to install the openstack-quantum-openvswitch package, which provides you with the necessary python bits to bridge the Quantum API to actual OVS commands/configuration. Next you want to ensure the quantum configuration file is correct, mine looks like this:

# cat /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
[DATABASE]
sql_connection = mysql://username:password@dbhost:3306/ovs_quantum
reconnect_interval = 2

[OVS]
tenant_network_type = vlan
network_vlan_ranges = openstack:1001:2000
bridge_mappings = openstack:br-ext

[AGENT]
polling_interval = 2
root_helper = sudo quantum-rootwrap /etc/quantum/rootwrap.conf

In this case I'll be allowing vlan IDs 1001 to 2000 to attach to a named network called openstack. This name will be used when adding networks to the quantum API later, so keep that in mind. The next important piece is the bridge_mappings, which will map your named network to a bridge that can handle these VLANs. This file needs to be the same on your controller and your compute nodes which will all play an important part in managing your network.

The second most unobvious part, is the init script for quantum-server. It passes --config /etc/quantum/plugin.ini, which doesn't exist by default with the package. I simply symlink this to the plugin I'm using, which happens to be the OVS plugin. Example:

ln -s /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini /etc/quantum/plugin.ini

l3_agent and dhcp_agent configuration

Generally the l3_agent and dhcp_agent will run on your controller node. CentOS 6.3 does not have a version of the iproute2 utilities that supports namespacing. The kernel does support it, but not the userland utilities which Quantum relies on to configure certain aspects. In order to properly work, you need to disable namespacing:

use_namespaces = False

This should exist in in both agent's configuration files (/etc/quantum/l3_agent.ini and /etc/quantum/dhcp_agent.ini respectively). You will also want to ensure a proper external_network_bridge value for the l3_agent (br-ext in my example bridges above). Once you have the two agents running on your controller, you should be in great shape. However, the dnsmasq command used by the dhcp_agent is incorrect for the version of dnsmasq included with CentOS 6.3. I've filed a bug report with a patch that you can apply to get things going.

libvirt configuration changes

At a bare minimum, your libvirt qemu.conf file needs to include the following:

vnc_listen = "0.0.0.0"
user = "root"
group = "root"

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc", "/dev/hpet", "/dev/net/tun",
]

clear_emulator_capabilities = 0

More information can be found on this on the libvirt API page. Reason we need this is for the libvirt driver we will be using for Nova. If you aren't using LibvirtOpenVswitchDriver you likely won't need this, but there is probably some other libvirt changes you will need.

Network configuration

Juniper

As I mentioned before, you need to configure both networking hardware and quantum to get things going. I'm running Juniper gear, so here's an example of setting up a vlan with gateway:

set interfaces vlan unit 1002 family inet address 10.10.10.1/24
set vlans openstack-net1 vlan-id 1002
set vlans openstack-net1 l3-interface vlan.1002
commit

This is on my core network switch, so you will need to make sure the vlan is trunked down to your access switch aka the one or many that are connected to your compute nodes. On the access switches you can do something like this:

set vlans openstack-net1 vlan-id 1002
set groups openstack-configuration interfaces <*> ether-options link-mode full-duplex
set groups openstack-configuration interfaces <*> ether-options speed 1g
set groups openstack-configuration interfaces <*> unit 0 family ethernet-switching port-mode trunk
set groups openstack-configuration interfaces <*> unit 0 family ethernet-switching vlan members openstack-net1
set groups openstack-configuration interfaces <*> unit 0 family ethernet-switching native-vlan-id 80
set interface ge-0/0/10 apply-groups openstack-configuration
set interface ge-0/0/11 apply-groups openstack-configuration
commit

An important thing to note here, is that I'm setting the native-vlan-id to 80, which is the vlan ID that the IP address I assigned to br-ext on my compute nodes can communicate on. This is essentially the same as assigning the port in access mode to the vlan id 80, since any 802.1q tagged packets will be sent here. My openstack-net1 vlan will be trunked down to the ports ge-0/0/10 and ge-0/0/11 which means I can tag packets from my compute nodes onto vlan id 1002.

Quantum

From a node with the quantum cli tool installed, you can finally add a new network that will us ethe vlan ID 1002.

$ quantum net-create --shared openstack-net1 --provider:network_type vlan \
      --provider:physical_network openstack --provider:segmentation_id 1002

See the docs for specifics, but in this case we're creating a network that is shared amongst tenants. The various --provider flags are specific to the OVS plugin. Here we are saying the network_type is vlan (OVS plugin also supports GRE based networks), that we want to use the named network openstack (remember we mapped it to br-ext earlier?), and that our vlan_id is 1002. I don't know why they chose to use segmentation_id here, it's a bit confusing, but to each his own.

Conclusion

Getting OpenStack Quantum running on a Folsom based cluster using CentOS 6.3 is challenging. Especially since the docs mostly assume you're using Ubuntu or running the dev-stack. It is possible to do, you just have to work through problems one at a time and utilize config management to ensure consistency in configuration. Hopefully the couple bug reports I mentioned throughout will get addressed in a timely manner so it will be easier for future installs. If you have any questions, I'm generally around in the #openstack channel on freenode and can probably help out to an extent. I'd like to point out there's a bunch of great people there that contributed to my own understanding of the Quantum network service.

Allan Feid - openvswitch