UPDATE: October 29, 2012

This article doesn’t work on Ubuntu 12.10, to use Ubuntu 12.10 please refer to my new article here.

Openvswitch is a better way of managing your virtual networking stacks for KVM and Xen.  This can be used instead of the bridge-utils package, and has the ability to use VLANs, LACP, QoS, sFlow, and others.  In this article we will be using Ubuntu 12.04 (beta1) to configure Openvswitch to hand out networking interfaces to KVM guests.

Please keep in mind that at the time of this writing Ubuntu 12.04 is beta, which means it is not fully baked for production.  Though it is looking very stable.

Install Updates and Prerequisite Software

Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.

bash 
apt-get update && apt-get dist-upgrade
apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch openvswitch-datapath-source

Destroy the Default Libvirt Bridge (virbr0)

We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).

bash 
virsh net-destroy default
virsh net-autostart --disable default

Stop Libvirt and Qemu

This will prevent libvirt from bringing up the legacy bridge.

bash 
service libvirt-bin stop
service qemu-kvm stop

Enable Openvswitch for Brcompat

bash 
vi /etc/default/openvswitch-switch
BRCOMPAT=yes

Purge Ebtables from System

Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.

bash 
aptitude purge ebtables

Fix DKMS Module Build

When we originally installed openvswitch, you might have noticed an error talking about the module not available for your kernel, if you didn’t notice it you can see the same error by trying to start the openvswitch-switch service.

bash 
service openvswitch-switch start

If you receive the module error, then we will need to build the openvswitch-datapath module.

bash 
module-assistant auto-install openvswitch-datapath

Note that after installing new kernels you will most likely need to repeat this step, to rebuild the module.

Restart the Openvswitch Services

bash 
service openvswitch-switch restart
service openvswitch-controller restart

Reboot

bash 
reboot

Check Configuration

Based on your reboot one of two things happened…  Your modules came back up loaded and ready for business, though more likely they didn’t come up.

Here is an example of a working configuration.  If you get this then move on to configuration of the switch itself.

bash 
lsmod | grep brcom
brcompat_mod 13512 0
openvswitch_mod 83993 2 brcompat_mod
service openvswitch-switch status
ovsdb-server is running with pid 1885
ovs-vswitchd is running with pid 1908
ovs-brcompatd is running with pid 2096

Here is an example of a non-working configuration.  If your modules are not loaded then you will need to get that sorted before we blow a configuration on there and try to pass traffic.

bash 
lsmod | grep brcom
service openvswitch-switch status
ovsdb-server is running with pid 1885
ovs-vswitchd is running with pid 1908
ovs-brcompatd is running with pid 2096

Ensuring the Modules Load at Boot

The easy way to fix this is to restart the openvswitch-switch service whenever you reboot your hypervisor.

bash 
service openvswitch-switch restart
* Killing ovs-brcompatd (2096)
* Killing ovs-vswitchd (1908)
* Killing ovsdb-server (1885)
* Starting ovsdb-server
* Configuring Open vSwitch system IDs
* Starting ovs-vswitchd
* Starting ovs-brcompatd
* iptables already has a rule for gre, not explicitly enabling

But that might not be ideal for your situation.  As an alternative we can try and address the timing issue which is causing the whole issue in the first place.  Basically what is happening is that upstart is starting openvswitch too late, and it is failing.  If we adjust the waits in /etc/init/failsafe.conf

We want to change this…

text 
$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 40
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 59
$PLYMOUTH message --text="Booting system without full network configuration..." || :

To this…

text 
$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :

Now after a reboot you shouldn’t have this problem with the modules not loading.

Reboot

bash 
reboot

Check Modules

bash 
lsmod | grep brcom
brcompat_mod           13512  0
openvswitch_mod        83993  1 brcompat_mod

Design our Network Layout

In our example we will use 2 physical interfaces, one of which (eth0) we will consider our management interface, it will have a static IP address and will be “the hypervisor” the other one (eth1) will be the uplink for our Openvswitch.  Now eth1 will have an untagged fake bridge (br0) which will be the parent for our other fake bridge (br10) which will be tagged 10.  If we had more to add they would still use br0 as the parent and then be tagged on the subordinate fake bridge.  Another important thing to keep in mind is that your uplinked switch port must support and be configured to allow your devices to tag for these other VLANs.

Configure Openvswitch Fake Bridges

bash 
ovs-vsctl add-br br0
ovs-vsctl add-port br0 eth1
ovs-vsctl add-br br10 br0 10
ovs-vsctl list-br
br0
br10

Create Dummy Interface Entries

We need to edit /etc/network/interfaces to add some entries

bash 
cat /etc/network/interfaces
auto eth0
iface eth0 inet static
address 10.0.0.141
netmask 255.255.255.0
gateway 10.0.0.1

auto eth1
iface eth1 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto br0
iface br0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto br10
iface br10 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

These bring up the interfaces with No IP address, now if you run ifconfig -a they should be visible.

Once you have a guest up on br0 or br10 you will see a  “vnet0” interface.