NetBSD is a lean, mean, fast free, open source operating system and is nicely supported under Xen, has nice features like the PF packet filter and the pkgsrc ports-like collection and runs in nearly every single hardware architecture on earth. Because of this, I decided to run NetBSD 3.1 on Xen. NetBSD can run either as the privileged domain (called dom0) or as an unprivileged guest (called domU) domain. Since I was already running Linux under Xen as a domU, I am mostly interested in running NetBSD 3.1 as a domU guest on Xen. dom0 can be either Red Hat Enterprise Linux 5.0 or Fedora Core 6, but feel free to use any other Linux distribution as most of them are Xen-ready.

As far as I know, there are some restrictions between the Xen hypervisor + dom0 kernel and domU kernel:

• You cannot mix PAE-enabled and non-PAE kernels.

  For example, you cannot run a PAE-enabled dom0 kernel and/or PAE-enabled hypervisor and a non-PAE dom0/domU kernel.

  This is currently a problem since Fedora Core 6 and Red Hat Enterprise Linux 5.0 both ship with a PAE-enabled Xen hypervisor and Xen-enabled kernels, but NetBSD does not currently ship a PAE-compatible, Xen-enabled kernel.

• You cannot mix 64-bit and 32-bit kernels.

  You cannot run a 64-bit Xen hypervisor and 64-bit dom0 kernel and a 32-bit domU kernel.

Since both Fedora Core 6 and Red Hat Enterprise Linux 5.0 ship by default with a PAE-enabled (36-bit addressable memory space) Xen hypervisor and dom0 Xen-enabled Linux kernel, the first thing that I had to do in order to run NetBSD 3.1 as domU under Xen was to recompile the Linux kernel and the Xen hypervisor with PAE support completely disabled. This is described next.

Build Xen hypervisor and dom0 kernel without PAE

You can skip to the next section if you already have a non-PAE, working Xen installation.

The first thing I had to do is to downl the SRPM (source RPM) for the latest Linux kernel, for example kernel-2.6.19-1.2895.fc6.src.rpm, then install it by running:

# rpm -i kernel-2.6.19-1.2895.fc6.src.rpm

In file /usr/src/redhat/SPECS/kernel-2.6.spec replace the following:

%ifarch i686
%define buildpae 1
# we build always xen HV with pae
%define xen_flags verbose=y crash_debug=y pae=y
%endif

with:

%ifarch i686
%define buildpae 0
# we build always xen HV with pae
%define xen_flags verbose=y crash_debug=y
%endif

This will cause the Xen hypervisor to be built without PAE support. Additionally, no PAE-enabled extra kernels will be built. The Xen kernel, however, uses its specific configuration file that has to be changed in order to disable PAE support. To disable PAE support for the Xen kernel, I reconfigured the kernel with no PAE support by running:

# rpmbuild -bp /usr/src/redhat/SPECS/kernel-2.6.spec
# cd /usr/src/redhat/BUILD/kernel-2.6.19/linux-2.6.19.i386
# cp configs/kernel-2.6.19-i686-xen.config .config
# make menuconfig

Make sure PAE is disabled by navigating to Processor type and features, then High Memory Support is set to either off or 4GB (but not 64GB).

Next, I copied the updated configuration file back to /usr/src/redhat/SOURCES, where it belongs. Also, we need to insert # i386 at the beginning of the file so that the RPM build process can derive the exact processor architecture from the config file when building the RPMs:

# cat <(echo "# i386") .config > ../../../SOURCES/kernel-2.6.19-i686-xen.config

The processor architecture is supplied to make during the build process in the form of ARCH=i386.

Now, let’s build the RPMs:

# rpmbuild -ba --target i686 ../../../SPECS/kernel-2.6.spec

We need to specify i686 as the target architecture since Fedora and Red Hat don’t use i386 anymore for kernels themselves — i386 is now only used for some common RPMs like kernel-headers.

Once the RPMs have been built, check the files under /usr/src/redhat/RPMS/i686. At least there should be a file called kernel-xen-2.6.19-1.2895.i686.rpm. This RPM contains several files, but the ones that we are interested in are:

• /boot/config-2.6.19-1.2895xen

  Contains the kernel configuration. Make sure either CONFIG_X86_PAE is set to n or is undefined.

• /boot/vmlinuz-2.6.19-1.2895xen

  The Linux Xen-enabled kernel.

• /boot/xen.gz-2.6.19-1.2895

  The Xen hypervisor. In the most recent versions of Fedora Core and Red Hat Enterprise Linux, the Xen hypervisor and the Xen-enabled kernel are packaged in the same RPM. This is the right thing to do since both are tightly coupled.

Install the new Xen kernel and hypervisor:

# rpm -ivh --force /usr/src/redhat/RPMS/i686/kernel-xen-2.6.19-1.2895.i686.rpm

Reboot:

# reboot

I assume the system will boot correctly and into the new Xen hypervisor and Xen-enabled Linux kernel. You can check that by running:

# uname -a
Linux xen 2.6.19-1.2895xen #1 SMP Sat Feb 3 16:56:34 CET 2007 i686 i686 i386 GNU/Linux

The next step is installing NetBSD 3.1 as a domU. This is covered next.

Installing NetBSD 3.1

The first step is preparing the Xen’s domU configuration file and its corresponding storage backend. Xen can use file-backed storage for a domU or block-backed storage (i.e. a disk partition or logical volume). Typically, block-backed storage is faster than file-backed storage, so I set up a 10GiB logical volume for NetBSD:

# lvcreate -n netbsd xen -L 10G

I also used NetBSD’s Internet-based installation since it’s the easiest way to get a working NetBSD installation and the NetBSD community have built Xen-enabled NetBSD kernels:

• netbsd-INSTALL_XEN3_DOMU

  A Xen-based, domU kernel used to install NetBSD.

• netbsd-XEN3_DOMU

  A Xen-based, domU kernel used to run the installed system.

Both files can be downloaded from /pub/NetBSD/NetBSD-3.1/i386/binary/kernel. Download and uncompress both of them:

# wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-3.1/i386/binary/kernel/netbsd-*XEN3_DOMU.gz
# zcat netbsd-INSTALL_XEN3_DOMU.gz > /boot/netbsd-INSTALL_XEN3_DOMU
# zcat netbsd-XEN3_DOMU.gz > /boot/netbsd-XEN3_DOMU

If you are running SELinux, you will need to relabel these files properly or xm will be unable to load them into memory:

# chcon root system_u:object_r:boot_t /boot/netbsd*

Next, create the Xen configuration file for NetBSD. In my case, it looked like this:

# cat /etc/xen/auto/netbsd
kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"
memory = 256
name = "netbsd"
vif = [ 'mac=00:16:3e:00:00:11, bridge=xenbr0' ]
disk = [ 'phy:/dev/xen/netbsd,hda,w' ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Now, we will install NetBSD by starting the domain:

# xm create -c /etc/xen/auto/netbsd

This will start the new domain and will attach to its console. You can follow the Example Installation NetBSD document to assist you in installing NetBSD and also Xensource NetBSDdomU Wiki page.

Once the installer has finished, do not reboot. At the end of the installation process, you’ll be brought back to the main install screen. Select e: Utility menu, then a: Run /bin/sh, then type the following at the shell:

mount /dev/xbd0a /mnt
cp -pR /dev/rxbd* /mnt/dev
cp -pR /dev/xbd* /mnt/dev
halt -p

This will copy the required special device files and shut down the guest. Now, you will have to modify the domain config file in order to use the standard NetBSD domU kernel, /boot/netbsd-XEN3_DOMU. Edit /etc/xen/auto/netbsd and replace:

kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"

with:

kernel = "/boot/netbsd-XEN3_DOMU"

And boot the domain again:

# xm create -c /etc/xen/auto/netbsd

During boot, you will see some errors like:


wsconscfg: /dev/ttyEcfg: Device not configured


This is due to the NetBSD guest only having access to one physical console. To kill those errors, edit /etc/ttys from within the NetBSD guest and turn off all terminals except "console", like:

console "/usr/libexec/getty Pc"         vt100   on  secure
ttyE0   "/usr/libexec/getty Pc"         vt220   off secure
ttyE1   "/usr/libexec/getty Pc"         vt220   off secure
ttyE2   "/usr/libexec/getty Pc"         vt220   off secure
ttyE3   "/usr/libexec/getty Pc"         vt220   off secure
...

Also, comment out all screens in /etc/wscons.conf:

#screen 0       -       vt100
#screen 1       -       vt100
#screen 2       -       vt100
#screen 3       -       vt100
#screen 4       -       -
#screen 4       80x25bf vt100

That’s all. Now we have a fully functional NetBSD 3.1 domU guest running on Xen

References

The information and instructions on this post are based on:

1. NetBSDdomU — How to install NetBSD as a domU on a Linux host.
2. Example Installation — NetBSD example installation.

At the end of the first article, I pointed out I was having problems with UDP traffic. In turn, it was worse than I ever expected, since it was affecting DNS name resolution, DHCP services and other services running as inside a domainU. This is the reason why I rethought the implementation and now have it working on a production machine acting, among as other things, as a DHCP server and DNS server.

In this second try I decided not to mess around with Xen’s default network configuration, so please undo all the changes you did so you end end up with a pristine Xen configuration. In this new scenario all the native traffic (tagged an untagged Ethernet frames) is being captured by Xen’s switch, xenbr0, and sent to the right network interface. If the traffic being received is a 802.11q tagged frame, the target will receive it tagged and thus will have to implement measures to untag and process it accordingly.

Introduction

So, let’s say we have the following logical network topology and virtual machines:

                   |
                  LAN
                   |
-------------------+----------------------------------
|                  |                                 |
|                peth0 ---- xen-br0                  |
|                              |                     |
|            -----------------------------           |
|            |                           |           |
|          vif0.0                     vif1.0         |
|            |                           |           |
|            |            +--------------+------------
|            |            |              |
|            |            |  ------------+------------
|            |            |  |           |           |
|           eth0          |  |          eth0         |
|            |            |  |           |           |
|     -------+-------     |  |     ------+-------    |
|     |             |     |  |     |            |    |
| eth0.1000      eth0.10  |  | eth0.2000     eth0.10 |
|     |             |     |  |     |            |    |
| VLAN 1000      VLAN 10  |  | VLAN 2000     VLAN 10 |
|     |             |     |  |     |            |    |
|    www           ssh    |  |    ftp          ssh   |
|                         |  |                       |
|        Domain0          |  |        DomainU        |
---------------------------  -------------------------

The Xen’s switch configuration can be seen with the following command:

root@xen:~# brctl show
bridge name  bridge id           STP enabled   interfaces
xenbr0       8000.feffffffffff   no            peth0
                                               vif0.0
                                               vif1.0

For each domain — this includes domain0 or any domainU — there is a vif|X|.|Y| interface attached to Xen’s bridge xen-br0, where |X| is the domain ID (0 for domain0 and a monotonically increasing number for every domainU). Then, we have every network interface card inside the domain, in the form of eth|Y|. Thus, if a domainU with ID #3 defines two network interfaces, eth0 and eth1, there will two corresponding virtual network interfaces in domain0, named vif3.0 and vif3.1.

Instead of trying to export VLAN interfaces to one or more domainUs, we export the whole, native (tagged or not) network interface to the domainU and, inside this domainU, we can configure VLAN subinterfaces if needed.

Sample scenario

Let’s say we want to offer the following services per VLAN:

• WWW server on VLAN 1000
• FTP server on VLAN 2000
• SSH access to administer the WWW sever, reachable only through the VLAN 10
• SSH server to administer the FTP server, reachable only through the VLAN 10

But we also want to partition the physical machine in two, so domain0 serves WWW traffic while domainU servers FTP traffic:

Thus, we need the following VLAN subinterfaces:

• eth0.10 and eth0.1000 on domain0
• eth0.10 and eth0.2000 on domainU

Configuring VLAN subinterfaces in domainU is straight forward. However, it’s a little bit more difficult for domain0.

Configuring VLAN subinterfaces for domain0

First of all, make sure you are using bridging for your Xen configuration. Make sure the following line is uncommented in /etc/xen/xend-config.sxp:

(network-script network-bridge)

And comment any other network-script configuration lines, like:

(network-script network-nat)

or

(network-script network-route)

It seems we can’t bring up VLAN subinterfaces before Xen’s network script is fired up since Xen’s network scripts perform some black magic on the network interfaces, mainly renaming eth0 to peth0 and bringing up a dummy interface named eth0. Any subinterface related to the original eth0 seems to stop working after the renaming takes place.

Thus, I coded up an init script used to bring up the VLAN subinterfaces that gets invoked just after Xen’s network script has finished. Note that it’s targeted for RedHat-based distributions:

#!/bin/sh
#
# Init file for Network-VLAN
# STARTS AFTER XEN (which is S50 and K01)
#
# chkconfig: 2345 51 89
# description: VLAN networking

. /etc/init.d/functions

case "$1" in
start)
 echo -n $"Configuring VLAN interfaces:"

 if [ ! -f /var/lock/subsys/network-vlan ]; then
  (
  modprobe 8021q || exit 1
  vconfig add eth0 10 || exit 2
  ifconfig eth0.10 up 10.0.0.1 netmask 255.0.0.0 || exit 3
  vconfig add eth0 1000 || exit 2
  ifconfig eth0.1000 up 11.0.0.1 netmask 255.0.0.0 || exit 3
  ) > /dev/null 2>&1

  RETVAL=$?
  [ "$RETVAL" = 0 ] && ( success ;\\
    touch /var/lock/subsys/network-vlan ) || failure
 fi
 echo

 ;;

stop)
 echo -n $"Unconfiguring VLAN interfaces:"

 if [ -f /var/lock/subsys/network-vlan ]; then
  (
  ifconfig eth0.10 down && vconfig rem eth0.10 ;
  ifconfig eth0.1000 down && vconfig rem eth0.1000
  ) > /dev/null 2>&1

  RETVAL=$?
  [ "$RETVAL" = 0 ] && ( rm -f /var/lock/subsys/network-vlan ;\\
    success ) || failure
 fi
 echo
esac

Save this script as /etc/init.d/network-vlan, then run:

chmod +x /etc/init.d/network-vlan
chkconfig --add /etc/init.d/network-vlan

The script runs just after Xen’s init script has renamed the real Ethernet interface and has brought up a dummy interface called eth0. Then, the network-vlan script brings up two VLAN subinterfaces, one for VLAN 10 and another one for VLAN 1000, and then assigns each one its own IP address.

Additionally, these are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

Note that eth0 in this context refers to the real Ethernet interface, since Xen’s init script has not been ran yet. I didn’t configure any IP address for this interface since I only want to process tagged traffic. Beware that on many switches — i.e., Cisco 2960 and 3560 —, VLAN1 is, by default, the native VLAN and traffic on the native VLAN doesn’t get tagged.

Configuring VLAN subinterfaces for domainU

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

I didn’t configure any IP address for this interface since I only want to process tagged traffic. Read the note above on untagged frames and native VLANs.

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.10:

DEVICE=eth0.10
BOOTPROTO=static
IPADDR=10.0.0.2
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.2000:

DEVICE=eth0.2000
BOOTPROTO=static
IPADDR=12.0.0.1
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

Bonding

For those who desire to use bonding, it seems some tweaking of the networking scripts is required. I recommend them to look at this post on Bonding not working with network-bridge.

Conclusion

I’m sure there are better ways to configure VLAN subinterfaces in domain0, but it was in a hurry and couldn’t find of a better way to get it done.

If anyone out there has a different way of achieving this, please let me know

In this short article, I describe the steps needed to configure Xen to attach itself to multiple VLANs using a one-bridge-per-VLAN network interface mapping, then attaching each Xen domainU on as many VLANs as needed.

In the sample scenario, we will use a Cisco Catalyst 3560G-24TS switch carrying traffic from five different VLANs:

• VLAN2 is the administrative VLAN used to administer all the networking gear and boxes.
• VLAN10 carries Internet traffic coming from the first ISP.
• VLAN20 carries Internet traffic coming from the second ISP.
• VLAN100 carries the access network traffic.
• VLAN200 carries the core network traffic.

The final Xen configuration will provide five bridging network interfaces, one per VLAN. Each Xen domainU can freely attach to any of these bridging network interfaces in order to gain access to the traffic being carried by each VLAN.

The bridging interface, |brname| is named after the following convention: xenbr|vlan|:

• xenbr2 is the bridging interface standing on VLAN2.
• xenbr10 is the bridging interface standing on VLAN10.
• xenbr20 is the bridging interface standing on VLAN20.
• xenbr100 is the bridging interface standing on VLAN100.
• xenbr200 is the bridging interface standing on VLAN200.

Also, Xen creates an manages several virtual network interfaces, named in the form of vif|X|.|Y|, where |X| equals the Xen domain numeric ID and |Y| is a sequential interface index. Thus, starting up a Xen domainU given the following virtual network interface definition:

vif = [ 'mac=00:16:3e:00:00:44, bridge=xenbr10',
        'mac=00:16:e3:00:00:45, bridge=xenbr20' ]

Will cause the Xen domain to get assigned, let’s say, a domain ID of 2, and two virtual network interfaces named vif2.0 — attached to xenbr10 — and vif2.1 — attached to xenbr20.

Setting up the bridging interfaces:

This can be done manually, by invoking brctl addbr |brname| in order to create a new bridging interface.

For example, the following commands will create five bridging interfaces, one for each supported VLAN:

brctl addbr xenbr2
brctl addbr xenbr10
brctl addbr xenbr20
brctl addbr xenbr100
brctl addbr xenbr200

or else can be automated to get done during system startup, by creating a file named /etc/sysconfig/network-scripts/ifcfg-|brname|, where |brname| is the name assigned to the bridging interface, like /etc/sysconfig/network-scripts/ifcfg-xenbr2 (the configuration file for the bridging interface standing on VLAN2):

DEVICE=xenbr2
BOOTPROTO=static
IPADDR=192.168.0.10
NETMASK=255.255.0.0
ONBOOT=yes
TYPE=Bridge

Setting up the VLAN interfaces and add them up to the existing bridging interfaces:

This can be done manually, by invoking vconfig add |ifname| |vlan| to configure VLAN number |vlan| by using 802.1q tagging on interface |ifname|. This will active a virtual interface named |ifname|.|vlan|:

• Any traffic sent to this interface will get tagged for VLAN |vlan|.
• Any traffic received from interface |ifname| carrying an 802.1q VLAN tag matching |vlan| will be untagged and received by this interface.

vconfig add eth0 2
vconfig add eth0 10
vconfig add eth0 20
vconfig add eth0 100
vconfig add eth0 200

This will add five new VLAN interfaces, one for every supported VLAN.

Once the VLAN interfaces are ready, we add them to their corresponding bridging interfaces by using brctl addif |brname| |ifname|.|vlan|:

brctl addif xenbr2 eth0.2
brctl addif xenbr10 eth0.10
brctl addif xenbr20 eth0.20
brctl addif xenbr100 eth0.100
brctl addif xenbr200 eth0.200

The process of adding up a new VLAN interface and then adding it up to an existing bridging interface can be configured using a single configuration file named ifcfg-|ifname|.|vlan|, like /etc/sysconfig/network-scripts/ifcfg-eth0.2:

DEVICE=eth0.2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
VLAN=yes
BRIDGE=xenbr2

Keeping Xen from reconfiguring the network:

Since we have already configured the network manually, we don’t want Xen to mess up with the configuration. In order to keep Xen from reconfiguring the network, simply make sure none of the following lines appear uncommented in the file /etc/xen/xend-config.sxp:

(network-script network-bridge)
(network-script network-route)
(network-script network-nat)

Additional notes:

I have been experiencing a very strange behavior on Xen domainU guests while using this network configuration: it seems that UDP traffic gets stuck at the network stack and does not flow through unless I load the ip_conntrack.ko kernel module.

Failing to load the ip_conntrack.ko kernel module, even with an unconfigured, empty firewall, allows ICMP and TCP traffic to flow from and to the guest network stack, but UDP traffic, like DNS queries, gets stuck and doesn’t even touch the physical network interface.

This is really strange, isn’t it?

Xen is an hypervisor that allows running several partitions, called “domains”. Each domain is totally independent, virtualized and solated by the Xen hypervisor, which controls access to the hardware and shares the physical resources among all running domains.

There are two domain classes:

1. Privileged domain: has some limited access to the hardware, and is used to provide for virtual driver support for all the non-privileged domains. There can only be one privileged domain running on a machine at any time and its called “domain0″.

   For example, the privileged domain has access to the video frame buffer and can run an accelerated X server, and has limited access to real devices, like eth0, hda, and so on.

2. Unprivileged domain: has no direct hardware access, and all the I/O operations are virtualized by using virtual devices, like the virtual block device or virtual network device, which are in turn hooked to the privileged domain. There can be as many unprivileged domains as system resources allow for.

   Unprivileged domains don’t have a video frame buffer and, thus, all GUI support must get implented using software like VNCserver or FreeNX, then remotely controlling the session from another domain or machine.

Xen selects a random hardware MAC address if not specifically told to use a particular one. This causes problems with networking since the MAC address for the domain virtual interface (called vif) changes between domain reboots and switches and computers keep the corresponding entry on their ARP caches for a few minutes.

Let be A a virtual unprivileged domain running under Xen’s control. After A has started, any other domain or real machine, will be unable to talk to A until the moment A forces an ARP cache update by sending out a packet (for example, an ICMP packet).

I see two solutions to this problem:

1. When domain A starts, it initiates an ARP cache refresh of other domains or machines.

   This can be achieved by sending a gratuitious ARP cache on interface wakeup, which not every Linux distribution does, or send some ICMP echo request packets to force sending ARP requests. This can be easily implemented in init scripts.

2. Forcing domain A to use a fixed MAC address.

   This can be achieved by using the following line in A domain configuration file:

   vif = [ 'mac = XX:XX:XX:XX:XX:XX' ]

   The latter solution is the easiest and, usually, the preferred one.

   NOTE: This also covered in the Xen FAQ.