# cat /etc/hostname.vr2
dhcp
inet 1.1.1.11 255.255.255.0 NONE
up
# sh /etc/netstart vr2

The problem with this approach is that dhclient never gets daemonized because netstart gets it annoyed: dhclient notices that something else reconfigured the interface and commits suicide. So, then I thought about reversing the order of the first two lines:

# cat /etc/hostname.vr2
inet 1.1.1.11 255.255.255.0 NONE
dhcp
up
# sh /etc/netstart vr2

Now dhclient daemonizes but also removes all previously configured IP addresses, so the statically configured address configured via the first line is wiped by dhclient. Not very nice.

Turns out the solution lies in /etc/dhclient.conf:

# cat /etc/dhclient.conf
interface "vr2" {
        supersede domain-name "example.com";
        supersede domain-name-servers 1.1.1.1;
}

alias {
        interface "vr2";
        fixed-address 1.1.1.11;
        option subnet-mask 255.255.255.0;
}

The alias stanza allows one to define an additional, aliased IP address for an interface. Which allows the machine to be always reachable on a fixed IP address.

Neat.

The process consists on the following main steps:

1. Setting up the DHCP server
2. Setting up the TFTP server and configuring PXE boot
3. Setting up the NFS server
4. Bootstrapping a Ubuntu installation into the client’s root filesystem
5. Booting up the diskless workstation

1. Setting up the DHCP server

PXE-enabled workstations need to get an additional option during the DHCP negotiation that will point them to a TFTP server where the PXE-compatible boot loader code can be downloaded. Configuring dnsmasq to hand this option to clients is just as easy as adding the following line to /etc/dnsmasq.conf:

dhcp-boot=pxelinux.0,tftp.lan,10.42.242.13

and restarting the dnsmasq service.

2. Setting up the TFTP server and configuring PXE boot

Now that DHCP has been configured, the next step is setting up the TFTP server. The TFTP server will be used by PXE-compatible clients to download the PXE boot loader code, and also the Linux kernel and Linux initial RAM disk.

I will use H. P. Anvin’s TFTP server as it’s widely used and works fairly well:

root@tftp.lan:# apt-get install tftpd-hpa

tftpd-hpa does not integrate automatically with xinetd so if you want to run the TFTP server under xinetd, you will have to create the following file, which was ported from the inetd description that is created automatically in /etc/inetd.conf when tftpd-hpa is installed:

service tftp
{
        disable         = no
        id              = chargen-dgram
        socket_type     = dgram
        protocol        = udp
        user            = root
        wait            = yes
        server          = /usr/sbin/in.tftpd
        server_args     = -s /var/lib/tftpboot/
}

then restarting the xinetd service.

Configuring PXE boot is just a matter of copying the PXE boot loader code, a configuration file, the Linux kernel and initial RAM disks under the TFTP root. First, install syslinux and copy the PXE boot loader code to the TFTP server root:

root@tftp.lan:# apt-get install syslinux
root@tftp.lan:# cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
root@tftp.lan:# mkdir /var/lib/tftpboot/pxelinux.cfg

The Linux PXE boot loader code, syslinux, expects that a configuration file describing what kernel, its boot parameters, and the initial RAM disk to use to be stored within a directory named pxelinux.cfg just under the TFTP server root.

Client-specific config files can be created (based on the client MAC address, for example). We will create a config file that is suitable for any client. This configuration file is called default and can be created by running the following commands:

root@tftp.lan:# KERNEL_VERSION=2.6.22-14-generic
root@tftp.lan:# NFS_IPADDR=$(host nfs.lan | cut -d' ' -f4)
root@tftp.lan:# cat > /var/lib/tftpboot/pxelinux.cfg/default << EOF
> LABEL linux
> KERNEL vmlinuz-${KERNEL_VERSION}
> APPEND root=/dev/nfs initrd=initrd.img-${KERNEL_VERSION} nfsroot=${NFS_IPADDR}:/home/nfsroot ip=dhcp rw
> EOF

3. Setting up the NFS server

In this step, we will configure the NFS server and export the directory where the client’s root filesystem will be stored.

Let’s start by installing the NFS server packages:

root@nfs.lan:# apt-get install nfs-kernel-server nfs-common

I will use /home/nfsroot as the root for the client’s root filesystem

root@nfs.lan:# mkdir /home/nfsroot

Next, add the following line to /etc/exports in order to export the the client’s root filesystem:

/home/nfsroot *,gss/krb5(rw,no_subtree_check,async,no_root_squash)

Then re-export all the filesystems:

root@nfs.lan:# exportfs -avr

4. Bootstrapping a Ubuntu installation into the client’s root filesystem

The following steps will bootstrap the installation of a minimal Ubuntu Hardy Heron GNU/Linux system into the client’s root:

root@nfs.lan:# debootstrap --arch i386 hardy \
  /home/nfsroot http://ch.archive.ubuntu.com/ubuntu/

Only the minimum required packages will be downloaded from the Internet and installed into /home/nfsroot. The output of the previous command should look like this:

I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional required dependencies: libdb4.6
I: Checking component main on http://ch.archive.ubuntu.com/ubuntu...
I: Retrieving adduser
I: Validating adduser
...
I: Base system installed successfully.

Once the system has been bootstrapped, we need to populate fstab. At least, /proc and / must get mounted, but other filesystems might be referenced in this file too, like additional NFS exports, swap files, and so on. The difference with respect a traditional, disk-based Ubuntu installation, is that the root filesystem gets mounted via NFS by the intial RAM disk, and is referenced by the kernel’s /dev/nfs block device.

The contents of /home/nfsroot/etc/fstab should look like this:

# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc            /proc         proc   defaults       0      0
/dev/nfs        /             nfs    defaults       0      0

Another difference with a traditional Ubuntu system is that we don’t want the network interfaces be managed by Network Manager. In fact, the main (probably Ethernet) network interface was already configured by the kernel based on PXE’s supplied configuration. Letting Nework Manager reconfigure or manage this network interface might mean losing the connection to the NFS server and, thus, rendering the system unusable.

To stop Network Manager from managing the main network interface, the contents of /home/nfsroot/etc/network/interfaces must look like this:

# Used by ifup(8) and ifdown(8). See the interfaces(5) manpage or
# /usr/share/doc/ifupdown/examples for more information.

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface, commented out for NFS root
iface eth0 inet manual

Finally, we will give the client a generic name. This can be done by storing that generic name into /etc/hostname:

root@nfs.lan:# echo client.lan > /home/nfsroot/etc/hostname

Booting up the diskless workstation

Once everything is set up, the last thing to do is testing that the diskless workstation is able to boot via PXE. The mechanics of booting via PXE depend on the machine. On some machines, it’s just a matter of pressing ESC to get into a menu that allows to override the boot device. On others, the same can be done by pressing F12 during the POST. On most modern systems it’s possible to reconfigure the system to always boot from the network.

If the client is able to boot from PXE successfully, this is what you will briefly see on your screen:

CLIENT MAC ADDR: 00 11 22 33 44 55 66  GUID: 00000000 0000 0000 0000 000000000000
CLIENT IP: 192.168.0.2  MASK: 255.255.255.0  DHCP IP: 192.168.0.1
GATEWAY IP: 192.168.0.1

PXELNUX 3.36 Debian-2007-08-30  Copyright (C) 1994-2007 H. Peter Anvin
UNDI data segment at:   00094140
UNDI data segment size: 94B0
UNDI code segment at:   0009D5F0
UNDI code segment size: 20B0
PXE entry point found (we hope) at 9D5F:0106
My IP address seems to be C0A80001 192.168.0.2
ip=192.168.0.2:192.168.0.1:192.168.0.1:255.255.255.0
TFTP prefix:
Tring to load: pxelinux.cfg/00-01-02-03-04-05
Tring to load: pxelinux.cfg/C0A80001
Tring to load: pxelinux.cfg/C0A8000
Tring to load: pxelinux.cfg/C0A800
Tring to load: pxelinux.cfg/C0A80
Tring to load: pxelinux.cfg/C0A8
Tring to load: pxelinux.cfg/C0A
Tring to load: pxelinux.cfg/C0
Tring to load: pxelinux.cfg/C
Tring to load: pxelinux.cfg/default
boot:
Loading vmlinuz...

If the system boots up, we will be dropped into a text-mode console where we can log in as root with no password. Of course, the first thing you should do is creating a password for the root user, but if you have been able to get to this point, you probably know how to do it

Once you are logged in as your in your new diskless workstation, you will probably want to add more functionality. For example, installing the OpenSSH server, or the packages for the typical Ubuntu desktop system. Before you do that, we will need to add more Ubuntu repositories to /etc/apt/sources.list. This is how mine looks like:

deb http://ch.archive.ubuntu.com/ubuntu/ feisty main restricted
deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty main restricted
deb http://ch.archive.ubuntu.com/ubuntu/ feisty-updates main restricted
deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty-updates main restricted
deb http://ch.archive.ubuntu.com/ubuntu/ feisty universe
deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty universe
deb http://ch.archive.ubuntu.com/ubuntu/ feisty multiverse
deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty multiverse
deb http://ch.archive.ubuntu.com/ubuntu/ feisty-backports main restricted universe multiverse
deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu feisty-security main restricted
deb-src http://security.ubuntu.com/ubuntu feisty-security main restricted
deb http://security.ubuntu.com/ubuntu feisty-security universe
deb-src http://security.ubuntu.com/ubuntu feisty-security universe
deb http://security.ubuntu.com/ubuntu feisty-security multiverse
deb-src http://security.ubuntu.com/ubuntu feisty-security multiverse

Finally, I decided to install OpenSSH, OpenNTPD and the typical Ubuntu desktop:

root@client.lan:# apt-get update
root@client.lan:# apt-get install openssh-server openntpd ubuntu-desktop

dnsmasq acts as a caching DNS server and DHCP server. It reserves a DNS domain, called the local DNS domain and usually being .lan, for local name resolution. When queried for an A RR inside the local DNS domain, dnsmasq looks at file /etc/hosts for a match. If one is found, its corresponding IP is returned as the query result. When queried for a PTR RR, it looks into file /etc/hosts for a match by IP and, if one is found, its correspoding hostname, qualified with the local DNS domain, is returned. Thus, /etc/hosts behaves much like a DNS master zone file.

Also, if the DHCP server funcionality of dnsmasq is enabled, when a query under the local DNS domain fails (no record is found in /etc/hosts), it will try to resolve the query from the DHCP lease database.

The DHCP lease database is usually stored at /tmp/dhcp.leases. Its format is pretty simple: it’s a text file, where each line represents an active DHCP lease. Each line is made up of five fields:

1. Time of lease expiration

   In epoch time (seconds since 1970). States when the lease will expire. Most DHCP clients will try to renew the lease when it reaches 80% of its valid lifetime.

2. Client MAC address

   The MAC address corresponding to the client to which the lease belongs.

3. Leased IP address

   A valid IP address, taken from the DHCP pool, which is actually and currently leased to the client whose MAC address is listed in the previous field.

4. Client hostname

   If known, holds the unqualified host name of the client machine. Else, an asterisk is stored here.

5. Client ID

   Simon Kelley defines it as:

   The client-ID is used as the computer’s unique-ID in preference to the MAC address, if it’s available. Some DHCP clients provide it, and some don’t. The ones that do normally derive it from the MAC address unless explicity configured, but it could be something like a serial number, which would protect a computer from losing its identify if the network interface were replaced.

   If not know, an asterisk is stored here.

A sample DHCP database lease:

# cat /tmp/dhcp.leases
1147729862 00:16:3e:3b:56:f1 192.168.0.11 rhel *
1147725355 00:0c:29:09:3d:58 192.168.0.10 rhel-devel *

In this case, there are two active DHCP leases, one for client rhel, another one for rhel-devel.

OpenWRT uses a rc.d script stored at /etc/init.d/S50dnsmasq which, for a squashfs firmware is a symbolic link to /rom/etc/init.d/S50dnsmasq. This rc.d script tries to configure the dnsmasq daemon using NVRAM variables, which helps a lot when reflashing. However, I have found more convenient to the use the traditional /etc/dnsmasq.conf file instead.

Replacing the OpenWRT rc.d script with a custom one, in order to leverage dnsmasq.conf, is as simple as removing /etc/init.d/S50dnsmasq and invoking the dnsmasq daemon directly:

rm -f /etc/init.d/S50dnsmasq
cat > /etc/init.d/S50dnsmasq < < EOF
#/bin/sh
/usr/sbin/dnsmasq
EOF

Here is a sample of a /etc/dnsmasq.conf file I use on my Linksys WRT54G router running OpenWRT White Russian RC5:

# filter what we send upstream
domain-needed
bogus-priv
filterwin2k
localise-queries

# allow /etc/hosts and dhcp lookups via *.lan
local=/lan/
domain=lan
expand-hosts

# enable dhcp (start,end,netmask,leasetime)
dhcp-authoritative
dhcp-range=10.0.0.10,10.0.0.100,255.255.255.128,12h
dhcp-leasefile=/tmp/dhcp.leases

# use /etc/ethers for static hosts; same format as --dhcp-host
# [hwaddr] [ipaddr]
read-ethers

# other useful options:
# default route(s): dhcp-option=3,192.168.1.1,192.168.1.2
#    dns server(s): dhcp-option=6,192.168.1.1,192.168.1.2
dhcp-option=3,10.0.0.126
dhcp-option=6,10.0.0.122