Configuring a diskless Ubuntu
December 9th, 2007
This post is not about doing a PXE-based network installation of Ubuntu. There are already many posts describing how to do this. This post is about setting up an Ubuntu workstation in diskless mode, such as the workstation boots via PXE and the root filesystem is mounted over NFS.
The process consists on the following main steps:
- Setting up the DHCP server
- Setting up the TFTP server and configuring PXE boot
- Setting up the NFS server
- Bootstrapping a Ubuntu installation into the client’s root filesystem
- Booting up the diskless workstation
1. Setting up the DHCP server
PXE-enabled workstations need to get an additional option during the DHCP negotiation that will point them to a TFTP server where the PXE-compatible boot loader code can be downloaded. Configuring dnsmasq to hand this option to clients is just as easy as adding the following line to /etc/dnsmasq.conf:
dhcp-boot=pxelinux.0,tftp.lan,10.42.242.13
and restarting the dnsmasq service.
2. Setting up the TFTP server and configuring PXE boot
Now that DHCP has been configured, the next step is setting up the TFTP server. The TFTP server will be used by PXE-compatible clients to download the PXE boot loader code, and also the Linux kernel and Linux initial RAM disk.
I will use H. P. Anvin’s TFTP server as it’s widely used and works fairly well:
root@tftp.lan:# apt-get install tftpd-hpa
tftpd-hpa does not integrate automatically with xinetd so if you want to run the TFTP server under xinetd, you will have to create the following file, which was ported from the inetd description that is created automatically in /etc/inetd.conf when tftpd-hpa is installed:
service tftp
{
disable = no
id = chargen-dgram
socket_type = dgram
protocol = udp
user = root
wait = yes
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot/
}
then restarting the xinetd service.
Configuring PXE boot is just a matter of copying the PXE boot loader code, a configuration file, the Linux kernel and initial RAM disks under the TFTP root. First, install syslinux and copy the PXE boot loader code to the TFTP server root:
root@tftp.lan:# apt-get install syslinux root@tftp.lan:# cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/ root@tftp.lan:# mkdir /var/lib/tftpboot/pxelinux.cfg
The Linux PXE boot loader code, syslinux, expects that a configuration file describing what kernel, its boot parameters, and the initial RAM disk to use to be stored within a directory named pxelinux.cfg just under the TFTP server root.
Client-specific config files can be created (based on the client MAC address, for example). We will create a config file that is suitable for any client. This configuration file is called default and can be created by running the following commands:
root@tftp.lan:# KERNEL_VERSION=2.6.22-14-generic
root@tftp.lan:# NFS_IPADDR=$(host nfs.lan | cut -d' ' -f4)
root@tftp.lan:# cat > /var/lib/tftpboot/pxelinux.cfg/default << EOF
> LABEL linux
> KERNEL vmlinuz-${KERNEL_VERSION}
> APPEND root=/dev/nfs initrd=initrd.img-${KERNEL_VERSION} nfsroot=${NFS_IPADDR}:/home/nfsroot ip=dhcp rw
> EOF
3. Setting up the NFS server
In this step, we will configure the NFS server and export the directory where the client’s root filesystem will be stored.
Let’s start by installing the NFS server packages:
root@nfs.lan:# apt-get install nfs-kernel-server nfs-common
I will use /home/nfsroot as the root for the client’s root filesystem
root@nfs.lan:# mkdir /home/nfsroot
Next, add the following line to /etc/exports in order to export the the client’s root filesystem:
/home/nfsroot *,gss/krb5(rw,no_subtree_check,async,no_root_squash)
Then re-export all the filesystems:
root@nfs.lan:# exportfs -avr
4. Bootstrapping a Ubuntu installation into the client’s root filesystem
The following steps will bootstrap the installation of a minimal Ubuntu Hardy Heron GNU/Linux system into the client’s root:
root@nfs.lan:# debootstrap --arch i386 hardy \ /home/nfsroot http://ch.archive.ubuntu.com/ubuntu/
Only the minimum required packages will be downloaded from the Internet and installed into /home/nfsroot. The output of the previous command should look like this:
I: Retrieving Release I: Retrieving Packages I: Validating Packages I: Resolving dependencies of required packages... I: Resolving dependencies of base packages... I: Found additional required dependencies: libdb4.6 I: Checking component main on http://ch.archive.ubuntu.com/ubuntu... I: Retrieving adduser I: Validating adduser ... I: Base system installed successfully.
Once the system has been bootstrapped, we need to populate fstab. At least, /proc and / must get mounted, but other filesystems might be referenced in this file too, like additional NFS exports, swap files, and so on. The difference with respect a traditional, disk-based Ubuntu installation, is that the root filesystem gets mounted via NFS by the intial RAM disk, and is referenced by the kernel’s /dev/nfs block device.
The contents of /home/nfsroot/etc/fstab should look like this:
# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/nfs / nfs defaults 0 0
Another difference with a traditional Ubuntu system is that we don’t want the network interfaces be managed by Network Manager. In fact, the main (probably Ethernet) network interface was already configured by the kernel based on PXE’s supplied configuration. Letting Nework Manager reconfigure or manage this network interface might mean losing the connection to the NFS server and, thus, rendering the system unusable.
To stop Network Manager from managing the main network interface, the contents of /home/nfsroot/etc/network/interfaces must look like this:
# Used by ifup(8) and ifdown(8). See the interfaces(5) manpage or # /usr/share/doc/ifupdown/examples for more information. # The loopback network interface auto lo iface lo inet loopback # The primary network interface, commented out for NFS root iface eth0 inet manual
Finally, we will give the client a generic name. This can be done by storing that generic name into /etc/hostname:
root@nfs.lan:# echo client.lan > /home/nfsroot/etc/hostname
Booting up the diskless workstation
Once everything is set up, the last thing to do is testing that the diskless workstation is able to boot via PXE. The mechanics of booting via PXE depend on the machine. On some machines, it’s just a matter of pressing ESC to get into a menu that allows to override the boot device. On others, the same can be done by pressing F12 during the POST. On most modern systems it’s possible to reconfigure the system to always boot from the network.
If the client is able to boot from PXE successfully, this is what you will briefly see on your screen:
CLIENT MAC ADDR: 00 11 22 33 44 55 66 GUID: 00000000 0000 0000 0000 000000000000 CLIENT IP: 192.168.0.2 MASK: 255.255.255.0 DHCP IP: 192.168.0.1 GATEWAY IP: 192.168.0.1 PXELNUX 3.36 Debian-2007-08-30 Copyright (C) 1994-2007 H. Peter Anvin UNDI data segment at: 00094140 UNDI data segment size: 94B0 UNDI code segment at: 0009D5F0 UNDI code segment size: 20B0 PXE entry point found (we hope) at 9D5F:0106 My IP address seems to be C0A80001 192.168.0.2 ip=192.168.0.2:192.168.0.1:192.168.0.1:255.255.255.0 TFTP prefix: Tring to load: pxelinux.cfg/00-01-02-03-04-05 Tring to load: pxelinux.cfg/C0A80001 Tring to load: pxelinux.cfg/C0A8000 Tring to load: pxelinux.cfg/C0A800 Tring to load: pxelinux.cfg/C0A80 Tring to load: pxelinux.cfg/C0A8 Tring to load: pxelinux.cfg/C0A Tring to load: pxelinux.cfg/C0 Tring to load: pxelinux.cfg/C Tring to load: pxelinux.cfg/default boot: Loading vmlinuz...
If the system boots up, we will be dropped into a text-mode console where we can log in as root with no password. Of course, the first thing you should do is creating a password for the root user, but if you have been able to get to this point, you probably know how to do it 
Once you are logged in as your in your new diskless workstation, you will probably want to add more functionality. For example, installing the OpenSSH server, or the packages for the typical Ubuntu desktop system. Before you do that, we will need to add more Ubuntu repositories to /etc/apt/sources.list. This is how mine looks like:
deb http://ch.archive.ubuntu.com/ubuntu/ feisty main restricted deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty main restricted deb http://ch.archive.ubuntu.com/ubuntu/ feisty-updates main restricted deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty-updates main restricted deb http://ch.archive.ubuntu.com/ubuntu/ feisty universe deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty universe deb http://ch.archive.ubuntu.com/ubuntu/ feisty multiverse deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty multiverse deb http://ch.archive.ubuntu.com/ubuntu/ feisty-backports main restricted universe multiverse deb-src http://ch.archive.ubuntu.com/ubuntu/ feisty-backports main restricted universe multiverse deb http://security.ubuntu.com/ubuntu feisty-security main restricted deb-src http://security.ubuntu.com/ubuntu feisty-security main restricted deb http://security.ubuntu.com/ubuntu feisty-security universe deb-src http://security.ubuntu.com/ubuntu feisty-security universe deb http://security.ubuntu.com/ubuntu feisty-security multiverse deb-src http://security.ubuntu.com/ubuntu feisty-security multiverse
Finally, I decided to install OpenSSH, OpenNTPD and the typical Ubuntu desktop:
root@client.lan:# apt-get update root@client.lan:# apt-get install openssh-server openntpd ubuntu-desktop
Kerberizing Leopard’s Remote Login (built-in SSH) service
December 7th, 2007
0. Introduction
Enabling Kerberos/GSSAPI support in Leopard’s Remote Login (SSH) service is straightforward. As Leopard’s Remote Login is built using OpenSSH, most of what is described here applies perfectly to other flavors of UNIX.
Kerberos/GSSAPI authentication allows for Single Sign-On capabilities in OpenSSH in such a way that it makes very convenient to work with or manage multiple machines while only having to authenticate to the network every certain amount (Kerberos uses a form of cached credentials that expire within hours, typically). In this document I describe how to enable support for Kerberos/GSSAPI authentication to Leopard’s implementation of OpenSSH. Leopard includes many of the tools from the MIT’s implementation of Kerberos, so this makes the whole process even easier.
The process consists on:
- Creating the Kerberos principal
- Creating
/etc/krb5.conf - Adding the Kerberos principal to the keytab file
- Reconfiguring OpenSSH to support Kerberos authentication
- Restarting OpenSSH
- Enabling Kerberos authentication on OpenSSH clients
- Testing
1. Creating the Kerberos principal
When a Kerberized SSH client tries to authenticate to a Kerberized OpenSSH service, the client will request a Kerberos service ticket for that specific OpenSSH service. The Kerberos service ticket contains, among other things, a shared secret (for example, a 3DES private key) that has to be distributed to both the client and the server, called K<>client,server. The KDC will generate such shared secret and will encrypt it twice. Once, with a shared secret between the KDC and the client principal, called KKDC,client, and another one with a shared secret between the KDC and the server principal, KKDC,server.
In this step, we will create the Kerberos principal that represents the OpenSSH server and consequently its shared secret, KKDC,server. Later on, we will upload this shared secret to the OpenSSH server and install it into the Keytab file, so that the OpenSSH server can access it and use it to decrypt the user’s service ticket in order to retrieve the shared secret, Kclient,server.
$ kadmin Authenticating as principal alice/admin@LAN with password. Password for alice/admin@LAN: kadmin: addprinc -randkey host/ssh.lan WARNING: no policy specified for host/ssh.lan@LAN; defaulting to no policy Principal "host/ssh.lan@LAN" created.
NOTE: If you are logged in as root into the KDC, you can use kadmin.local instead of kadmin if you prefer, as the former doesn’t require any authentication (being able to run it as root is enough to prove yourself authorized).
The Kerberos principal associated with the OpenSSH has been created, the shared secrets generated, stored and protected by the stash key. We can check the status of the principal using the following command:
kadmin: getprinc host/ssh.lan Principal: host/ssh.lan@LAN Expiration date: [never] Last password change: Thu Dec 06 19:42:50 CET 2007 Password expiration date: [none] Maximum ticket life: 0 days 10:00:00 Maximum renewable life: 7 days 00:00:00 Last modified: Thu Dec 06 19:42:50 CET 2007 (root/admin@LAN) Last successful authentication: [never] Last failed authentication: [never] Failed password attempts: 0 Number of keys: 2 Key: vno 5, Triple DES cbc mode with HMAC/sha1, no salt Key: vno 5, DES cbc mode with CRC-32, no salt Attributes: Policy: [none] kadmin: exit
Now that the principal has been created, we can move onto the next step.
2. Creating /etc/krb5.conf
One of the requirements for Kerberizing OpenSSH server is configuring the Kerberos libraries via /etc/krb5.conf. These libraries are used by the OpenSSH server, kpasswd, the OpenSSH client and tools like kadmin or kinit.
This configuration file will also allow us to connect to the Kerberos administration server remotely by using kadmin, in order to download the OpenSSH shared secret. In my case, /etc/krb5.conf looks like this:
[libdefaults]
default_realm = LAN
dns_fallback = "no"
default_tgs_enctypes = des-cbc-crc
default_tkt_enctypes = des-cbc-crc
[realms]
LAN = {
kdc = kdc.lan:88
admin_server = kdc.lan:749
}
[domain_realm]
.lan = LAN
[login]
krb4_convert = true
krb4_get_tickets = false
3. Adding the Kerberos principal to the keytab file
root@ssh.lan:# kadmin Authenticating as principal root/admin@LAN with password. Password for root/admin@LAN: kadmin: ktadd -k /etc/krb5.keytab host/ssh.lan@LAN Entry for principal host/ssh.lan@LAN with kvno 7, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. Entry for principal host/ssh.lan@LAN with kvno 7, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
This will add the OpenSSH shared secret to the Kerberos /etc/krb5.keytab file. We need to add the shared secrets to this file properly, as Leopard has a built-in KDC service and machine-specific shared secrets in this file that are used, among others, by the Screen Sharing service, the AFP service and the SAMBA service.
To check that OpenSSH shared secret was added to the keytab file, we use ktutil to read and display the contents of the Keytab file:
root@ssh.lan:# ktutil ktutil: rkt /etc/krb5.keytab ktutil: list slot KVNO Principal ---- ---- --------------------------------------------------------------------- 1 3 afpserver/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 2 3 afpserver/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 3 3 afpserver/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 4 3 cifs/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 5 3 cifs/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 6 3 cifs/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 7 3 vnc/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 8 3 vnc/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 9 3 vnc/LKDC:SHA1.E803840ADEAA3E91BF895E969F56FEDFF321BDD5@LKDC... 10 7 host/ssh.lan@LAN 11 7 host/ssh.lan@LAN
Once installed in the keytab file, OpenSSH will be able to access the shared secret, KKDC,server used to decrypt Kerberos service tickets shown by clients.
It is also necessary to change OpenSSH’s server configuration file, /etc/sshd_config to explicitly enable Kerberos/GSSAPI authentication. To achieve this, modify /etc/sshd_config file and add the following lines (by default they are either uncommented or disabled):
# GSSAPI options GSSAPIAuthentication yes GSSAPICleanupCredentials yes
5. Restarting OpenSSH
To last step on the server side requires that we restart the OpenSSH service so the previous changes are taken into effect:
root@ssh.lan:# launchctl aunchd% stop com.openssh.sshd launchd% start com.openssh.sshd
6. Enabling Kerberos authentication on OpenSSH clients
Some OpenSSH clients have Kerberos authentication disabled by default. The easiest way to enable Kerberos/GSSAPI authentication in clients is by adding the following directives to $HOME/.ssh/config:
Host *
Protocol 2
GSSAPIAuthentication yes
GSSAPIKeyExchange yes
GSSAPIDelegateCredentials no
GSSAPITrustDns no
This will enable Kerberos/GSSAPI authentication on the client side. The fact that Kerberos/GSSAPI authentication succeeds depends on several factors, like the user having a valid Kerberos TGT, the remote server having Kerberos/GSSAPI authentication enabled, and that the clocks on both the client and server machines are synced (most implementations refuse Kerberos/GSSAPI authentication if the clocks differ more than 5 minutes).
7. Testing
First, we need to get a TGT from the Kerberos KDC:
$ kinit Please enter the password for falfaro@LAN:
To check that it worked, we list the tickets available to us, and make sure we see a ticket valid for service krbtgt/*:
$ klist Kerberos 5 ticket cache: 'API:Initial default ccache' Default principal: falfaro@LAN Valid Starting Expires Service Principal 12/06/07 18:06:26 12/07/07 04:06:26 krbtgt/LAN@LAN renew until 12/13/07 18:06:26
Now, let’s try to log in into the OpenSSH server. Kerberos/GSSAPI authentication should be negotiated, as shown by the debugging information:
core2:~ falfaro$ ssh -v solana OpenSSH_4.5p1, OpenSSL 0.9.7l 28 Sep 2006 debug1: Reading configuration data /Users/falfaro/.ssh/config debug1: Applying options for * debug1: Reading configuration data /etc/ssh_config debug1: Connecting to solana [10.42.242.12] port 22. debug1: Connection established. debug1: identity file /Users/falfaro/.ssh/id_rsa type -1 debug1: identity file /Users/falfaro/.ssh/id_dsa type 2 debug1: Remote protocol version 2.0, remote software version OpenSSH_4.5 debug1: match: OpenSSH_4.5 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.5 debug1: Offering GSSAPI proposal: gss-gex-sha1-toWM5Slw5Ew8Mqkay+al2g==, gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==, gss-group14-sha1-toWM5Slw5Ew8Mqkay+al2g==, gss-gex-sha1-A/vxljAEU54gt9a48EiANQ==, gss-group1-sha1-A/vxljAEU54gt9a48EiANQ==, gss-group14-sha1-A/vxljAEU54gt9a48EiANQ==, gss-gex-sha1-bontcUwnM6aGfWCP21alxQ==, gss-group1-sha1-bontcUwnM6aGfWCP21alxQ==, gss-group14-sha1-bontcUwnM6aGfWCP21alxQ== debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-cbc hmac-md5 none debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host 'solana' is known and matches the RSA host key. debug1: Found key in /Users/falfaro/.ssh/known_hosts:8 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Authentication succeeded (gssapi-with-mic). debug1: channel 0: new [client-session] debug1: Entering interactive session. Last login: Thu Dec 6 20:29:59 2007 from foo.lan $ hostname ssh.lan $ exit logout Connection to solana closed.
If we run klist again, we will see that, in addition to the TGT we saw before, we will see the Kerberos service ticket for the OpenSSH server:
Kerberos 5 ticket cache: 'API:Initial default ccache' Default principal: alice@LAN Valid Starting Expires Service Principal 12/06/07 18:06:26 12/07/07 04:06:26 krbtgt/LAN@LAN renew until 12/13/07 18:06:26 12/06/07 18:07:27 12/07/07 04:06:26 host/ssh.lan@LAN renew until 12/13/07 18:06:26
8. Troubleshooting
The GSSAPI authentication mechanism is very picky about many things, and error codes and messages are usually very cryptic but not very helpful about what the cause of the error is.
I found that acommon source of authentication problems are entries in /etc/hosts confusing GSSAPi about the host name. For example, entries like this in /etc/hosts:
192.168.0.1 ssh ssh.lan
can cause SSH to think the host name is ssh instead of the expected FQDN — ssh.lan.The previous entry causes reverse name resolutions for IP 192.168.0.1 to return ssh as the host name, but not ssh.lan. Since the Kerberos keytab is host/ssh.lan but IP 192.168.0.1 is resolved to ssh this may cause authentication attempts to fail because no Kerberos keytab for host/ssh exists in /etc/krb5.keytab. The most common symptom is trying to SSH to localhost, getting a failed authentication and but ticket for the target host. Example:
$ kinit Password for alice@LAN: $ klist Ticket cache: FILE:/tmp/krb5cc_501 Default principal: alice@LAN Valid starting Expires Service principal 05/31/09 01:08:21 06/01/09 01:08:20 krbtgt/LAN@LAN
Then,
$ ssh ssh.lan alice@ssh.lan's password:
But no Kerberos ticket for ssh.lan was even requested:
$ klist Ticket cache: FILE:/tmp/krb5cc_501 Default principal: alice@LAN Valid starting Expires Service principal 05/31/09 01:08:21 06/01/09 01:08:20 krbtgt/LAN@LAN
One possible solution consists of removing the matching entry from /etc/hosts, provided that the existing DNS name server can properly resolve the IP address of the server to the correct FQDN. If this is not possible, another solution is making sure the FQDN name is listed before the non-FQDN name in the corresponding line in /etc/hosts.
8. References
Some additional and very useful references:
- Kerberos 5 setup for NFSv4.
Describes common pitfalls, like using a non-FQDN host name in
/etc/hostsor not using clock synchronization.