LVM snapshots and non-destructive Linux upgrades
May 16th, 2009
This post roughly describes what I do when I want to non-destructively upgrade my Linux system. By non-destructive means a procedure that allows me to upgrade but also to rollback if something goes wrong. As an example, I wanted to upgrade my Ubuntu system from Jaunty to Karmic. Since Karmic is now Alpha 1, the chances of the upgrade going bad or eating my data were high. Here is where LVM and LVM snapshots come into scene.
Basically, the idea consists of taking a snapshot of the root filesystem using an LVM snapshot, then reboot the system to use the filesystem from the LVM snapshot as the root filesystem, perform an in-place upgrade of Jaunty to Karmic and see what happens. Since the upgrade takes place on a system running with the root filesystem mounted from the LVM snapshot, the original root volume is kept intact. Hence, if something goes wrong I can always reboot in order to use the original root filesystem and the system would behave as if no modifications were done at all during the upgrade.
The LVM snapshot volume should be big enough to, in the worst case, store a completely new Linux installation. The average Ubuntu installation requires less than 8GB of disk, so the LVM snapshot should be about that size plus some slack required to download the packages. In my case, and since I have free enough disk space, I chose 16GB just to be on the safe side.
Resize the root filesystem and root volume
This step is only required if there is no space in the volume group to accommodate for the snapshot volume. In my case, the volume group is full so I need to shrink the root filesystem and the root volume. resize2fs does not currently allow one to shrink a filesystem online, so I booted from the Jaunty LiveCD and entered rescue mode. From there:
# e2fsck -f /dev/root/root # resize2fs /dev/root/root 80G # lvresize -L 80G /dev/root/root
Create the LVM snapshot
To create an 16GB LVM snapshot volume of my root volume:
# lvcreate -s -n karmic-root -L 16G /dev/root/root
Prepare the boot environment
Mount the filesystem from the LVM snapshot volume and modify /etc/fstab to replace the device name where the original root filesystem is with the device name where the snapshotted rool filesystem lives:
# mount /dev/root/karmic-root /mnt # vi /mnt/etc/fstab
In my case, the line for the new root filesystem looks like:
/dev/root/karmic-root / ext4 defaults 0 1
Reboot into the snapshot
Trigger the grub menu and modify the kernel entry that corresponds to the Ubuntu system. The idea is to use the device name for the LVM snapshot. This is how the new kernel in the menu looks like:
kernel /vmlinuz-2.6.28-11-generic root=/dev/root/karmic-root ro
Then press b to boot the system. The system should boot normally, but instead of using the original root filesystem it should be using the filesystem from the LVM snapshot:
$ grep /dev/mapper /proc/mounts /dev/mapper/root-karmic--root / ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
In-place upgrade
I won’t describe how to do an in-place upgrade of Ubuntu. There are many resources out there that describe how to do that. The point here is that the upgrade will modify the snapshot while the original root filesystem is kept intact.
Destroy the snapshot
If something goes wrong with the update, and it usually goes wrong when upgrading to an Alpha version, to bring the system back to a usable state is just a matter of rebooting the system and using the right entry listed in grub. In-place upgrades of Ubuntu will typically add a new kernel to the list of entries in grub but won’t modify the existing ones.
After rebooting into the original system, the snapshot can be removed:
# lvremove /dev/root/karmic-root
If you don’t intend to experiment with upgrades, perhaps you want to resize the root LVM volume, then the root filesystem back to their original size.
July 23rd, 2009 at 1:14 PM
[...] Alfaro blogged about using LVM to do non-destructive Linux [...]
July 23rd, 2009 at 1:15 PM
[...] Alfaro blogged about using LVM to do non-destructive Linux [...]
July 23rd, 2009 at 1:15 PM
[...] Alfaro blogged about using LVM to do non-destructive Linux [...]
July 23rd, 2009 at 1:21 PM
[...] Alfaro blogged about using LVM to do non-destructive Linux [...]
July 23rd, 2009 at 2:40 PM
[...] Alfaro blogged about using LVM to do non-destructive Linux [...]
October 6th, 2009 at 5:33 PM
I had a problem with the /boot-partition after upgrade once so I have 2 boot-partitions and grub installed onto the partition for the mirror-system and a chainloader on the first so that Karmic can remove whatever it wants from /boot without ruining my Jaunty-installation.
October 29th, 2009 at 8:27 AM
Hi Felipe! I just wanted to thank you for your excellent post. I’m looking forward to try the technique that you’ve described. Keep up the good work!
Best regards,
Marcus
February 26th, 2010 at 2:48 AM
Let’s say I want to apply patches to a RHEL 5.4 server and if need be I can revert back in time to a good state of RHEL 5.4. What methodology would I do?