FreeBSD 9 booting from a 3 disks ZFS

       857 words, 5 minutes

I already managed to boot FreeBSD from ZFS ; a single disk ZFS pool. Now, let’s see how to boot FreeBSD-9-RELEASE from a ZFS pool made of 3 disks using RAIDZ configuration.

I used the FreeBSD-9.0-RELEASE-amd64-dvd1.iso CD-ROM image.

Introduction

In this example, I’m using a virtual machine running on ESXi v5, with 2 vCPU, 4GB of RAM and 3x 8GB virtual disk attached on a “LSI Logic Parallel” virtual adapter. The “Guest Operating System” is configured as “Other / FreeBSD (64-bit)”.

That said, boot the ISO and select the “Live CD” option when prompted. The CD/DVD is mounted read-only and I’m using bits in /tmp to alter the filesystem:

  # mkdir /tmp/boot
  # mount_unionfs /tmp/boot /boot

I do the stuff using the Bourne shell, as I know it better that C-shell, and verify that the VM does know about my virtual disks:

  # exec /bin/sh
  # sysctl kern.disks
  kern.disks: cd0 da2 da1 da0

Preparing the disks

ZFS requires a GPT configuration. Configure parts of those disks as swap space and the rest as a ZFS pool:

  # for DISK in 0 1 2; do gpart create -s gpt da${DISK}; done
  da0 created
  da1 created
  da2 created
  
  # for DISK in 0 1 2; do 
  > gpart add -s 128k -t freebsd-boot -l boot${DISK} da${DISK}
  > gpart add -s 2G -t freebsd-swap -l swap${DISK} da${DISK}
  > gpart add -t freebsd-zfs -l disk${DISK} da${DISK}
  > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da${DISK}
  > done
  da0p1 added
  da0p2 added
  da0p3 added
  bootcode written to da0
  da1p1 added
  da1p2 added
  da1p3 added
  bootcode written to da1
  da2p1 added
  da2p2 added
  da2p3 added
  bootcode written to da2

The swap spaces will be known as swap0, swap1, swap2.
The ZFS will be composed of disk0, disk1, disk2.

I can loose one disk of this set and still be able to boot from the two remaining ones.

Configuring ZFS

Load the ZFS related modules and create a RAIDZ with the 3 available disks:

  # kldload zfs.ko
  # zpool create -f -m none tank raidz \
    /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2 
  # zpool status tank
    pool: tank
   state: ONLINE
   scan: none requested
  config:
  
  	NAME           STATE     READ WRITE CKSUM
  	tank           ONLINE       0     0     0
  	  raidz1-0     ONLINE       0     0     0
  	    gpt/disk0  ONLINE       0     0     0
  	    gpt/disk1  ONLINE       0     0     0
  	    gpt/disk2  ONLINE       0     0     0
  
  errors: No known data errors

Configure general parameters and create the various datasets you want:

  # zfs set checksum=fletcher4 tank
  # zfs create -o mountpoint=/media -o compression=off \
    -o dedup=sha256,verify tank/freebsd9
  # zpool set bootfs=tank/freebsd9 tank
  # zfs create -o compression=gzip tank/freebsd9/usr
  # zfs create -o compression=gzip tank/freebsd9/var

I’m setting checksum=fletcher4 to ensure better integrity checking. I’m setting up deduplication from the root of my datasets and add compression to datasets where most data will be.

Note that you can’t, AFAIK, boot from a compressed dataset. This is why tank/freebsd9 isn’t compressed at all. Also note that installing the system under freebsd9 is used to ease ZFS sharing and snapshot managing. This way, I can have tank/freebsd8 and tank/freebsd10 on the same ZFS pool and boot from any of them.

Install FreeBSD

The installation is only a matter of untarring archives:

  # cd /usr/freebsd-dist
  # for FILE in base.txz doc.txz kernel.txz lib32.txz; do
  > tar --unlink -xpJf ${FILE} -C /media/
  > done

A few more steps are required to end the configuration:

  # cd /
  # chroot /media /bin/sh
  # cat >> /etc/rc.conf
  zfs_enable="YES"
  hostname="zfreebsd.tumfatig.net"
  # cat >> /boot/loader.conf
  zfs_load=YES
  vfs.root.mountfrom="zfs:tank/freebsd9"
  ^D
  # cat >> /etc/fstab
  /dev/gpt/swap0 none swap sw 0 0
  /dev/gpt/swap1 none swap sw 0 0
  /dev/gpt/swap2 none swap sw 0 0
  ^D
  # passwd root
  Changing local password for root
  New Password:
  Retype New Password:
  # tzsetup
  (...)
  # cd /etc/mail; make aliases
  /usr/sbin/sendmail -bi -OAliasFile=/etc/mail/aliases
  /etc/mail/aliases: 28 aliases, longest 10 bytes, 283 bytes total
  chmod 0640 /etc/mail/aliases.db
  # exit

Before exiting chroot, you may want to run sysinstall to “Configure” various aspect of the system ; like network configuration, daemon to be started, user management… If you created users to log in, you can install the user’s SSH key now.

Final settings

Update the ZFS pool cache and copy it to the FreeBSD instance:

  # cd /
  # zpool export tank && zpool import tank
  # cp -p /boot/zfs/zpool.cache /media/boot/zfs/

Unmount the filesystem and set the mounpoints:

  # zfs unmount -a
  # zfs set mountpoint=legacy tank/freebsd9
  # zfs set mountpoint=/usr tank/freebsd9/usr
  # zfs set mountpoint=/var tank/freebsd9/var

Cross your fingers and reboot.

Feedback

So far, the installation is working great.

The only strange thing is the long booting time. It takes about a minute or two to get to the boot loader. Not sure if this is related to FreeBSD or to the VMware configuration. Since the OS is booted, it seems to run at full speed.

Here’s a bit of number from ZFS:

  # zpool list    
  NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
  tank  17.9G   906M  17.0G     4%  1.01x  ONLINE  -
  # zfs list -r -o name,used,available,referenced,mountpoint,compressratio           
  NAME                USED  AVAIL  REFER  MOUNTPOINT  RATIO
  tank                610M  11.1G  40.0K  none        1.74x
  tank/freebsd9       596M  11.1G   345M  legacy      1.74x
  tank/freebsd9/usr   250M  11.1G   250M  /usr        2.85x
  tank/freebsd9/var  1.72M  11.1G  1.72M  /var        3.96x

Related posts: