FreeBSD bhyve hypervisor to run OpenBSD virtual machines
2342 words, 11 minutes
Because I’m not 100% satisfied with my OmniOS bhyve experiment for running OpenBSD virtual machines, I’m giving it a try on a stock FreeBSD 14. And as usual, I’ll write down how I did it in case you are interested too :)
What happens with Illumos?
I have network issues while running OpenBSD on the bhyve implementation of OmniOS. They happen a bit less on a ThinkPad A485 with Realtek 8168 than on a Partaker H2 with Intel I211 but they still do. The reproducible test is quite simple, from another machine on a LAN, send or get quite a lot of data to or from the virtual machine.
# dd if=/dev/urandom of=TEST bs=1m count=1024
# while true; do scp TEST openbsdhvm:/home/ ; sleep 1; done
Everything works like a charm (~460Mbps) for a moment. Then, when about 400GB have been transferred, the VM network stack starts to woe. Connectivity goes up and down, SSH connexion gets long delay… The VM is not useable anymore as a connected asset.
Yet, using the VM from the console is responsive as usual ; and the
Illumos host also doesn’t show signs of illness. The top command
reveals no load, interrupt or hungry process. Everything appears as if
the VM was doing nothing. I couldn’t find any information in the logs of
the guest or the host. After the VM is rebooted, everything starts
working again normally ; until is breaks again.
If you have ideas, don’t hesitate to ping me :)
FreeBSD installation
Read Chapter 2. Installing FreeBSD .
Grab an install media. I used FreeBSD-14.0-RELEASE-amd64-memstick.img. None of the ISO or IMG file worked while with my Ventoy USB key.
Install FreeBSD as usual. I used a ZFS root on a single disk and choose to enable all security features.
FreeBSD first-boot configuration
Connect to the server using SSH, deploy the public key for the unprivileged user and root. While I’m there, I like to setup a forward(5) file to get email informations properly.
$ mkdir ~/.ssh && cat > ~/.ssh/authorized_keys
(...)
^D
$ echo changeme@example > ~/.forward
$ su -
# mkdir ~/.ssh && cat > ~/.ssh/authorized_keys
(...)
^D
# echo "changeme" > ~/.forward
# vi /etc/ssh/sshd_config
(...)
PermitRootLogin prohibit-password
PasswordAuthentication no
# service sshd restart
Yes, I’m allowing SSH root access 😱. But this server is not exposed on the Internet. So if someone tries to gain a root access, I’ll have other things to take care of.
During installation, I use DHCP. To switch static IP, simply run a few commands:
# sysrc ifconfig_em0="ether 00:00:5E:00:53:76"
# sysrc ifconfig_em0="inet 192.0.2.76 netmask 255.255.255.0"
# sysrc ifconfig_em0_ipv6="inet6 2001:db8::76 prefixlen 32"
# sysrc defaultrouter="192.0.2.1"
# sysrc ipv6_defaultrouter="2001:db8::1"
# cat > /etc/resolv.conf
search home.arpa
nameserver 192.0.2.1
nameserver 2001:db8::1
# nohup sh -c 'service netif restart && service routing restart'
Enable and configure pf(4)
# service pf enable
# service pflog enable
# kldload pf
# sysrc kld_list+=pf
# vi /etc/pf.conf
ext_if="igb0"
block in
pass in on $ext_if proto tcp to port ssh
pass out modulate state
# pfctl -f /etc/pf.conf -n
# pfctl -e -f /etc/pf.conf
# service pflog start
Update FreeBSD
# freebsd-update fetch
# freebsd-update install
# reboot
bhyve hypervisor: using native tools
Have a look at the Virtualisation Handbook section , there are loads of information there.
Enable bhyve is as simple as loading the proper kernel module.
# kldload vmm
# sysrc kld_list+=vmm
Every virtual machines will have its own tap(4) device to access
network. Creating a bridge(4) and adding the tap interfaces as members
enables network connection between them. Keep the bridge as-is and all
VM are isolated. Add a physical interface to the bridge and the VMs will
be able to appear on the LAN.
For the purpose of this article, two VMs will be created and gain direct access to my LAN resources.
# ifconfig tap0 create
# ifconfig tap1 create
# sysctl -w net.link.tap.up_on_open=1
# ifconfig bridge0 create
# ifconfig bridge0 addm igb0 addm tap0 addm tap1
# ifconfig bridge0 up
The sysctl variable ensures the tunnel devices will be marked up when the control device is opened.
To persist this configuration, files have to be modified.
# echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
# sysrc cloned_interfaces="bridge0 tap0 tap1"
# sysrc ifconfig_bridge0="addm igb0 addm tap0 addm tap1"
Because I liked how OmniOS managed its VM, I’ll do the same kind of thing: have a dataset to host the virtual machines stuff and using Zvol as virtual disks - instead of using image files.
# zfs create -o mountpoint=/guests tank/guests
# zfs create tank/guests/iso
The server is now ready to launch virtual machines.
Run a FreeBSD guest
Grab the FreeBSD 14 installation image:
# cd /guests/iso
# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
# cd -
Create a virtual disk:
# zfs create -o volmode=dev -V 16G tank/guests/freebsd
Install FreeBSD using the example script:
# sh /usr/share/examples/bhyve/vmrun.sh \
-c 2 -m 1024m -t tap0 -d /dev/zvol/tank/guests/freebsd \
-i -I /guests/iso/FreeBSD-14.0-RELEASE-amd64-bootonly.iso \
freebsd
The console can be set to “xterm” if you wish too.
In this network configuration, the VM can benefit from the LAN DHCP server.
At the end of the installation, selecting “Shutdown” will turn the VM off, stop the script and get you back to the hypervisor prompt.
Run the FreeBSD virtual machine using the example script:
# sh /usr/share/examples/bhyve/vmrun.sh \
-c 2 -m 1024m -t tap0 -d /dev/zvol/tank/guests/freebsd \
freebsd
The script runs in the foreground and connects to the VM console.
To stop the VM from the console, simply use halt -p.
To start the VM again, run the script again.
Run an OpenBSD guest
To boot non-FreeBSD guests, you need some extras material. In this particular case, a UEFI (and/or legacy BIOS) firmware is needed.
# pkg info bhyve-firmware
OpenBSD 7.4 installation ISO doesn’t support UEFI boot. This will change for 7.5. As for now, I’ll be using the installation IMG. One could boot OpenBSD using legacy BIOS. But I got used to using UEFI so lets go this way.
Also, I don’t run OpenBSD guests with a graphical console. I’m using the serial console emulation.
Grab the OpenBSD 7.4 installation image:
# cd /guests/iso
# fetch https://cdn.openbsd.org/pub/OpenBSD/7.4/amd64/miniroot74.img
# cd -
Create a virtual disk:
# zfs create -o volmode=dev -V 16G tank/guests/openbsd
Install OpenBSD:
# bhyve -A -D -H -P -S -u -w -c 2 -m 4G \
-s 0,amd_hostbridge \
-s 3,virtio-blk,/dev/zvol/tank/guests/openbsd \
-s 4,ahci-hd,/guests/iso/miniroot74.img \
-s 10,virtio-net,tap1,mac=00:00:5E:00:53:99 \
-s 20,virtio-rnd \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
openbsd
(...)
probing: pc0 com0 com1 mem[640K 3048M 16M 3M 1024M]
disk: hd0 hd1*
>> OpenBSD/amd64 BOOTX64 3.65
boot> set tty com0
switching console to com0
>> OpenBSD/amd64 BOOTX64 3.65
boot>
cannot open hd0a:/etc/random.seed: No such file or directory
booting hd0a:/bsd: 3969732+1655808+3886664+0+708608
[109+444888+297417]=0xa76798
entry point at 0x1001000
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights
reserved.
Copyright (c) 1995-2023 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 7.4 (RAMDISK_CD) #1322: Tue Oct 10 09:07:38 MDT 2023
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
real mem = 4254203904 (4057MB)
avail mem = 4121255936 (3930MB)
random: good seed from bootblocks
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbfbcf000 (12 entries)
bios0: vendor BHYVE version "14.0" date 10/17/2021
bios0: FreeBSD BHYVE
(...)
Don’t forget to use set tty com0 while in the OpenBSD bootloader. Then
proceed to installation as usual. Use (G)PT as the formatting layout ;
even though the installer will complain a bit. When done, (H)alt the
system and quit the console.
Running the VM will then be as simple as:
# bhyve -A -D -H -P -S -u -w -c 2 -m 4G \
-s 0,amd_hostbridge \
-s 3,virtio-blk,/dev/zvol/tank/guests/openbsd \
-s 10,virtio-net,tap1,mac=00:00:5E:00:53:99 \
-s 20,virtio-rnd \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
openbsd
It can also be run with the example script:
# sh /usr/share/examples/bhyve/vmrun.sh \
-c 2 -m 4G -t tap1 -C stdio \
-d /dev/zvol/tank/guests/openbsd \
-E -f /usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
openbsd
bhyve hypervisor: using vm-bhyve
Running stock tools is my preferred way of doing things. But the need to use tmux or such to maintain the VM up and running doesn’t fit me well - too many vmd(8)/vmctl(8) and vmadm(1M)/zadm(1) habbits.
There are various tools available in the packages repo. After a quick doc review, I went for vm-bhyve .
Note: if you already configured bhyve using the previous section, you should probably revert the network configuration to avoid weird behaviours. Same goes with the VM dataset if you’ll use the same.
Enable bhyve is as simple as loading the proper kernel module.
# kldload vmm
# sysrc kld_list+=vmm
Following the “Quick-Start” directions, setting it up is fast.
# pkg install vm-bhyve
# sysrc vm_enable="YES"
# sysrc vm_dir="zfs:tank/guests"
# vm init
# cp /usr/local/share/examples/vm-bhyve/* /guests/.templates/
Once again, I’m going for the network configuration where all my VMs have access to the LAN.
# vm switch create homelan
# vm switch add homelan igb0
Using vm persists configuration in the configured dataset.
# cat /guests/.config/system.conf
switch_list="homelan"
type_homelan="standard"
ports_homelan="igb0"
Run a FreeBSD guest
Grab the FreeBSD 14 installation ISO:
# vm iso https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
Create the virtual machine:
# vm create -t freebsd-zvol -s 16G -m 1024m -c 2 freebsd
# sed -i -e 's/"public"/"homelan"/' /guests/freebsd/freebsd.conf
Install FreeBSD:
# vm install -f freebsd FreeBSD-14.0-RELEASE-amd64-bootonly.iso
When installation is finished, (S)hutdown turns the VM off and the vm
script gets you back to the hypervisor prompt.
The vm will be run in background. To access to the console, a dedicated command is required.
# vm start freebsd
Starting freebsd
* found guest in /guests/freebsd
* booting...
# vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
freebsd default bhyveload 2 1024m - No Running (40698)
# vm console freebsd
The VM console can be quitted using ~~. or ~ + Ctrl-D
Run an OpenBSD guest
To boot non-FreeBSD guests, you need some extras material. In this particular case, a UEFI (and/or legacy BIOS) firmware is needed. In case you have not done it previsously, install the firmware.
# pkg info bhyve-firmware
OpenBSD 7.4 installation ISO doesn’t support UEFI boot. This will change for 7.5. As for now, I’ll be using the installation IMG. One could boot OpenBSD using legacy BIOS. But I got used to using UEFI so lets go this way.
Also, I don’t run OpenBSD guests with a graphical console. I’m using the serial console emulation.
Grab the OpenBSD 7.4 installation image:
# vm iso https://cdn.openbsd.org/pub/OpenBSD/7.4/amd64/miniroot74.img
The original OpenBSD template uses grub and image file. The grub configuration is not recommended nowadays. And I want to use ZFS volumes. So create a custom OpenBSD template:
# vi /guests/.templates/openbsd.conf
loader="uefi"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="homelan"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk1_type="ahci-hd"
disk1_name="../.iso/miniroot74.img"
bhyve_options="-A -D -H -P -S -u -w"
virt_random="yes"
Create the virtual machine:
# vm create -t openbsd -s 16G -m 1024m -c 2 openbsd
Install OpenBSD, telling the boot loader to use the console:
# vm start -f openbsd
probing: pc0 com0 com1 mem[640K 1000M 16M 3M]
disk: hd0 hd1*
>> OpenBSD/amd64 BOOTX64 3.65
boot> set tty com0
switching console to com0
>> OpenBSD/amd64 BOOTX64 3.65
boot> <ENTER>
(...)
Don’t forget to use set tty com0 while in the OpenBSD bootloader. Then
proceed to installation as usual. Use (G)PT as the formating layout ;
eventhough the installer will complain a bit. When done, (H)alt the
system and quit the console.
Remove the installation media definition and start the VM using the
simple vm command:
# vm configure openbsd
:g/^disk1/d
:wq
# vm start openbsd
# vm console openbsd
The VM console can be quitted using ~~. or ~ + Ctrl-D.
Create and deploy OpenBSD images
Creating virtual machine using the same base can be tiresome. And this
task can be automated using several manners. The same way I did on
OmniOS
,
I decided to use images using vm-bhyve.
You can create an OpenBSD instance using the previous steps. I like to modified the installation by configuring a few standard sthings (like SSH public keys, sshd and smtpd configuration) and have the new VM delete all traces of previous installation and upgrade to the latest syspatches. This is not mandatory though.
# vm create -t openbsd -s 16G -m 1024m -c 2 openbsd74
# vm start -f openbsd74
(...)
boot> set tty com0
(...)
Exit to (S)hell, (H)alt or (R)eboot? [reboot] s
To boot the new system, enter 'reboot' at the command prompt.
openbsd74# chroot /mnt /bin/ksh
# echo "ssh-ed25519 (...)" > /root/.ssh/authorized_keys
# TERM=vt220 vi /etc/ssh/sshd_config
# echo change_me@example > /root/.forward
# TERM=vt220 vi /etc/mail/smtpd.conf
# echo "(...)" > /etc/mail/secrets
# chown root:_smtpd /etc/mail/secrets
# chmod 0640 /etc/mail/secrets
# cp /etc/examples/doas.conf /etc/
# TERM=vt220 vi /etc/doas.conf
# cat >> /etc/rc.firsttime
echo "************************************************************************"
echo "This system was build from a template."
echo -n "System hostname? (short form, e.g. 'foo') "
read _hostname
/usr/bin/sed -E -i "s/openbsd74/$_hostname/g" /etc/myname
/bin/rm /etc/ssh/ssh_host*
echo "Applying syspatches..."
/usr/sbin/syspatch
echo "Updating packages..."
/usr/sbin/pkg_add -u
echo "Rebooting in 5 seconds..."
/bin/sleep 5
/sbin/shutdown -r now
^D
# exit
# halt -p
syncing disks... done
The operating system has halted.
Please press any key to reboot.
Remove the installer reference and create the OpenBSD template image:
# vm configure openbsd74
:g/^disk1/d
:wq
# vm image create -d "OpenBSD 7.4/amd64" openbsd74
Creating a compressed image, this may take some time...
Image of openbsd74 created with UUID 60817e2e-b268-11ee-9aeb-009027e529f7
Deploy a new VM from an image. Sizing and/or network and/or disk
configuration can be modified using the configure action.
# vm image list
UUID NAME CREATED
DESCRIPTION
60817e2e-b268-11ee-9aeb-009027e529f7 openbsd74 Sun Jan 14 00:06:26 CET
2024 OpenBSD 7.4/amd64
# vm image provision 60817e2e-b268-11ee-9aeb-009027e529f7 puffy
Unpacking guest image, this may take some time...
# vm configure puffy
# vm start puffy
# vm console puffy
The template source can be either deleted or kept. Depending on what you prefer.
# vm destroy openbsd74
Start VM automatically
vm-bhyve uses the “startall” command at boot time. Definition on which
VM to start and how many time to wait before starting the next one is
done via environment variables.
# sysrc vm_list+="puffy"
# sysrc vm_dealy="5"
If you wonder, the OpenBSD dmesg output is available here .
And that’s all for now!