Discovering SmartOS as a Home Hypervisor
1811 words, 9 minutes
Doing some tidying in my network, computers and VPS collection, I went looking for a piece of software that could be used :
- at home, as a file server, a backup server and a VM lab.
- at colloc, as an hypervisor.
I (re)discovered SmartOS and decided to look at how it would fit or not. I keep my notes here.
Preparation
Read about creating a SmartOS bootable USB key .
I simply copied the smartos-20231116T064739Z file to my Ventoy flash drive. Then inserted the installation media and booted from it.
Installation
The SmartOS Setup wizard starts automatically.
My specifics
There are a few non-standard things related to my installation. So I first drop to a shell, configure my stuff and then switch back to the installation process.
Press [enter] to continue^C
Bringing up a shell. When you are done in the shell hit ^D to return to the
system configuration tool.
[root@ /]# █
The default keyboard layout is US. I want to use the FR layout.
# kbd -s French
USB keyboard
My machine has 6 NICs and only one is active during installation. For some reasons, the interface numbering in the OS doesn’t match the one labelled on the machine. And it changes everytime I’m testing a new installation. The following commands helps me guess which one to use during installation.
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
igb1 Ethernet down 0 half igb1
igb2 Ethernet down 0 half igb2
igb3 Ethernet down 0 half igb3
igb0 Ethernet down 0 half igb0
igb4 Ethernet down 0 half igb4
igb5 Ethernet up 1000 full igb5
Everything is now ready for the installer process.
# exit
Resuming the system configuration tool.
<Enter>
During my testings, I had to reset the ZFS disk configuration. It is not as simple as using dd on the first bits of disks. There seem to be ZFS information in the end of disks.
The simple method is to get a shell on SmartOS and issue the followind command:
# sdc-factoryreset
WARNING: This machine will reboot and destroy its ZFS pools after rebooting.
Do you want to proceed with the factory reset? (y/n) y<Enter>
This will destroy ALL DATA on the system, including potential customer data.
Are you sure? (y/n) y<Enter>
Rebooting in 5 seconds ... now!
SmartOS Setup
The installation process is as simple as “answer my questions”. Here are the answers I gave:
- Enter the number of the NIC for the ‘admin’ interface: 6
- (admin) IP address (or dhcp): dhcp
- Enter the default gateway IP [none]: 192.0.2.1
- Enter the Primary DNS server IP [8.8.8.8]: <Enter>
- Enter the Secondary DNS server IP [8.8.4.4]: <Enter>
- Default DNS search domain: tumfatig.example
- Enter an NTP server IP address or hostname [0.smartos.pool.ntp.org]: fr.pool.ntp.org
- Selected zpool layout [yes]: manual
This particular machine has two disks. I want to use the mSATA Flash disk for the system and the SATA SSD for the data. SmartOS tries to figure out the best configuration for you but it doesn’t match what I have in mind.
Launching a shell.
Please manually create/import a zpool named "zones".
(...)
# diskinfo
TYPE DISK VID PID SIZE RMV SSD
SCSI c1t0d0 SanDisk Extreme Pro 119.25 GiB yes no
SATA c2t0d0 ATA BT58SSD10M 238.47 GiB no yes
SATA c2t1d0 Samsung SSD 870 QVO 8TB 7452.04 GiB no yes
# zpool create -B -f zones c2t0d0
# zpool create -f areas c2t1d0
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
areas 7.27T 102K 7.27T - - 0% 0% 1.00x ONLINE -
zones 238G 100K 238G - - 0% 0% 1.00x ONLINE -
# exit
- Specify a (configured) zpool from which to boot [none]: zones
- Source for SmartOS to install [media]: latest
- Install pkgsrc? [y]: <Enter>
- Enter root password: (…)
- Confirm password: (…)
- Enter system hostname [press enter for none]: smartos
Review the configuration and commit installation.
- Is this correct, proceed with installation? [y]: <Enter>
- System setup has completed. Press enter to reboot. <Enter>
When the screen turns black, remove the installer media and watch SmartOS booting.
Basic configuration
Log in on the console or via SSH.
Localized keyboard
Have the console use the French keyboard layout.
# kbd -s French
# echo "default_keymap=france" >> /usbkey/config
Secured SSH connections
Register the SSH public key(s).
# mkdir -p /usbkey/config.inc
# echo ssh-ed25519 (...) > /usbkey/config.inc/authorized_keys
# echo "root_authorized_keys_file=authorized_keys" >>/usbkey/config
Allow connection using SSH keys only.
# sed -i.orig \
-e 's/PermitRootLogin yes/PermitRootLogin prohibit-password/' \
-e 's/PasswordAuthentication yes/PasswordAuthentication no/' \
/usbkey/ssh/sshd_config
# svcadm restart ssh
The Global Zone is non-persistent. SmartOS populates standard configuration files (/etc …) during boot using the usbkey directory content.
A reboot will make sure the system configured as expected.
Sending system emails
Configure email relay and credentials that can be used by the system to send electronic mails.
# vi /usbkey/config
(...)
mail_smarthost=myrelay.example
mail_sender_domain=tumfatig.example
mail_auth_user=<change_me>
mail_auth_pass=<change_me>
mail_adminaddr=example@tumfatig.example
Reboot to apply changes.
Change root password
When using US layout, I often use temporary and easy (were keys are identical in US and FR layouts) password. When I get an FR layout on the console or SSH access, I modify the root password.
The official documentation about changing the root password in the Global Zone makes it really easy.
# umount /etc/shadow
# cp /usbkey/shadow /etc/shadow
# passwd root
# cp /etc/shadow /usbkey/shadow
ZFS compression and deduplication
Enable ZFS compression on the data storage pool. I choose LZ4 because zfs(8) says:
The lz4 compression algorithm is a high-performance replacement for the lzjb algorithm. It features significantly faster compression and decompression, as well as a moderately higher compression ratio than lzjb, but can only be used on pools with the lz4_compress feature set to enabled. See zpool-features(7) for details on ZFS feature flags and the lz4_compress feature.
# zpool get feature@lz4_compress areas
NAME PROPERTY VALUE SOURCE
areas feature@lz4_compress active local
# zfs set compression=lz4 areas
# zfs get compression areas
NAME PROPERTY VALUE SOURCE
areas compression lz4 local
# zfs get compression
NAME PROPERTY VALUE SOURCE
areas compression lz4 local
zones compression off default
zones/archive compression lzjb local
zones/boot compression off default
zones/config compression off default
zones/cores compression gzip local
zones/cores/global compression gzip inherited from zones/cores
zones/dump compression off local
zones/opt compression off default
zones/swap compression off default
zones/usbkey compression off default
zones/var compression off default
Deduplication is a great feature, when you read about it. I did test the feature years ago and was not convinced with its benefits. Another recent reading indicates than it may be an overlooked feature ; or at least highly depending on your usage and data. I will create a few datasets, enable dedup on them and will have the opportunity to remove it depending on what happens in time.
# for d in backup nocloud vms; do \
zfs create areas/$d ; \
zfs set dedup=edonr,verify areas/$d ; \
done
# zfs get dedup,compression
NAME PROPERTY VALUE SOURCE
areas dedup off default
areas compression lz4 local
areas/backup dedup edonr,verify local
areas/backup compression lz4 inherited from areas
areas/nocloud dedup edonr,verify local
areas/nocloud compression lz4 inherited from areas
(...)
zones dedup off default
zones compression off default
(...)
Update SmartOS
Switching from one release to another seem quite easy. During my testing, I only did upgrades. And they were straightforward.
# piadm avail
20231005T000607Z
# piadm install 20231005T000607Z zones
# piadm list
PI STAMP BOOTABLE FILESYSTEM BOOT IMAGE NOW NEXT
20230921T034751Z zones/boot next yes yes
20231005T000607Z zones/boot available no no
# piadm activate 20231005T000607Z zones
# piadm list
PI STAMP BOOTABLE FILESYSTEM BOOT IMAGE NOW NEXT
20230921T034751Z zones/boot available yes no
20231005T000607Z zones/boot next no yes
# reboot
Extra packages
SmartOS uses pkgsrc as a third-party package management. I’d rather not install much on the Global Zone. But a few basic tools might be handy.
# pkgin update
reading local summary...
processing local summary...
processing remote summary (https://pkgsrc.smartos.org/packages/SmartOS/trunk/tools/All)...
pkg_summary.xz
# pkgin in tmux
(...)
installing tmux-3.3anb2...
(...)
marking tmux-3.3anb2 as non auto-removable
SMB shares and Time Machine
One of the purpose of this NAS system is to serve as a backup system, for UNIX, Linux and macOS computers. Another is to provide access to files from various workstations ; the more common technology is Samba.
I don’t want to do it from the Global Zone. Let’s use a Brand Zone to do this.
Get a Zone image
Check what is available:
# imgadm avail name=base-64-lts
UUID NAME VERSION OS TYPE PUB
c02a2044-c1bd-11e4-bd8c-dfc1db8b0182 base-64-lts 14.4.0 smartos zone-dataset 2015-03-03
24648664-e50c-11e4-be23-0349d0a5f3cf base-64-lts 14.4.1 smartos zone-dataset 2015-04-17
b67492c2-055c-11e5-85d8-8b039ac981ec base-64-lts 14.4.2 smartos zone-dataset 2015-05-28
96bcddda-beb7-11e5-af20-a3fb54c8ae29 base-64-lts 15.4.0 smartos zone-dataset 2016-01-19
088b97b0-e1a1-11e5-b895-9baa2086eb33 base-64-lts 15.4.1 smartos zone-dataset 2016-03-04
1f32508c-e6e9-11e6-bc05-8fea9e979940 base-64-lts 16.4.1 smartos zone-dataset 2017-01-30
390639d4-f146-11e7-9280-37ae5c6d53d4 base-64-lts 17.4.0 smartos zone-dataset 2018-01-04
c193a558-1d63-11e9-97cf-97bb3ee5c14f base-64-lts 18.4.0 smartos zone-dataset 2019-01-21
e75c9d82-3156-11ea-9220-c7a6bb9f41b6 base-64-lts 19.4.0 smartos zone-dataset 2020-01-07
1d05e788-5409-11eb-b12f-037bd7fee4ee base-64-lts 20.4.0 smartos zone-dataset 2021-01-11
c8715b60-7e98-11ec-82d1-03d16599f529 base-64-lts 21.4.0 smartos zone-dataset 2022-01-26
85d0f826-0131-11ed-973d-2bfeef68011c base-64-lts 21.4.1 smartos zone-dataset 2022-07-11
93bdf06a-01ef-11ed-81ff-bf0efad842c7 base-64-lts 20.4.1 smartos zone-dataset 2022-07-12
e44ed3e0-910b-11ed-a5d4-00151714048c base-64-lts 22.4.0 smartos zone-dataset 2023-01-10
Import the latest available release:
# imgadm import e44ed3e0-910b-11ed-a5d4-00151714048c
Importing e44ed3e0-910b-11ed-a5d4-00151714048c (base-64-lts@22.4.0) from "https://images.smartos.org"
Gather image e44ed3e0-910b-11ed-a5d4-00151714048c ancestry
Must download and install 1 image (250.9 MiB)
Downloaded image e44ed3e0-910b-11ed-a5d4-00151714048c (250.9 MiB)
Imported image e44ed3e0-910b-11ed-a5d4-00151714048c (base-64-lts@22.4.0)
Configure the Zone image
The configuration is done using a JSON file.
# vim /areas/manifests/smb.json
{
"zpool": "areas",
"brand": "joyent",
"image_uuid": "e44ed3e0-910b-11ed-a5d4-00151714048c",
"alias": "smb",
"hostname": "smb",
"ram": 512,
"quota": 6144,
"resolvers": [ "8.8.8.8", "8.8.4.4" ],
"nics": [
{ "nic_tag": "admin", "ips": [ "dhcp", "addrconf" ] }
]
}
Manage the Zone
Create the Zone:
# vmadm create -f /areas/manifests/smb.json
Check the Zone state:
# vmadm list
UUID TYPE RAM STATE ALIAS
13dc0e2e-a59d-42d2-941a-eb9f3ce42dc5 OS 1024 running smb
Delete the Zone:
# vmadm delete 13dc0e2e-a59d-42d2-941a-eb9f3ce42dc5
Successfully deleted VM 13dc0e2e-a59d-42d2-941a-eb9f3ce42dc5
To be continued
From there, I faced a few features that do fit me well.
It is recommended to create a dedicated Zone for an SMB service . While this makes sense to not turn the Global Zone into a mess, it also makes you dedicate a dataset to a SMB Zone. Mr Jeffrey A Jenner explains how it can be made more admin-friendly . But I don’t like the idea of tweaking the storage and risking loosing the data if I delete the Zone.
The other thing I don’t like is how SmartOS and/or vadm expect the
images and the zones to be deployed in the root of a zpool. Mixing the
virtual machines, the data in a flat layout isn’t something I like to do.
I won’t dig further with SmartOS at the moment. I like the Illumos world and will continue looking at such distributions. Maybe I’ll get back to SmartOS if other options don’t suit me well either.
For the record, here’re a few more links that I used to test things but won’t write about, yet.