So for a long time I have wanted to replace my file server, because it's an old HP Microserver that is really slow. And at the same time I have had this beast of a desktop tower PC that I haven't used with a Xeon and 32GiB ECC memory that has just been turned off due to noise. The obvious solution is to re-purpose that install and move it to a location where noise doesn't matter, e.g. the closet (a.k.a the server room) where the file server lives.
To do this move I had a certain list of requirements:
With these requirements, the goal is that whichever disk should be able to die and the system should still just boot. Without any issues. Sure the raid or system like a raid will complain. But the system should still come up even if one disk die.
I followed these steps of partitioning grabbed from the NixOS on ZFS page but modified my steps.
# Defining some helper variables (these will be used in later code # blocks as well, so make sure to use the same terminal session or # redefine them later) DISK1=/dev/disk/by-id/ata-VENDOR-ID-OF-THE-FIRST-DRIVE DISK2=/dev/disk/by-id/ata-VENDOR-ID-OF-THE-SECOND-DRIVE # Creating partitions # This creates the ESP / Boot partition at the beginning of the drive # but numbered as the third partition: sgdisk -n3:1M:+512M -t3:EF00 $DISK1 # This create the storage partition numbered as the # first partition: sgdisk -n1:0:0 -t1:BF01 $DISK1 # Clone the partitions to the second drive: sfdisk --dump $DISK1 | sfdisk $DISK2
Creating a ZFS pool can be intimidating for the first times you do it, there
is a lot of options and flags. And you have both properties/features that use
the flag -o and then you have file system properties that use -O.
To me this is still confusing, but I managed to get it together after some fiddling around.
You can read more about these flags on these locations:
And many more places around the web…
# Some flags to consider # # Disable ZFS automatic mounting: # -O mountpoint=none # # Disable writing access time: # -O atime=off # # Use 4K sectors on the drive, otherwise you can get really # bad performance: # -o ashift=12 # # This is more or less required for certain things to # not break: # -O acltype=posixacl # # To improve performance of certain extended attributes: # -O xattr=sa # # To enable file system compression: # -O compression=lz4 # # To enable encryption: # -O encryption=aes-256-gcm -O keyformat=passphrase # Then we create the pool which is mirrored over both drives and # encrypted with ZFS native encryption (if you added those flags): zpool create <FLAGS> zroot mirror $DISK1-part1 $DISK2-part1
Here you can create which ever you want, this is not exactly how my set up mine due to my usage of tmpfs as root, but that's a topic for another post.
zfs create -o mountpoint=legacy zroot/root # For / zfs create -o mountpoint=legacy zroot/root/home # For /home zfs create -o mountpoint=legacy zroot/root/nix # For /nix
This will seem really obvious at this point, but what's not obvious is how to keep them in sync. In the past (with MBR) I would make an mdraid level 1 across the drives and have that as /boot and the system would boot fine from any drive. It's not that simple with UEFI.
Although, this guy have a good write up on how to achieve this same thing on Debian/Ubuntu. And that probably works on other distributions as well. https://outflux.net/blog/archives/2018/04/19/uefi-booting-and-raid1/
mkfs.vfat $DISK1-part3 mkfs.vfat $DISK2-part3
This takes a bit of work to do, many drives many steps, etc. If you have rebooted your live media because you need to rescue or try again in some way with the configuration you should start with importing the pool and loading the key to the encrypted disk.
This isn't needed if you're still up and running on the same session as where you created your pool.
zpool import zroot zfs load-key zroot
To mount the file systems you probably want to do something like this:
mount -t zfs zroot/root /mnt # Create directories to mount file systems on mkdir /mnt/{nix,home,boot,boot-fallback} # Mount the rest of the ZFS file systems mount -t zfs zroot/root/nix /mnt/nix mount -t zfs zroot/root/home /mnt/home # Mount both of the ESP's mount $DISK1-part3 /mnt/boot mount $DISK2-part3 /mnt/boot-fallback
Here we have the magic of the NixOS module systems kicking in that will make this entire setup awesome and easy to manage.
Just start by the regular nixos-generate-config --root /mnt.
Then you want to use grub as a boot loader and not systemd-boot.
I have the following configured to make my system support ZFS:
{
boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "<8 random numbers>";
}
Then I have the following to boot with grub and keep my two grubs in sync:
{
# You can probably use his one on true, but my UEFI is buggy.
boot.loader.efi.canTouchEfiVariables = false;
# This is the regular setup for grub on UEFI which manages /boot
# automatically.
boot.loader.grub.enable = true;
boot.loader.grub.efiSupport = true;
boot.loader.grub.device = "nodev";
# This will mirror all UEFI files, kernels, grub menus and things
# needed to boot to the other drive.
boot.loader.grub.mirroredBoots = [
{ devices = [ "/dev/disk/by-uuid/<FS-ID>" ];
path = "/boot-fallback"; }
];
}
NixOS makes it really easy to set up encrypted ZFS that is reliable when it comes to updates and usage.
And with the module system NixOS makes it easy to keep the two ESP partitions in sync.
I have tested the entire process to make sure it works on my hardware by:
So now that computer have moved to the closet (a.k.a. the server room) and I've configured it so I can unlock the drive over SSH on boot following: https://nixos.wiki/wiki/NixOS_on_ZFS#Unlock_encrypted_zfs_via_ssh_on_boot
Date: 2019-08-04 Sun 10:20