Xen with Alpine Dom0 on LVM and LUKS

I've set up a few personal servers in the last couple months, and landed on a "standard" setup I'm comfortable with. It's a bit unconventional and takes a few manual steps to install but it's quite nice to use afterwards.

Firstly, boot into the Alpine Xen ISO, run setup-alpine with the target system's information, and choose "none" for all the disk prompts at the end.

Set up the disk manually (obviously, this will delete any data there currently is on the disk):

drive=/dev/sda

# create partitions
parted -s $drive -- mklabel gpt
parted -s $drive -- mkpart ESP fat32 1MB 512MB
parted -s $drive -- mkpart root ext4 512MB 100%
parted -s $drive -- set 1 esp on
wipefs -a ${drive}1
wipefs -a ${drive}2

# at this point, if you have a second drive, you can set it up as above and then RAID1 the data partitions
# I don't know of an easy way to do RAID on the boot partitions (let me know if there is one!)
# mdadm --create --verbose --level=1 --metadata=1.2 --raid-devices=2 /dev/md0 $drive1 $drive2
# then use /dev/md0 instead of ${drive}2

# init LUKS/LVM
cryptsetup luksFormat ${drive}2
cryptsetup luksOpen ${drive}2 lvmroot
pvcreate /dev/mapper/lvmroot
vgcreate lvmroot /dev/mapper/lvmroot
lvcreate -L50G -n dom0 lvmroot

# mount
mkfs.fat -F 32 -n boot ${drive}1
mkfs.ext4 /dev/mapper/dom0
mount /dev/mapper/dom0 /mnt
mkdir /mnt/boot
mount /dev/${drive}1 /mnt/boot

Then run setup-disk -m sys /mnt/ to install to disk. The system should reboot into the installed Xen/Alpine.

Install the xen-qemu package: apk add xen-qemu

Add a network bridge to /etc/network/interfaces, moving the IP assignment on the existing ethN entry to it:

auto eth0
iface eth0 inet static

auto br0
iface br0 inet static
	address 192.168.12.202/24
	gateway 192.168.12.253
	bridge_ports eth0
    bridge_stp 0

By default, Dom0 (administrative domain, or the "host console") is allocated all RAM so there isn't any available for VMs. If you try to start a DomU now you'll get a "failed to free memory for the domain" error.

Edit /etc/grub.d/20_linux_xen and add dom0_mem=3G,max=3G to the Xen bootloader line before placeholder. It should look something like this:

        ${xen_loader}   ${rel_xen_dirname}/${xen_basename} dom0_mem=3G,max=3G placeholder ${xen_args} \${xen_rm_opts}

Run grub-mkconfig -o /boot/grub/grub.cfg and reboot, and you should be good to go.

This is probably not the best way to fix this, but it works for me. If you know why and/or have a better solution, send me a message and I'll update this post.

Here's a typical DomU config:

kernel = "/usr/lib/xen/boot/hvmloader"
type='hvm'
memory = 2048
cpus=2
name = "test"
vif = ['bridge=br0']
disk = ['phy:/dev/lvmroot/test,hda,w', 'file:/root/iso/alpine-virt-3.22.2-x86_64.iso,hdc:cdrom,r']
acpi = 1
device_model_version = 'qemu-xen'
boot="d"
sdl=0
serial='pty'
vnc=1
vnclisten="127.0.0.1"
vncpasswd=""

The disk can be created with lvcreate -L<size>G -n <name> lvmroot. Then, start the VM with xl create path/to/filename.cfg. xl console will give you an inline serial terminal, but it's also possible to connect with a VNC client).

When you're done installing, remove the ISO from the disk section of the config, and xl destroy and xl create the VM.

If there are any errors starting the VM, check /var/log/xen/qemu-dm-<name>.log (note that xl commands returning "segmentation fault" isn't an error, oddly enough.)

↪ Xen with Alpine Dom0 on LVM and LUKS