- Python 54.3%
- Nix 29%
- Shell 16.7%
| hosts | ||
| k8s | ||
| modules | ||
| scripts | ||
| systemd | ||
| .gitignore | ||
| flake.lock | ||
| flake.nix | ||
| nix-cluster.zip | ||
| README.md | ||
Lyrexo NixOS cluster
This flake configures three NixOS machines:
| Host | Hardware and OS | Role |
|---|---|---|
server-01 |
Intel N100, x86_64 bare metal | Initial K3s server; primary workloads; Longhorn replica |
server-02 |
Apple M4 Mac mini running an aarch64 NixOS guest in bridged UTM | Second K3s server; workloads/failover; Longhorn replica |
rpi-gateway |
Raspberry Pi 4, booting NixOS from USB SSD | WireGuard hub, Btrfs NAS/NFS, automatic K3s service gateway, Longhorn backup target |
The Pi is not a K3s node. The Mac mini host remains macOS; only its UTM guest runs NixOS.
Important limits
- Reserve DHCP addresses on the router in
10.7.4.0/24and provide local DNS forserver-01.home.arpa,server-02.home.arpa, andrpi-gateway.home.arpa. Router address is10.7.4.1. The Pi DNS name must resolve locally without Internet. - WireGuard uses
10.100.0.10(server-01),.11(server-02), and.12(Pi), UDP 51820. It uses peer/32routes; Internet traffic does not pass through the VPN. - Two embedded-etcd voters do not provide quorum if either compute node is unavailable. The Pi is not an etcd voter.
- The router must separately forward any desired public inbound traffic to the Pi. This repository does not configure the router.
- Longhorn's two replicas run on compute nodes. The Pi NAS is a backup destination, not a live Longhorn replica.
Prepare the repository and Nix
Use any Linux or macOS computer with Git and Nix to prepare and publish the flake. The Mac mini is needed only if you choose UTM as the aarch64 Linux build guest. Use /etc/nixos/nix-cluster as the checkout path on NixOS nodes and build guests.
Homebrew has no nix formula. Install Nix on macOS or Linux using the official terminal installer (it requires administrator approval):
curl --proto '=https' --tlsv1.2 -fsSL https://install.determinate.systems/nix -o /tmp/install-determinate-nix.sh
sh /tmp/install-determinate-nix.sh install --determinate --no-confirm
Open a new terminal after installation and check:
nix --version
Unpack the delivered nix-cluster.zip into ~/nix-cluster, then create and commit the lockfile:
mkdir -p ~/nix-cluster
unzip -q ~/Downloads/nix-cluster.zip -d ~/nix-cluster
cd ~/nix-cluster
nix flake lock
nix flake check
Create the public GitHub repository MaseDev33/nix-cluster, then publish the checkout:
cd ~/nix-cluster
git init -b main
git add .
git commit -m "Initial NixOS cluster configuration"
git remote add origin https://github.com/MaseDev33/nix-cluster.git
git push -u origin main
Do not install nodes until the public repository contains flake.lock and nix flake check succeeds. Flake evaluation/checks run on any supported Nix host. Full NixOS system builds require a Linux builder matching each target architecture: x86_64-linux for server-01, aarch64-linux for server-02 and the Pi. Build the Pi SD image on an aarch64-linux host such as the UTM guest. If your preparation machine cannot build a target, use its matching Linux installer to build/install that host, or configure a matching remote builder.
On an aarch64-linux host, clone the published flake and evaluate:
sudo mkdir -p /etc/nixos
sudo git clone https://github.com/MaseDev33/nix-cluster.git /etc/nixos/nix-cluster
cd /etc/nixos/nix-cluster
nix flake check
nix eval --raw .#nixosConfigurations.server-01.config.system.build.toplevel.drvPath
nix eval --raw .#nixosConfigurations.server-02.config.system.build.toplevel.drvPath
nix eval --raw .#nixosConfigurations.rpi-gateway.config.system.build.sdImage.drvPath
Build the Pi image there with nix build .#nixosConfigurations.rpi-gateway.config.system.build.sdImage. Build the server-01 system on an x86_64-linux builder; build server-02 on an aarch64-linux builder. Alternatively, nixos-install --flake ... builds the target system from its own matching NixOS installer. Physical hardware behavior still requires actual hardware testing.
SSH administrator key
On your admin computer, create a key and keep its private half there:
ssh-keygen -t ed25519 -a 64 -f ~/.ssh/lyrexo_cluster -C lyrexo-cluster-admin
For the N100 and UTM guest, use the local console after installation and put the one-line contents of ~/.ssh/lyrexo_cluster.pub into /tmp/lyrexo_cluster.pub. The Pi image builder injects that public key into the Pi image. Never put private keys, the K3s token, or generated kubeconfigs in Git.
Install the nodes
server-01: N100 bare metal
- Boot a NixOS installer in UEFI mode. On the N100's internal SSD only, create an EFI FAT32 partition labelled
EFIand an ext4 root partition labellednixos-root. Mount them at/mnt/bootand/mnt. - Clone the already-published flake into the new root and install:
mkdir -p /mnt/etc/nixos
git clone https://github.com/MaseDev33/nix-cluster.git /mnt/etc/nixos/nix-cluster
cd /mnt/etc/nixos/nix-cluster
bash scripts/install-server-01.sh
- Set the root password when prompted, reboot, and log in at the local console. Create
/tmp/lyrexo_cluster.pub, then bootstrap and activate:
bash /etc/nixos/nix-cluster/scripts/bootstrap-node.sh server-01 /tmp/lyrexo_cluster.pub
nixos-rebuild switch --flake /etc/nixos/nix-cluster#server-01
The installer script does not partition or format disks.
server-02: M4 Mac mini UTM guest
- In UTM on macOS, create an aarch64 Linux VM with 16 GB RAM, UEFI, a virtual disk, and bridged networking to the home LAN. Assign its DHCP reservation and local DNS name
server-02.home.arpa. Do not install NixOS on macOS. - Boot the aarch64 NixOS installer in the VM. Partition only its virtual disk (EFI FAT32 label
EFI, ext4 root labelnixos-root); mount at/mnt/bootand/mnt. - Clone the flake into the guest root and install:
mkdir -p /mnt/etc/nixos
git clone https://github.com/MaseDev33/nix-cluster.git /mnt/etc/nixos/nix-cluster
cd /mnt/etc/nixos/nix-cluster
bash scripts/install-server-02.sh
- Set the root password, reboot, create
/tmp/lyrexo_cluster.pubat the guest console, and bootstrap:
bash /etc/nixos/nix-cluster/scripts/bootstrap-node.sh server-02 /tmp/lyrexo_cluster.pub
When prompted, enter the token created on server-01 (see K3s setup below). Activate server-02 after WireGuard is configured:
nixos-rebuild switch --flake /etc/nixos/nix-cluster#server-02
rpi-gateway: Raspberry Pi 4 and existing NAS
- Keep the existing NAS separate. On the aarch64-linux build host, build an SD image and inject only the administrator public key:
cd /etc/nixos/nix-cluster
sudo bash scripts/install-rpi-gateway.sh /tmp/lyrexo_cluster.pub /tmp/rpi-gateway.img
- Copy
/tmp/rpi-gateway.imgto the machine used to flash media if needed. Flash it to the USB 3 boot SSD with Raspberry Pi Imager or Etcher. Select the boot SSD—not the existing NAS. The script does not flash physical disks. - Boot the Pi from USB, set its DHCP reservation and local DNS name to
rpi-gateway.home.arpa, then SSH in from the admin computer:
ssh -i ~/.ssh/lyrexo_cluster lyrexo@rpi-gateway.home.arpa
sudo mkdir -p /etc/nixos
sudo git clone https://github.com/MaseDev33/nix-cluster.git /etc/nixos/nix-cluster
sudo bash /etc/nixos/nix-cluster/scripts/bootstrap-node.sh rpi-gateway /home/lyrexo/.ssh/authorized_keys
sudo nixos-rebuild switch --flake /etc/nixos/nix-cluster#rpi-gateway
The Pi selects exactly one existing Btrfs filesystem of at least 3.5 TB and mounts it at /mnt/nas; discovery fails closed if the match is ambiguous. It never formats or repartitions the NAS. The mount uses compress=zstd,nofail, with a monthly scrub. NFSv4 exports only to 10.100.0.0/24.
WireGuard, K3s, and storage
Bootstrap creates /etc/wireguard/private.key, prints that node's public key, and creates the K3s token on server-01. Exchange only the printed public keys. Store on the Pi:
- Pi
/etc/wireguard/peers/server-01.pub: server-01 public key - Pi
/etc/wireguard/peers/server-02.pub: server-02 public key
Store the Pi public key in /etc/wireguard/peers/gateway.pub on both compute nodes. On each host, make the parent directory mode 0700, key files mode 0644, then run:
sudo systemctl start wireguard-peer-sync.service
sudo wg show wg0
Securely provide server-01's /etc/k3s-token to server-02 at its bootstrap prompt. Once the Pi and both compute peers have handshakes:
# On server-01:
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o wide
K3s is configured with Traefik and ServiceLB disabled, Flannel on wg0, and the requested 5/15/5-second node monitoring. Both compute nodes are K3s servers. Two-node etcd still requires both nodes for quorum.
On both compute nodes, verify the NFS automount at /mnt/nas, open-iscsi, NFS utilities, and /usr/local/bin/iscsiadm. Install a Longhorn chart version explicitly compatible with the Kubernetes version in the lockfile; then on server-01 apply:
sudo mkdir -p /mnt/nas/longhorn-backups /mnt/nas/backups/asterisk
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f /etc/nixos/nix-cluster/k8s/longhorn-rwx.yaml
The manifest defines retained longhorn-rwx, two replicas, and nightly backups to the Pi NFS share. Do not install example workload applications from this repository; it contains none.
Automatic Service exposure
Apply the read-only gateway RBAC on server-01:
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f /etc/nixos/nix-cluster/k8s/gateway-rbac.yaml
After the gateway-sync-token Secret is populated, export /etc/rancher/k3s/gateway.yaml using scripts/export-gateway-kubeconfig.sh and a temporary administrator kubeconfig. Store the generated file only on the Pi, root-owned mode 0600.
The Pi timer runs k3s-gateway-sync every 10 seconds. It queries server-01 then server-02, discovers LoadBalancer and NodePort Services, handles TCP/UDP and healthy endpoints, prefers server-01, and atomically reconciles its nftables rules. Deleted Services lose their rules; API outages preserve current rules. No NixOS port-list edit is needed for new Services.
Run mocked tests from the checkout:
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s scripts -p 'test_*.py' -v
Asterisk backup and operations
The compute nodes include path and timer units for /mnt/nas/backups/asterisk. Create that directory on the Pi. On both compute nodes provision /etc/ssh/asterisk_backup_key (mode 0600) and a verified GitHub key in /etc/ssh/ssh_known_hosts. Create the private repository MaseDev33/asterisk-configs-private, authorize the backup key, and initialize the NAS checkout with origin git@github.com:MaseDev33/asterisk-configs-private.git. Then enable:
sudo systemctl enable --now asterisk-git-backup.path asterisk-git-backup.timer
To update, commit and push changes to the public flake, including flake.lock, then update one host at a time:
sudo nixos-rebuild switch --flake /etc/nixos/nix-cluster#server-01
sudo nixos-rebuild switch --flake /etc/nixos/nix-cluster#server-02
sudo nixos-rebuild switch --flake /etc/nixos/nix-cluster#rpi-gateway
For recovery, inspect journalctl -u k3s, wg show, journalctl -u k3s-gateway-sync, nft list table inet k3s_gateway, journalctl -u discover-nas, and the previous NixOS generation. Never reinitialize etcd or format the NAS as a first response.
Validation boundary
Mocked Python tests do not validate Nix evaluation/builds, UTM bridging, physical boot, WireGuard handshakes, router rules, NFS, etcd quorum, Longhorn replication/backups, or GitHub credentials. Test those on the build host and target hardware before relying on the cluster in production.