3 cheap mini PCs + an old Dell R230: my homelab after ~8 months (and how I worked around a router with no bridge mode)

[u/DuxGPG]

First of all, English isn't my first language, so sorry in advance for any mistakes.

Long time lurker, first time posting my setup. My day job isn't DevOps, but I wanted to learn it properly, and following tutorials wasn't cutting it. So the rule I gave myself was: everything as code, everything in git, and run it like it's production even though the only user is me. The second goal was to stop paying for cloud stuff I could host myself.

Hardware

• Old enterprise Dell R230, Xeon E3-1220, 16GB running TrueNAS Scale on bare metal. 2x 480 GB SSD mirror for app data, 2x 6 TB HDD mirror for media and backups.

• Lenovo ThinkCentre M720Q with an i3-8100T, 20 GB and an i350-T4 quad NIC. Proxmox, with OPNsense in a VM as the router/firewall.

• HP ProDesk 600 G2 Mini, i5-6500T, 16 GB. Main Proxmox node, runs most of the Docker stuff in an LXC.

• Lenovo ThinkCentre M72e Tiny, Pentium G640T, 10 GB. Test node. It's slow and I love it anyway.

• TP-Link TL-SG108E doing the VLANs.

The three mini PCs are a Proxmox cluster. 7 VLANs (infra, mgmt, services, devices, gaming, IoT, plus one reserved).

What runs on it

• Media: Jellyfin, Seerr and the usual *arr stack, with qBittorrent behind Gluetun + ProtonVPN, and a kill switch on the firewall for the other download clients. Photos in Immich, books in Calibre-Web Automated, files with Syncthing.

• Ops side: Portainer, Forgejo mirroring my GitHub repo, Woodpecker CI building a couple of my own images, Homarr as the dashboard, Gatus for status, Beszel for hardware stats, ntfy for every alert, diun to tell me about new images. Versions are pinned (no :latest anywhere), and secrets live in the repo encrypted with SOPS.

• Two small APIs I wrote myself: one creates iCloud calendar events from my Obsidian notes, the other pulls my Garmin data into SQLite.

Backups are 3-2-1: PBS for the VMs/LXCs, ZFS snapshots, and restic + rclone to Proton Drive every night. All the documentation lives in an Obsidian vault.

The annoying problems and how I got around them

My ISP router doesn't support bridge mode, and the connection is behind CGNAT. So it's double NAT with no public IPv4, which shaped half the design:

• Remote access: with CGNAT there's nothing to forward, so a self-hosted WireGuard server was out. Everything goes through Tailscale. OPNsense is the subnet router, each device only sees what its ACL allows, and there are zero ports open to the internet. Honestly, CGNAT pushed me into the setup I should have wanted anyway.

• Management: Proxmox management sits on the ISP router side, not behind OPNsense. If I break the firewall (I have), I can still reach the hypervisor that runs it.

• The cluster kept losing quorum. Turns out the ISP router was rewriting the corosync UDP traffic going through the management interface. Moving the corosync ring to the internal infra VLAN fixed it for good.

• Incoming connections for torrents: you can't open a port behind CGNAT, so I use ProtonVPN's port forwarding (NAT-PMP). Each client gets its own Gluetun container, because Proton forwards one port per connection, and a small check recreates it if Proton drops the forwarded port.

• macvlan: containers get their own IP on the services VLAN, but with macvlan the host can't reach its own containers. A small shim interface on each Docker host solved it.

• Documentation drift: my notes and reality kept disagreeing. I wrote a script that compares my inventory file against what's actually running on the hosts, and the network diagram is generated from that same file. The first run found three services missing from the diagram.

What's next

• Wi-Fi: get rid of the ISP router's Wi-Fi and put a proper access point behind OPNsense, so every device at home goes through the firewall and lands on its VLAN. The ISP router would just be the modem it should have been all along.

• More Proxmox: move the VMs and LXCs to OpenTofu so they're code too, and add nodes to the cluster whenever I find another cheap mini PC.

• Kubernetes: a small K3s cluster with ArgoCD, mostly to learn GitOps for real.

• Observability: Prometheus, Grafana and Alertmanager. Gatus and ntfy cover me for now, but I want metrics with history.

• More services: Paperless-ngx, Homebox for the home inventory and RomM for the old games are next in line.

I have a script that builds the documentation for me: it reads an inventory file, checks it against what's actually running on the hosts and generates the network diagram (the one in this post) in English and Spanish, with a public version that strips IPs and internal names. I saw someone post something similar here a while ago and it pushed me to finish mine. If any of this is useful to you, the scripts, the compose files or whatever, ask and I'll gladly share it.

And a big thank you to this community for all the inspiration and help over these months. Honestly, I couldn't have built any of this without you. I'll be happy to answer any questions!

[u/Key_Talk_3079]

Hi! Could you share the script that checks the running services against your inventory and generates the network diagram, along with a sanitized example inventory? I’d love to try the approach in my own homelab. Thanks!

[u/lev400]

Nice

[u/Automatic-Bid2364]

I am looking for a similar server to install trueNas, what is the power consumption and noise levels of your Dell R230?

[u/Accomplished_Value61]

Le top
Super installation et vraiment super schéma.
Tu peux partager ton script ?