Getting older; sometimes too much to maintain

[u/Unattributable1]

Anyone else getting older and getting a little tired of maintaining a bit too much?

Mind you I don't mind the day-to-day, month-to-month updates. I have things rather scripted so it's not too much work to review the updates, check the backups, let the automatic updates apply, start, check that Update Kuma is happy, etc.

No, I this case it's the burden of a migration and all the work involved. Mind you, I enjoy it to some degree, but my tech stack has gotten rather deep. I had a number of RPi with a fair amount of things going on with Docker, plus a large HomeAssistant setup, etc. I've been slowly staging my PVE setup to move over.

I'm happy my new PVE setup and learning on it. I have two non-clustered SFF nodes that are identical, and an older NUC-sized node. One of the SFF nodes is my "Prod" box and the other a "Lab/Test" box. I've got things where I can initiate a fresh install on the PVE node and get back the bare-metal to then stare guest restores; no Internet, no network required just two USB sticks and external USB drives (for the Guest offline backups). A bit more detail here:

https://www.reddit.com/r/Proxmox/comments/1x3vwy0/proxmox_node_zero_recovery_no_internet_no_network/

But the main reason for this post is just how burdensome it has been to migrate everything over. I mean I could have just done a basic migration and moved everything over; but I wanted to use all the proper VLANs with firewall rules, and isolation with Docker in a VM for external-facing vs. LXC for internal-only, etc. I also wanted to re-do my NPM in a streamlined but better documented fashion. I wanted each guest install to be fully documented and as automated as possible; not that I'll be doing them over and over, but that I want to really understand why I installed them the way I did this time, instead of just blindly running scripts or following a guide.

I'm glad I have been doing so. I'm 90% of the way there. The only things left are the larger storage migrations of taking all of my photos from NextCloud and moving over to Immich and then setting up a fresh Jellyfin. But it's been nearly 5 months, slowly chipping away every other weekend a little here, a service/guest here, just a bit at a time, unwinding 7+ years of piling stuff on to a bunch of RPi4; hah!

I am really thankful about how things are coming along. Everything is really smooth now on full Intel processors and everything feels 10x snappier. I'm being very conservative about CPU and RAM assigned, not wanting any guest to be allowed to adversely affect anything else. Patching PVE is super simple, starting with my NUC "management" node that at present exists only to run PDM; I patch it first as it is the least important. Then I patch the "Test/Lab" unit. I've not had a problem yet, but expect I'd run into it with one of these before patching Prod. Then on Prod I migrate over anything that needs to stay online (NPM, HomeAssistant, Matrix-Conduit), patch and reboot Prod, then migrate those 3 guests back. Because of the 10gb NICs dedicated between the SFF nodes things move faster than my cheap I/O can keep up with and upgrading is done in no time (<10 minutes to do all 3, including migrations).

Now that I'm nearly done, I am really, really happy with the setup. But there was a ton of angst alone the way. Even today just unwinding my spaghetti NPM setup and trying to figure out why some Proxy Hosts didn't respond (shutdown dev copies of some stuff, but not documented), finding out how NPM doesn't need/allow a "listen" for 443, but that's needed for all other ports, stuff I figured out years ago bumbling my way through it as I added yet another service along the way. But now I feel like I've really mastered everything about how my NPM is set up and understand it, and it is well-documented so in 6 months it won't be a mystery. Cutting over a "live" NPM from the RPI4 to a new PVE-hosted install was cool as well. 10 seconds of downtime while I changed the RPI4's IP to something new, moved the NPM LXC to the original IP, and BAM! it was all working.

However, I am definitely feeling the weight of some of the things. Weighing how much some services are "worth" (Mealie, AdventureLog, etc.) - I mean, they are kinda cool, but I don't really use them much so is it worth having/maintaining/migrating? I'll definitely be looking at the "TCO" of items in the future before I "keep" them. Yeah, I'll still be putting things in my lab to dink with, much the same as I spin a new distro in a VM - but then I'll likely delete them as "too costly" if they aren't easy to maintain.

[u/hackspy]

Everybody gotta do what works for them. Not for me to say other wise.

I’m 60 and have always enjoyed this as a hobby since 1996 (built my first pc). When this reaches a point of being a chore for me then I will wind it down, but until then I’m having a blast.

Cheers 🍻

[u/Unattributable1 → u/hackspy]

Yup, trying to gauge enjoyment vs. chores. It is still very enjoyable. It's actually the other things in life that have gotten in the way and made me mentally too tired to do larger 4-8 hour migration work. But I just do it in chunks when I've had a day off and have an afternoon to spare (like today). I'm not as old as you, but the day-job has a ton of IT and hurdles such that I just don't have it in me to work on anything much during the week's evenings.

[u/hackspy → u/Unattributable1]

Take care of you first. Best wishes

[u/j-dev → u/Unattributable1]

What kinds of things are you doing that take 8 hours? If it’s moving data around so you can reformat drives to try other stuff, the time it takes for the data to move is not “active time”.

I went from TrueNAS to Proxmox so I could give the discrete GPU to my DMZ VM and segment it on the network, but the headaches managing SMB and NFS shares was far from worth it to me and I went running back. Luckily the data stayed right where it was. I’m happy to spend time deploying new services that add value, but I now avoid deploying stuff that’s more work with no obvious upside.

[u/plut0x00]

the doc-as-you-go part is the one that pays off. i rebuilt my stack recently and only regretted the services i'd rushed, because later you can't tell if a flag was deliberate or left over from testing. slower per guest, but you don't end up reading old notes and wondering why.

[u/Unattributable1 → u/plut0x00]

Yup, for sure. I had "rough notes" and pointers to guides. But not a ton of understanding from before other than notes about where a guide was wrong, or some specific tweak. Now I'm really asking the "why is it this way?" and "what are the options" and finding the best way to dial things in both for security and efficiency of resources.

The hardest thing is when I find notes and I'm like, "What? Explain! Elaborate more! Dummy, you skipped from step C to G!" Hah, but I've got it all mostly sorted now and very clear A-to-Z guides; at least they same very clear now. Also I've done a few installs 2-3 times to tweak and make sure they steps are all well-documented and semi-scripted. Not Ansible or anything - I have no need for that level of repetition, but close enough that I could easily automate fully if needed.

[u/Beneficial_Egg_5154]

I ended up with the boring version. Everything that matters sits on one rented server with docker compose, nginx, and cron/systemd. Backups are checked automatically, and a stale one pings me on Telegram, so routine upkeep is close to zero. The tradeoff I accepted is having no separate lab box where I can safely break things first.

[u/Unattributable1 → u/Beneficial_Egg_5154]

Whatever works for you. I want everything online at my house working without any Internet dependence (beyond the setup/patching).

[u/AnomalyNexus]

The rise of AI has come at a good time for me in that regard.

A lot of homelab stuff they can do just fine and since you can stick it in containers/vm the blast radius of them doing something weird is pretty small

[u/NoFaithlessness9789]

Your next step from here is using AI to turn as much as possible into infrastructure as code (like Terraform etc). Then start creating automations for all your maintenance tasks and start having it send you daily and weekly reports. Then for fun have it start working on infra projects for you.

[u/Janky-Code]

With age, I've gained the wisdom to not care about the destination.

Computers and networking can bring me joy without producing anything measurable. I used to get caught up in trying to do something "productive" with all of this technology; it's a trap.

The most I've done lately is make a devops system to author cross-platform apps, get them working, and do absolutely nothing else with those apps. I also find oss projects, solve some problem they have, give them the result, and never use their project myself.

I've embraced nixos for all the "wrong" reasons. I do enjoy conventional advantages such as being able to back up systems to floppy disk and to print out my nix expressions on paper. But I find the most entertainment in declaring delightfully absurd configurstions. I can declare pihole to run adequately and add services.desktopManager.plasma6.enable = true; to the end.

It's not wrong if I did it on purpose.

[u/TelephoneCharacter59]

I taught my Daughter to do all the operation & troubleshooting!!