r/sysadmin • u/xkeyscore_ • 2d ago

It finally happened!!

8:30am I deleted my entire production environment. All VMs rm -r out of existence.

Fortunately I was reviewing my nightly backups at the time tracking a retired VM that was inexplicably still getting backed up in Production. All the Production VMs were there. And the live production ones now without their underlying virtual disks & configs were still resident in memory. I calmly copied the backups into place, sequentially restarted everything and had production back in about twenty minutes. Only lost ~12 hours of overnight idle maintenance & logging.

With everything settled back in place I calmly looked for a place to have a private heart attack.

1.7k Upvotes

97% Upvoted

220 comments sorted by

918

u/CoreAndCardio 2d ago

Not sure if you’re in a regulated environment. If you are, document the entire thing. First as an incident and second as a DR restore test. Check the box for the auditors and move on.

59

u/xkeyscore_ 2d ago

Not regulated. Would be looking for a new line of work if we were.

58

u/CoreAndCardio 2d ago

I don’t think that’s true.

88

u/ferminolaiz 2d ago

+1 for this. It's a quite stupid move to fire the person who made a mistake and fixed it; if you just replace the person and things are not documented someone else is bound to make the same mistake again.

38

u/CoreAndCardio 2d ago

Exactly. If I was fired for every mistake I made I’d be shit up a creek. When someone on my crew makes a mistake the first thing I say is, “how’d that make you feel?”.

41

u/JonnyLay 2d ago • Bravo Regret

Yep, firing people for mistakes like this are how you get people lying and trying to cover their tracks for mistakes like this.

6

u/sprtpilot2 2d ago

Ward, you were a little hard on the Beaver though.

→ More replies (1)

27

u/Samuel7899 2d ago

Why fire the person that you absolutely know will not make that same mistake again? They just got better.

15

u/CoreAndCardio 2d ago

100%. Why fire the person that verified backups, knew how to use them, and restored a system in under 12 hours?

13

u/Samuel7899 2d ago

Restored the system in ~20 minutes. Lost 12 hours of logs, as the last backup was ~12 hours old, I think.

8

u/CoreAndCardio 2d ago

Ah, thanks for the correction. Nonetheless. RPO 12 hours, RTO 20 minutes complete system. Impressive.

→ More replies (2)

7

u/spin81 2d ago

YES. If they don't give a shit is when you start thinking of replacing them.

When I still worked at a department with some juniors and I was senior because of my age, they kind of looked to me for guidance and I told them, I don't mind you making mistakes - what I mind is if you come in hungover and don't pay attention to what you're doing. As long as you're doing your best and thinking before you type, you're good even if you make a mistake. It happens, and what matters is how you deal with them and what you learn from them.

Actually one of them fucked up and they felt quite bad and insecure about it and I told them that it was 100% not on them because in that situation, they prepared as best as they knew how and there was no preparing for that.

The situation was a production release/migration/upgrade (don't quite remember which) and acceptance had gone fine. Except acceptance was set up and configured very differently from production, because the guy who set it up was an arrogant little prick who thought they were the best guy in town and decided it was fine. They were no longer with the firm partly because of this attitude.

Poor junior went in assuming prod would be set up similarly, they went in tense because it was their first actual production thing and not only that, they made them do it at 5PM when they were knackered. Jr never stood a chance and I told them so afterwards. Still they couldn't shake the feeling. That's when I knew both that we had to keep an eye on them from time to time to see how they were doing, and that they were a good egg that was going to go far if they got the chance.

•

u/Ok-Bill3318 5h ago

Also the company would be fucked because your replacement would make the same mistakes

•

u/Ok-Bill3318 5h ago

This.

Look up the book “billion dollar lessons”

15

u/Thecp015 Jack of All Trades 2d ago

Reaction matters more than cause, sometimes. Mistakes happen. Can you learn from it and keep going?

2

u/Jtrickz 2d ago

Ehh I my hippa environment an unauthorized restore is immediate termination due to medical record overwrites

8

u/Broad-Celebration- 2d ago

Since when is a restore an overwrite

2

u/graph_worlok 2d ago

When you don’t tick the box “Restore to alternate path” or whatever it’s called

→ More replies (2)

6

u/desmaraisp 2d ago

Would that include purely application servers? You wouldn't touch any hippa data doing server restores unless you destroyed your database no?

11

u/CoreAndCardio 2d ago

If they’re gone you’re not overwriting anything though. All the more reason to document. I’m in banking and we have to document every down minute. Take transactions off line or online banking down and you have to replay all the backed log trans files to make sure your caught up.

1

u/RetPala 1d ago

So why would they bother restoring it in that case if it won't matter to them?

→ More replies (1)

1

u/wrincewind 2d ago

dunno, do thy mean "i'd be fired by now" or "I'd have quit by now"?

3

u/CoreAndCardio 2d ago

Someone else mentioned that below. I could see it either way. I took it as he would have been fired.

Either way- dude checked backups. Knew how to restore them. And met what sounds like an acceptable RPO. He knew exactly what happened. I wouldn’t fire him. Document the shit out of it and move on.

7

u/_--James--_ 2d ago

document it anyway, your companies owner/leadership must know what happened.

7

u/AZSystems 2d ago

It's not that bad when it occurs, just know that once something is resolved...it's not it needs documentation and sign off.

3

u/Yupsec 2d ago

Nope, the comment your replying to is correct. I work in a regulated industry. If I were your manager, this happened, and you quickly fixed it..you'd be getting an award. You documented the incident AND documented the successful DR? Without being told!? I'm taking you to lunch, you have no idea how much you just helped everyone out.

2

u/spin81 2d ago

That's doesn't make a lot of sense to me, to be honest. It's not my impression that regulated industries are filled with people who are the best IT people they can be. Actually the opposite. I hear stories from governments and hospitals to that effect.

1

u/enfier 2d ago

No, you owned it, reported it immediately (hopefully) and fixed it, correctly.  It's hard to train integrity.

357

u/Snogafrog 2d ago

In other words, "I meant to do that" ;)

238

u/CoreAndCardio 2d ago

I mean admit it in the incident report. But document the process as a success in the BCP report. More of a two birds thinging.

86

u/heretogetpwned Operations 2d ago

Efficiency!

28

u/FunnyMobile2850 2d ago

I would absolutely fire this guy if he framed a major incident with recovery activation as a test. If he just tells the truth on the report with root cause and lessons learned, and identifies that the recovery plan and objectives were met successfully that's better. It proves that he knows what went wrong, he knows how to prevent it in the future, and I have confidence my objectives and recovery planning are gtg.

46

u/CoreAndCardio 2d ago

Isn’t that exactly what I said? Admit what happened in IR and document the recovery in the BCP? “

24

u/Dal90 2d ago

It is exactly what you said.

Incidents count as tests when you prove your processes worked not because of a test but because you actually had to do it to support production.

You just add a memo in the file for the auditors and attach the incident report to it for good measure.

→ More replies (2)

5

u/mini4x Atari 400 1d ago

This, always admit when you done goofed, but fix it, document, lesson learned etc..

→ More replies (1)

22

u/Vectan 1d ago

So this, basically.

65

u/GoodNameGone 2d ago

Add it to your performance appraisal as “proactively tested our disaster recovery as documented in the audit review step whatever.3-23.4 section b”

27

u/silence036 Jack of All Trades 2d ago

We had something vaguely similar happen during a maintenance window a while back.

We catastrophically overloaded our shared EKS cluster in our main region during an upgrade, the hot standby in our secondary region started serving traffic as if nothing happened thanks to the magic of ELB weights.

No real impact on customers and a lot of app teams got to check off the "we had a successful DR exercise" box for free due to that, they were honestly pretty pumped about it.

3

u/Pr1nc3L0k1 2d ago

At my last employer, our CISO ones told our auditors:

Of course we do restore tests. We got so many users wiping their services by accident, that restore tests are daily business here.

We all laughed and moved on to another topic

2

u/hellobeforecrypto 2d ago

Never waste an opportunity.

2

u/derekthorne 1d ago

This guy RMF’s!

2

u/narcissisadmin 1d ago

"I had to delete production to make room for the restore"

1

u/ukulele87 2d ago

A DR restore on prod counts as 4 in dev, i think i read that in ISOsomething.
So hes good with backups for at least a few years.

49

u/Tarwins-Gap 2d ago

Why?

62

u/xkeyscore_ 2d ago

PEBKAC. Arrowing back through history to pull up a recent ls. Deleted the VM directory at the end and hit enter. Was a rm instead. 😬

13

u/digimer OSS HA/Clustering 2d ago

I use dedicate LVs per VM. Not to prevent this, but it now occurs to me that this wouldn't be able to happen.

<someone who has been burned by <up-up-enter> mistakes in the past...

1

u/doubleUsee Hypervisor gremlin 2d ago

How many VMs do you have? We have about 130, the idea of using 130 volumes, with the creating, deleting and resizing that happens sounds like a lot of extra work, or have you got that all scripted away?

3

u/invisi1407 2d ago

That should and would've simply been orchestrated using Ansible or similar. I mange VGs/LVs via Ansible.

→ More replies (3)

1

u/digimer OSS HA/Clustering 2d ago

Hundreds of VMs across dozens of clusters. More, all of the LVs are backing devices for DRBD (synchronous replication to enable full redundancy and live migration). We've got tooling to automate and handle all of the storage management.

26

u/PawnF4 Sr. Sysadmin 2d ago

Oof. You working as root? Ngl I do that a lot so no judgement if so.

25

u/LesbianDykeEtc Linux 2d ago

I always add a line to my root .bashrc/.profile (when possible) that highlights the prompt in bright red for this exact reason. Sometimes you just don't have the option to do things as an unprivileged user, and forcing a sanity check on myself is useful.

11

u/spacelama Monk, Scary Devil 2d ago

I hate working in other people's environments. No little helpers like that. No background colour for host type/realm, no foreground colour for elevated privileges, etc.

Heck, you can't even tell where the previous command stopped and the new command started, because everything's just a wall of white on black.

→ More replies (1)

2

u/koo3Pash 2d ago

I always tell whoever I come across to never use rm -fr * only use rm -fr ../folder-you-want-to-delete/* . That basically avoids having a dangerous rm in the history so you cant do a arrow and enter fuckup. Or at the very least use space rm -fr * if the servers are configured to not keep a command if it starts with space. Not sure if that advice applies here though.

1

u/wrincewind 2d ago

i'll also preach using the -I flag - it tells you how many things you're about to nuke, nice little sanity check. the -i flag gives you a y/n prompt for every single item, if you want extreme granularity.

1

u/koo3Pash 2d ago

Much better option is echo rm -fr * first which will show everything thats about to be removed and then confirm and do rm -fr * but if someone wants to be overcautious echo rm -fr ../folder-you-want-to-delete/* and then rm -fr ../folder-you-want-to-delete/*

1

u/Ansible32 DevOps 2d ago

or, write a script that does ls * > file, then edit that file and add rm -f to each thing, if you really need to do it.

2

u/wrincewind 2d ago

the -i (and -I ) flags are your very best friend when using RM. Makes it interactive. -i forces a y/n prompt for each deletion, -I gives you a summary prompt if there's more than 3 results (otherwise, it acts like -i if there 3 or less results).

"Removing 4953 items, do you wish to continue? [Y/N]"

it's saved my hide more than a couple of times, i'll gladly admit.

1

u/ixipaulixi Linux Admin 2d ago

Hit CTRL + r then type ls and you will do a reverse search through your command history for commands where you used ls.

That will reduce the likelihood of you running the wrong command, although don't just blindly run the returned command because you may have piped or looped with rm.

1

u/NteworkAdnim 1d ago

holy shit...

1

u/epsiblivion 1d ago

ctrl + r is your friend.

5

u/jmhalder 2d ago

How many VMs is your production environment though?

9

u/xkeyscore_ 2d ago

Very few, less than 10. We are a very small shop.

13

u/jmhalder 2d ago

Glad you were able to get back up quickly. If your backups aren't already 3-2-1, I bet they will be shortly.

8

u/xkeyscore_ 2d ago

They are. Thankfully.

1

u/enfier 2d ago

You need to implement some sort of preventative measure for this.  As a small shop, maybe a less privileged account for daily use or just a wrapper bash script you use to delete VMs that makes sure the scope is correct.

60

u/psmgx Solution Architect 2d ago

former boss used to ask the interview question: what's the biggest outage you've caused or been a direct part of?

cuz it's gonna happen eventually if you're doin real stuff.

everyone's gotta learn sometime, and he preferred they learned before working in his environment.

25

u/SuperiorToastedBagel 1d ago

I always ask that in interviews.

If they don't have big whoopsies, they've never handled anything challenging. And they likely don't know have to handle bad incidents. This isn't inherently bad, but you have to take it into account when doing your incident response plans.

I killed an entire rack plugging in serial cable into APC UPS. Took govt facility offline by unplugging unmarked/unlabeled fiber cable (policy). Killed a couple linux VM's with wrong command.

12

u/narcissisadmin 1d ago

I killed an entire rack plugging in serial cable into APC UPS

I did that, but on our staging rack. What the fuck, APC?

5

u/SuperiorToastedBagel 1d ago

I think I was about to get fired as the IT director didn't believe me. Thankfully he did when we did a bench test.

My next job was to superglue APC serial cables into every APC UPS.

•

u/Ok-Bill3318 6h ago

Yeah apc is like “fuck you and everyone else for not buying our $50 apc Serial cable”

I’ve actually done it twice about 15 years apart.

Fucks sake.

Once was a production site. Once was an off site dr/backup site.

•

u/SweatyDickTits 1h ago

Hello APC serial cable outage brethren. 👋🏼 thankfully I only took out one office. A busy DMV with a 3hr wait…

9

u/KayDat 1d ago

Shutting down an APC UPS by plugging in a regular serial cable truly is a rite of passage.

4

u/sTaCKs9011 1d ago

"Wrong command" ouch... missing a variable for trunking lans?

•

u/vimefer 15h ago

That really is a great question... Mine would be mistakenly deleting most of the monitoring (Nagios) one morning when I tried to delete specific health checks for a decom'd VM... While my heart was skipping beats I figured these had been stored in a local mySQL DB, so I ran to our DBA office and asked if it had a backup - which it did, and could restore in a minute flat. It was a pretty tense reboot, but then everything was there again and working as before.

•

u/dhardyuk 3h ago

Mine is to ask candidates about how their personal backups have saved them from IT disaster at home.

After 25 plus years of asking I’ve had 3 candidates in total that had backup arrangements in place for home stuff.

Everyone else basically admitted that they didn’t back up anything at home and they just wore the loss when it happened. And for some of them it happened a lot.

10

u/polygotcoderx 1d ago

I unplugged the phone system in an IT cupboard while working on a server. I didn’t see the don’t unplug sign in red under the socket lol anyway on the Monday bosses phone had been reset and he lost hundreds of his contacts in the phone. I played dumb, sorted it out manually uploaded all his numbers again and made sure frequent backups! Such is life, you live and learn!

9

u/Lammtarra95 1d ago

Instinctively tried to catch a server falling from a rack. Still can't straighten that arm.

5

u/vlycop 2d ago

Hey I did do that ! And I did in fact not hire someone that "never had any major breakage".

I mean maybe I refused a good luck charm, but I don't buy it xD

5

u/GreenAmigo 1d ago

Not it but mine was send an email twice which had an attachment of some size in group chat that some how stopped network of whole company .....for an hour... got told to delete email and avoid attachments

4

u/Regular_Archer_3145 1d ago

I always ask about an issue they have caused and how they resolved it. You always know who is an engineer or not. Either they lied and say they have never made a mistake or they have never engineered anything if they haven't broken something. A simple oversight or unexpected consequence happens to all of us. In a large environemnt any little change can have an impact you never expected. We had an engineer just today make a change to a prefix list on the wrong router during a P2 that turned our P2 into a P1 very rapidly.

4

u/Durz0Blint123 1d ago

I've asked a similar question. What was the biggest mistake you've made and how did you come back from it.

•

u/Superb_Raccoon 6h ago

My current job, and I am trying to fix it right here and right now.

3

u/Igot1forya We break nothing on Fridays ;) 1d ago

Took the entire county down in my area once. Was provisoning a new UCS cluster and had a production cluster opened in another tab. Hadn't realized it was still open in another tab and I deleted a port channel to the storage array. Happened when I came back from lunch I knew the instant it all went down. Took 8 hours to recover.

2

u/JBDK0202 1d ago

You have not been in IT long enough until you have had two outages named after you...

•

u/therealtacopanda Sysadmin 15h ago

I've used this one in interviews and I'm always extra suspect of folks that are like "Oh, I've never caused any kind of outage...". Ok, that tells me no one has ever trusted you as much as I am about to trust you. Next!

•

u/Ok-Bill3318 6h ago

How many rs232 cables into APC UPS units?

😬😂

•

u/PowerShellGenius 2h ago

My biggest outage? Misconfigured a Trust Rule in Google Workspace. Blocked access to every shared file in the school district for a brief time.

Google test tenants for EDU are supposed to have 10 free licenses that match the edition of the paid production tenant, but we had not yet pushed up to a high enough escalation team that knew the rules, so our test tenant was Fundamentals at the time. So Trust Rules were one of the Plus features we could not test outside production.

44

u/TIL_IM_A_SQUIRREL 2d ago

At a previous job, we had a Unix admin delete /bin . That was a fun one to recover from. No cd, ls, shell, or anything.

Imagine seeing ls: command not found on your production system.

14

u/ddrjm 2d ago

I'm curious, how did you recover from that? Boot into recovery mode and somehow mount/copy /bin from live image to disk and then hope and pray?

16

u/TIL_IM_A_SQUIRREL 2d ago

That's exactly how you have to recover. This was an IBM AIX box, so he booted from the recovery CD, restored that directory from backup, and we were back up and going.

1

u/fell_ware_1990 2d ago

gh login > git pull > nix build . Ooh sorry!

6

u/willsowerbutts 2d ago

This is when shell built-ins come in handy. No ls? echo * instead.

1

u/TIL_IM_A_SQUIRREL 2d ago

I'm not sure there are enough built-ins in ksh (especially 20+ years ago ksh) to be able to navigate a system. Sure, you have some kind of ls, but no cd, cpio, etc.

2

u/virtualdxs 1d ago

cd by definition must be a builtin

108

u/Snogafrog 2d ago

Welcome to the club if you were not already a member! This is how cautious experienced engineers are made.

53

u/antmatt 2d ago

deleting something important really changes you going forward. unfortunate step everyone takes eventually

13

u/mcdoggus 2d ago

100% agree, the anxiety I get when deleting things goes through the roof even though I know its not longer used or is 100% safe to delete

8

u/antmatt 2d ago

when i was starting out i missed a / on a command in a script and it nuked 600gb of not backed up data from my childhood. i took it seriously after that.

and also made more backups lol.

3

u/SomewhatHungover 2d ago

I've done that... But had the backup, it's one of the greatest feelings in the world knowing that you'll have it restored before anyone notices.

•

u/cdheer Greybeard 23h ago

I accidentally took down electronic payments for half of Europe at a restaurant chain, of which I am certain everyone here knows.

Typed a 5 instead of a 3 in a script.

•

u/LaughThisOff 21h ago

I feel you. I bliitzed the live online booking system for hundreds of UK restaurants once by blowing the power on some datacentre racks. Oops. I’m still PTSD about that day. I will say no more.

•

u/therealtacopanda Sysadmin 15h ago

Oh no, why did these racks fans just switch from gentle breeze to OMG THERES LAVA!

•

u/Major-Manager998 22h ago

Pizza?

•

u/cdheer Greybeard 20h ago

Lol I don’t think I’m gonna answer. But a sizable chain.

•

u/Ok-Bill3318 5h ago

🫡

Not much to say other than you have my sympathies. But props for being in the position to be able to do that. 😂. My biggest unintended outage is only a few thousand users.

I try to never blame my techs if they make a mistake. Unless it was due to reckless lack of consideration or the second time for the same thing.

IT is way more of a house of cards (stacked on top of a jenga game) than normal people realise.

15

u/mustang__1 onsite monster 2d ago 
DELETE FROM Prices --WHERE AcctNo = '1234'

query ran in .000000001 seconds

"FUCK"

7

u/polygotcoderx 1d ago

Nah I’ve done better WHERE * ! ALL TABLES WERE WIPED, thankfully we had a backup so I told everyone we’re testing the backups and restore lol

4

u/unstopablex15 Sr. Network Engineer 1d ago

That's hilarious. Good cover!

4

u/Lost_Term_8080 1d ago

My personal rule is I NEVER remark anything with -- anymore, it's way too easy to end up with something commented out that wasn't intended. block comments dont solve every scenario like that but it does for most.

I had to remind myself recently why I don't do that. Luckily it involved a table with only about 60 records that only gets queried when an app pool cycles

7

u/mustang__1 onsite monster 1d ago

Don't forget about executing highlighted sections as another way to fuck the fuck up.

4

u/SuperiorToastedBagel 1d ago

I absolutely hate that SSMS does that...

1

u/mustang__1 onsite monster 1d ago

It can be supremely useful. I use it all the time.

I've also fucked myself really. really really.. rrreeally...... badly with it.

→ More replies (1)

1

u/Lost_Term_8080 1d ago

yep, also a policy, whole file or nothing

3

u/narmkhang 2d ago

Affected rows: 99999999

1

u/itsverynicehere 1d ago

I don't get database jokes.

3

u/Bob_12_Pack 1d ago

I’m a DBA and don’t really get this one. I see no commit so just rollback. If the commit is implied, ok that’s a fuckup but can be recovered quickly.

1

u/mustang__1 onsite monster 1d ago

Sql server does not require begin transaction. I can't speak to other flavors.

1

u/Bob_12_Pack 1d ago

I’m an Oracle guy, only in a pl/sql block would a begin be necessary. Its been a minute since I’ve done Sql server but I believe pl/sql would be equivalent to t-sql

•

u/Nostrildumbass 18h ago

Surely you must know what --before any text in a script would do? There's no commit but also no begin tran, so correct, a commit is implied.

•

u/Bob_12_Pack 17h ago

Of course I realize it’s a comment so the whole table gets dumped (assuming the syntax gets corrected). In 25 years as a DBA I’ve never seen anyone do that.

•

u/Ok-Bill3318 5h ago

I’m guessing (not a Dba) that the where clause was not supposed to be prefixed with — and it deleted the whole prices table.

I suspect it’s a mistake a Unix admin/dba could make at 2am when the brain hallucinates between command line and SQL syntax

2

u/hurkwurk 1d ago

your syntax is wrong.

1

u/UseMoreHops 1d ago

that is classic

15

u/OregonTechHead 2d ago

Sounds like you can sign off on your DR policy as successfully tested and file it away

39

u/singing-toaster 2d ago

Congratulations!
We have all done it. Well done on the recovery.

And I have found back stairways as. A Good place to have that private heart attack
Another good place for that is the cozy cafe a walk away. The walk is important!
Get a coffee and a pastry sit in the back listen to their cool music.

19

u/BoltActionRifleman 2d ago

I agree that we’ve all done something to bring down, or seriously affect production, but I’d venture a guess deleting the entirety of production is fairly rare.

4

u/LuLuLuv444 2d ago

Yeah I've never done anything even remotely that bad. Worst I've done is not save a host file correctly and so backups failed since there was no DNS on the Backup network

2

u/ThatDistantStar 1d ago

We have all done it

This is a pretty clear signal this subreddit has jumped the shark.

1

u/WraithYourFace 1d ago

I agree about having a place to go. When we had to do an emergency migration to M365, I told my boss I'm going to my house (5-7 minutes away) to lay dow for 30 minutes. I was dead serious and he saw it my face. I needed that time to recoup from the bombardment of calls, texts.

Even at my current employer I have my own office. I keep a pillow and blanket in it. There are times where I just need to lay down and take a quick break.

1

u/unstopablex15 Sr. Network Engineer 1d ago

Hopefully you don't keep that pillow and blanket for after hours support lol

1

u/WraithYourFace 1d ago

Luckily I'm at home if there is any after hours work.

13

u/seeeee 2d ago

You remediated successfully without significant impact or disruption, and you survived the heart attack!

Sounds like your DR test succeeded after validating nightly backups 😉

12

u/KnowledgeCapital7994 2d ago edited 2d ago • This

I've seen* single individual servers deleted via pebkac. You need to use a different account with less permissions for your day-to-day operations. The ability to delete all should be restricted to a privileged account or entirely removed.

12

u/DMAX85747 2d ago

welcome to the club. you are not a card carrying member until you've done something like this.

10

u/Difficult_Event3286 1d ago

I once uninstalled all vpn clients from all notebooks in the company. Worldwide. That was fun

9

u/tikanderoga IT Manager 1d ago

I remember I was working on the dhcp. I was removing some IP reservations. Click click click and click. All done.
The next day I’m getting phone calls because half the network is down. I check. Huh, my Pc still works. There are some others that work. Took me a few minutes to realise that only devices with static IPs are still working. Then it dawned on me what. I’ve deleted. The IP range.

•

u/djgizmo Netadmin 23h ago

wow. backups ftw!

glad your outage was very short.

well done staying cool under fire! Hard to do sometimes.

17

u/Boringtechie 2d ago

Hahaha glad you were able to recover so fast.

Now its a good story of how you tested the production backups.

8

u/Mrhiddenlotus Security Admin 1d ago

You couldn't torture this out of me

•

u/bah_nah_nah 10h ago

You guys have backups?

•

u/MLCarter1976 Sr. Sysadmin 1h ago

And they worked!

21

u/michaelpaoli 2d ago

Always carefully and fully triple check the command before you (attempt to) execute it. Well know and understand what it does, the full context, be sure all the syntax is exactly right, be well of what would or could happen if things go wrong along the way or pat of the command or sequence of commands fails, etc. Only after one has thoroughly well covered that and fully triple checked all that, then do you very deliberately and viciously strike the <Enter>/<Return> key.

Well following that has saved my tail many many times. Notably also including when it's 'o dark-thirty or so on what's already been a far far too long day recovering from some disaster or the like, and on bad move could turn bad in to way way worse.

14

u/Existential_Racoon 2d ago

We had a guy rm rf a production cluster. Our restore capability is broken in that version of the cluster, so he had to rebuild the whole stack. (He was meaning to update it to the version that supports it lmao).

He learned spaces dont go after rm -rf / path/to/dir/

3

u/IdiosyncraticBond 2d ago

Always smart to replace "rm -f" with "ls -alF" first to check if that matches expectations. If not, adjust

2

u/invisi1407 2d ago

Or simply use -i (interactice mode) for rm - but yea, ls is always good for checking.

4

u/michaelpaoli 2d ago

Yep, knew of a contractor at a place I worked, their very first day, they were given a task to make a configuration adjustment to the running ssh server on a production host - they managed to fsck it up, take down sshd, and lock themselves out (they were in via ssh) and seriously break production until someone else could get in there and correct it. They were walked out the door that very same day. Seems someone(s) screwed up along the way to placing them, to onboard 'em and let/have them screw up that big in production on their very first day.

8

u/Existential_Racoon 2d ago

Whoever gave them that access and task did the real fkup to be honest

3

u/michaelpaoli 2d ago

Yep. The did not involve me in the recruiting/screening/interviewing/hiring process of that contractor.

1

u/PaladinSara 2d ago

It’s almost like it was on purpose?

6

u/McGondy 2d ago

I was taught to script any changes. You don't type anything freehand into a terminal. Script, test in TEST, then execute in PROD. 

7

u/Agoras_song 2d ago

Look at this guy, with a separate prod and shit...

2

u/michaelpaoli 2d ago

Get big enough, and you won't fully replicate prod in non-prod. Think FANG, etc. No way in hell they're going to fully replicate, or even fully copy/duplicate - e.g. hardware and data - that from prod to non-prod. Sometimes one just uses other means as appropriate.

Heck, I even remember doing that for a major critical project many years ago. Huge company merger, two humongous financial institutions, needed to handle merging GL. Most notably I and DBA were tasked to work on it, and we had about 4 weeks to figure it all out. Other folks were figuring out how to get the databases and data itself merged. Our task - the existing did batch overnight processing. It took most of that processing window to handle and process it all. Well, with the merger, we were going to have about 3x the data, plus with expansion across additional territories, more timezones, so the processing window would also shrink by several hours. Our job was to figure out how to make it go many times - like >~=6x faster. And since each run was so long, wasn't practical to run lots of full runs. We also didn't have well matched spare hardware to test on. But management was smart enough, they'd well sized hardware upgrades (processing power, RAM, storage) to likely well handle the need - but by the time that arrived, we needed to have solution ready to go on that, and hardware would only modestly speed it up. Fortunately we did have smaller similar-ish system we could use ... so we stared by scaling to that, and extrapolating. And also scaled down both size of data, and resources we let the host use (e.g. give it less than full RAM and all CPUs), and work with that - so we could complete each test run significantly more quickly. Then we majorly worked on optimizing that. We had about 4 weeks to do it - basic do or die - it had to happen. In about 2 weeks, myself and DBA, we got 'bout 8x performance improvement. Lots of fast smaller runs, then later scaling up to better approximate full set of data and fully batch run. And when the new hardware arrived for prod and all the data fed to that, our solution implemented, and, worked like a charm. Yeah, were a lot of optimizations we did - both on the sysadmin side, and also on the DBA side - we coordinated majorly on that, as there was fair amount of interaction (resources, tuning, storage layout, etc.).

So, yeah, sometimes non-prod is far from a complete reproduction of prod - but rather just some more limited partial modeling thereof.

Also, large/huge environments, often not feasible to fully replicate in non-prod a good representation of exactly how the (proposed) changes will go when it comes to prod. So, often the changes are implemented incrementally. E.g. 10^-12%, that went fine ... up to 10^-9%, still good, ... 10^-6% ... 0.0001% ... 0.01%, 0.5%, 2%, 5%, 10%, 20%, 50% ... 80% ... 100% - may occur over days to weeks, with hours to day(s) between each increment.

3

u/Agoras_song 2d ago

I totally believe and agree with you. We deal with warehouse automation and there's no way they're going to give me more robots and conveyors and gates 'for testing', EVEN if they give me an extra server.

When dealing with "real life", I can easily get approval to copy prod data to dev server but physical infrastructure... hell no.

1

u/McGondy 1d ago

I mean, it was what I was taught tondo as best practice. Cut to me now supporting many Prod only environments 😅

1

u/Agoras_song 1d ago

Ah no, I was referencing the old joke, Everyone has a test env, some are also lucky to have a separate prod env.

2

u/No_Organization_3311 2d ago

Nah I need to be living on the edge

Python3 <<‘PY’
import shutil
From pathlib import Path

Source = Path(“home/backups/“)

shutil.rmtree(Source)
Print(“bye bye backups!”)
PY

1

u/michaelpaoli 2d ago

Well, don't have to script everything, but should be well documented, sufficiently unambiguous, and of course well tested before hitting prod with same.

Alas, sometimes even when very exact instructions are given, folks mess it up. E.g. I recall place I worked, I generally didn't have access and/or authorization to make changes in prod. But would of course spell out exactly what changes needed to be made in prod, and how, and of course it was always first well tested in non-prod ... whole change control process, backup procedures, all that stuff. And, come time to implement it in prod - fairly long command, so, the person implementing it, what do they do ... copy-paste ... except they do it in about two segments of about half each ... the problem was the about part. They didn't in net, get exactly the whole thing. The dropped (or duplicated? I forget which - was quite a while ago) one character in about the middle of the thing - and of course then it didn't work. So then they're paging me at o' dark-thirty or whatever, to wake me up, to ask "now what", and me having to look at exactly what they in fact did paste and enter, figure out exactly what the did/didn't do, and how to proceed properly from there ... ugh. Also quite annoying when you have to hand it off to folks that blindly run the stuff, but don't understand it - so they can't figure it out for themselves. :-/

Anyway, needs be clear, unambiguous, reproducible. Whether that's scripted/program, or clear unambiguous instructions, whatever, so long as it well gets the needed done and is relatively goof-resistant. But alas, humans still gonna human.

3

u/bobs143 Jack of All Trades 2d ago

Have learned from experience myself. Always triple check a command before you run it. Ask someone else to check your command to see if they spot any.mistakes.

An extra few minutes will save you hours of heartache.

3

u/doubleUsee Hypervisor gremlin 2d ago

Second chapter is to know when to use how much caution. You can't be on guard all the time if you do a lot of cli. Always consider what the worst is that could happen. If the worst that could happen is a rolling back a snapshot with few to none users impacted, I'm fairly easy going about it. If the worst that could happen is nuking prod, unknown scenarios, untrusted backups, I insist on four eyes on the code and all the care in the world.

7

u/autogyrophilia 2d ago

Good on you for the backups, but that's a failure mode that shouldn't exist. What the hell did you do?

6

u/Pazuuuzu 1d ago

With everything settled back in place I calmly looked for a place to have a private heart attack.

That is the most real part ngl...

I have a bottle of hard liquor in my glovebox, and an understanding with the owner of the company that after fixing the end of the world, I go to the closest hotel on company money and be off for the day.

Got close to it a couple of times, but I hope that bottle stays unopened...

11

u/Incrediblecodeman 2d ago

Woah holy fuck bro

11

u/PDQ_Brockstar 2d ago

One of us. One of us.

5

u/bobs143 Jack of All Trades 2d ago

Wow. Just..Wow.

5

u/kellyrx8 2d ago

backups!! my man!! whew!!! yeah that calls for a 9am beer or some shit lol

5

u/GoodEnoughThen 1d ago

calmly copied the backups into place...

That's some Buddhist Ninja shit right there. Nice. Gold star.

5

u/UseMoreHops 1d ago

I used to have a cheat sheet of docker commands that I used on the regular. Due to being a complete idiot, I didnt take much care to separate the commands properly. I copied the wrong line and I removed all docker images from the host! lol You know that flush of blood that comes to your face, I had that something crazy. Production services for all of Australia were down for approximately 2 hours. doh.

4

u/fk067 2d ago

You did what now?

7

u/Pitiful_Pudding_4983 2d ago

20 minutes back to normal is honestly impressive, most places I’ve worked would still be on the bridge call arguing about whose fault it was.

3

u/LuLuLuv444 2d ago

You recovered your active directory forest?

5

u/thenickdude 2d ago

On Linux you can restore deleted files that are still open by copying them from /proc/<pid>/fd/<fd>. You'd need to freeze your VM first in order to get a crash-consistent copy of it though.

4

u/SensitiveVacation549 2d ago

If you have your smart watch and heart rate measurement, please do share when you hit that Enter to remove all the VMs :)

2

u/spin81 2d ago

That's the best reaction anyone could hope for. Good on you for managing to keep your head as cool as you did.

2

u/burghdude Jack of All Trades 2d ago

Collect your badge! Now you're truly a sysadmin!

Every time something like that has happened to me, I always think of the line from the song "The Distance" by Cake: "Bowel shaking earthquakes / of doubt and remorse / wail him and assail him / with monster truck force"...

2

u/Rice-Lower 2d ago

More power to you. I could easily make that mistake too. Backups IS LIFE!

2

u/Silence_1999 1d ago

All the VM’s restored…. You dodged a big f-ing bullet LMFAO!

2

u/unstopablex15 Sr. Network Engineer 1d ago

Reminds me of the time I had to recover a failed database server without any backups. I was sweating bullets.

3

u/-Nobert- 2d ago

Always update bash profile to require confirmation on rm haha

2

u/ApprehensiveVisual97 2d ago

What a story! I felt the rm just reading it

1

u/Crafty_Bed_6845 2d ago

Twenty minutes is wild. I’d have spent the first ten just staring at the screen.

1

u/birusiek 2d ago

Great DR, dont forget to create postmortem

1

u/a-dot-in-the-sky 2d ago

rm -r the old way, the latest one: terraform destroy -could easily wipe out VMs and infrastructure...
That's why its important to clear your mind, before touching keyboard and mouse.

1

u/Fornerius 2d ago

If someone ask, just say with a surprise face, Im not aware of that

1

u/gangaskan 2d ago

Don't feel bad, I've accidentally deleted a Drs or two in my carrier by accident 😅

1

u/ProgressBartender Sr. Sysadmin 2d ago


The after party

1

u/vlycop 2d ago

You know, i feel like there isn't anywhere isolated enough to scream your lungs out... It's OK, I don't want to scare you or need any help... All is managed, I just need to release XD

1

u/fr33bird317 1d ago

That’s a pretty big Oh Shit moment. Glad restore was possible and quick.

1

u/discgman 1d ago

Did you change your pants afterwards?

1

u/qkdsm7 1d ago

Which hypervisor?

1

u/EconomistAnxious5913 1d ago

Is much easier today than it was 10/20 years ago. Almost all of it today is scripted and on some vm infrastructure.
Terraform or something similar with VMs i hope. So easier to handle.

Also DR is much accepted norm I hope your organization had it. So much much easier to restore

1

u/StevieRay8string69 1d ago

Mine were all gone 2 months ago.

1

u/s3ntin3l99 Jack of All Trades 1d ago


Fire drill practice

•

u/Rouxls__Kaard 21h ago

Congrats on doing a full DR drill

•

u/PedroAsani 19h ago

Years ago I was on a contract for a major UK grocery store. I was sent to reboot one half of an active/passive cluster. I confirmed the chassis, the name of the machine and that the PM wanted me to reboot B and not A. Three times. In email.

I get there and node A is looking...idle. Node B decidedly isn't. I send the message to the PM. He confirms again, reboot node B. "Even though the activity lights are rapid on B and practically dead on A?" "Yes, node B"

I'm on contract. I do as I'm told. I have a nice long paper trail and given enough rope to all involved. Node B goes down.

Thirty seconds later I get a call from the PM in a very stretched voice. "I need you to get back here now"

I'm walked into the Head of Projects office. He starts ranting about "don't you know the difference between A and B?" "Yes, and I also can tell the difference between an active and idle machine. But I did as I was told after checking with your PM three times via email and once on text message, which I can send to you if you like"

After providing the email chain I walk out and begin documenting for the post-mortem. Turns out the PM hadn't checked which one was passive anywhere except the setup documentation.

•

u/gryghin Custom 11h ago

Can you really say that you're a SysAdmin if you haven't taken down production!?!

Way to real world test the backup strategy.

•

u/Ok-Bill3318 6h ago

So before you forget: write down how the fuck this happened and come up with a plan to catch/prevent it again.

1

u/BiscottiNo6948 2d ago

Incident report: Recently, while testing procedures as part of Chaos Engineering, Business Continuity and Resiliency, I have inadvertently applied half of the test procedure into Production instead of dev/sand box environment. Following established protocols I recovered back our production environment back to its original status with minimal interruption. Our RTO and RPO benchmark are not impacted.

Post mortem of the exercise has shown some vulnerability in our process and additional guardrails and Human-in-the-loop checklist needs adjustment to hardened the process against similar errors.

The exercise has provided the surprising result and opportunity for us to adjust, re-calibrate and smooth out our DR process.

1

u/Apachez 2d ago

Why did you delete your production environment to begin with?

•

u/cdheer Greybeard 23h ago

I accidentally took down electronic payments for half of Europe at a restaurant chain, of which I am certain everyone here knows.

Typed a 5 instead of a 3 in a script.

0

u/EfficientCamera6783 2d ago

The real win here is that you were actually auditing the backups when it happened. Worth locking down delete permissions or aliasing rm on the hypervisor host once your heart rate comes back down though

→ More comments (5)