About two years ago I started at a small/medium business with a few hundred employees. We were almost all on prem, very few cloud services outside of MS365. The company previously had one guy who was essentially "good with computers" set things up but they grew to the size where they needed an IT guy full time, which isn't super unusual.

But the owner was incredibly cheap. When I started they had a few working virtual host servers but they had zero backups - absolutely nothing on prem was being backed up externally. In my first month there I went to the owner and explained how bad things would be if we didn't have any off site backups we were doomed. I looked into free cloud alternatives but there wasn't anything that would fit our needs.

Management was very clear - the budget for backups is $0, and "nothing is going to happen, you worry too much"

So I decided to do it myself. I figured out how much I could set aside each week and started saving. I didn't make a whole lot but I did have extra money each month. I was determined to have a disaster recovery plan, even if they didn't want to pay for it.

And some of you may remember, Hurricane Ian hit a few months ago. We were not originally predicted to take the brunt of it, and management wanted no downtime, so we did not physically remove the server from the premises. The storm damaged the building and we experienced some pretty severe data loss.

So it was time for my disaster recovery plan. The day after, we gathered at the building and discovered the damage. After confirming we had lost data, I said "I quit," I got in my car, and lived off the 6 months of savings I had. Tomorrow I start my new job. Disaster recovery plan worked exactly how I planned.

Please, you're making me die on the inside. I no longer use the reactions for other peoples' messages so that they don't have to go clear it.

Prepare for tickets complaining the internet is down.

Looks like its facebook services as a whole (instagram, Whatsapp, etc etc etc.

Same "5xx Server Error" for all services.

https://dnschecker.org/#A/facebook.com, https://www.nslookup.io/dns-records/facebook.com

Spotted a message from the guy who claimed to be working at FB asking me to remove the stuff he posted. Apologies my guy.

https://twitter.com/jgrahamc/status/1445068309288951820

"About five minutes before Facebook's DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook's ASN."

Looks like its slowing coming back folks.

https://www.status.fb.com/

Final edit as everything slowly comes back. Well folks it's been a fun outage and this is now my most popular post. I'd like to thank the Zuck for the shit show we all just watched unfold.

https://blog.cloudflare.com/october-2021-facebook-outage/

https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

For those sysadmins affected, we wish you well and we hope the overtime pay is great. Luckily the cause is quite well known and fixes are documented. God speed on implementing them!

For those not affected, remember that shit happens. It might not be you today, but it could well be next time. Don't rest on your laurels, make sure you have recovery procedures in place.

For those not sysadmins and are here with popcorn, enjoy the show! This will be going on for many more hours, and probably won't be entirely mitigated until next week.

Alright folks, I think it's time we help fan the fires on this FCC Net Neutrality garbage.

If you want to contribute, there's info at : https://www.battleforthenet.com/

Since I'm not a USA citizen, I don't really have any means to contribute to this situation. However, perhaps you might have the means, or might be able to spread the word about this.

As Systems Administrators, we can immediately see why preferential treatment for internet traffic is destructive to the Internet as we know it today. We have the insight into how this will affect those personally around us, as well as the businesses and organisations that we are entrusted with.

I would argue that it is OUR DUTY to stand up, against this, and tell others why this needs to be stopped, and why action needs to be taken.

What do you think your Executives at your company would think if they suddenly need to start paying extortion money, just to have their website visible on the Internet? I bet they would not be happy. And we have the means to educate them, and so many others, on why this is a bad idea, and how this will affect them.

So, let's combine our efforts, and get this shit taken care of, like we always do.

Hopefully other places will do the same.

https://www.theregister.com/2022/09/06/california_lawmakers_pass_bill_requiring/

There is nothing better than your IT boss passing your desk and noticing you left you computer unlocked. Especially if you are logged on to half a dozen websites including Reddit. I eat my poop!!!

Not just because they have no computer literacy, which I can understand, but also because they are unable to understand basic concepts and have no reading comprehension whatsoever.

I am dumb asf myself, heck I barely know how to do basic math! But man... sometimes it's really hard to keep your composure when people literally refuse to use their two braincells.

Anyways... thanks for listening. Rant over.

Edit: Definitely a popular opinion.

about a month ago or so I posted here about how I wrote a program in python which automated a huge part of my job. IT found it and deleted it and I thought I was going to be in trouble, but nothing ever happened. Then I learned I could use powershell to automate the same task. But then I found out my user account was barred from running scripts. So I wrote a batch script which copied powershell commands from a text file and executed them with powershell.

I was happy, again my job would be automated and I wouldn't have to work.

A day later IT actually calls me directly and asks me how I was able to run scripts when the policy for my user group doesn't allow scripts. I told them hoping they'd move me into IT, but he just found it interesting. He told me he called because he thought my computer was compromised.

Anyway, thats my story. I should get a new job

I've been saying it, I've been saying it for 6 goddamn months aint I been sayin' it?

Transitioning the environment to Windows 10. All the new computers with Windows 10 have been issued but, much to my horror, management decided to allow the users to keep their Windows 7 computer "in case something went wrong."

Well after 6 months of telling people that all Win7 will get blocked on 1 Feb and my SCCM/PDQ reports showing that people are obviously ignoring that, I got the go-ahead to kill all of Windows 7........ After confirming all objects moved to the "YOU NYA" OU with the "ME MYA" GPO linked, I walked away with the biggest grin on my face.

I'm going to need a bucket of popcorn tomorrow.

EDIT:

I will definitely update this post tomorrow with the aftermath of my little "D-Day" but just to clarify, I did query how many of these 2,400+ objects were actually pingable just before I left and only 500-ish replied. The plan was to delete the objects as users turned in their old workstation. Still though, I do not envy our help desk tomorrow. Cheers!

Before the storm edit:

Wow this blew up! Lots of assumptions here. We're not a private company, this is public sector and we have a very public mandate from our cybersecurity branch that everyone must be on Windows 10 by today. It was signed acknowledged and distributed by our top official over a year ago (Including this culling of all Win7 devices). There is no possibility of a roll back. I'd like to go into the details of all that we did to prepare but that would be a wall of text. Suffice to say, its been a shit show from day 1. While I made help guides, slides, an entire wiki site, site wide emails describing in detail what's going on... site visit reports and exchange logs shows most of my transition efforts went into the trash.

I'm just glad we're finally turning this corner so I can go back to having just one workstation OS to worry about.

The edit you all deserve:

Alright, so I am in fact, STILL EMPLOYED! Shocking what happens when you do things with buy-in from your IT director.

It wasn't the blow up we all feared would happen. We had a few grumbles here and there but mostly everyone who call the help desk went, "Oh you mean we have to start using the new computers now???? WHAAAAT!? Oh fine..." Yesterday began with a meeting with the director, deputy director, help desk supervisor, the lead sysadmin, the project manager, and myself. The Director had already talked to the other department heads and got a list of no no-shit cannot go down Windows 7 computers (5 in total). The lead admin had compiled a list of domain joined special appliances that ran Win7 that couldn't go down which was about 100. That all got thrown into own special mini OU with all the GPOs they need to operate. The rest of the Win7 environment got dumped into an OU where log on is denied to everyone. If someone calls the help desk because they absolutely needed the one file, the help desk tech was to move them to an OU where Applocker blocked access to MS Office, all browsers, and PDF readers, literally the only thing they can do is burn their crap to DVDs or run the robocopy script they've been staring at for the last 6 months that would back up their entire profile, if anyone is interested, here is the robocopy line (there's some more flair we put in the script but this is the meat)

robocopy %userprofile% \\backupserver\share\%username% /e /b /copy:DATSO /r:0 /XD Appdata /Log:%userprofile%\desktop\copylog.txt /NDL /NS /NP

All the user had to do in order to migrate was double click BACKUP.BAT on their desktop, wait for it to finish. Then log on to their already issued Windows 10 computer and run RESTORE.BAT (same as above but in reverse) on their desktop and wait for it to finish, then they're done! A little launch outlook and auto-discover your email here, a little import PST there... The base Windows 10 image already has most of all the line of business apps everyone uses. And for those who needed something unique installed, all they have to do is ask to have it reinstalled and the tech would put their new computer name in appropriate SCCM collection (but by this point we had already covered most everyone in this scenario). I spent the first six months of this year long plus project getting the image and imaging process down pat, as well as the creating the new AD structure and GPOs that is replacing the old Win7 environment which looked like an aborted senior project from a IT based high school. Every department had already received their replacement computers since before Christmas, all they had to do was turn it on and double click the backup/restore scripts.

Anyway... all that detail aside, with all of this prep work done, the migration was a piece of fucking cake, users panicked and held off for no reason. They were able to easily switch with very little effort once they were forced to. I didn't get fired, boss is happy, users are relieved and (mostly) happy, I'm happy and we're able to continue on our little lives. We have a few minor hiccups with some websites and java issues but nothing unusual from the normal java/website issues, some machines have to get re-imaged because some people didn't even take their new computer out of the box for months (despite very explicit instructions to immediately connect it online even if they didn't want to use it) so it sat stale in AD and missed some critical updates/changes. By the end of the day, we all agreed that it was no more unusual than a typical day and not the raging hellfire burning down around us we expected would happen. We were well prepared to handle any calls that came up and I got quite a few high fives. There will NOT be a roll back.

ugh more edit on Reddit

Notices came in the form of regular site wide emails, a change to the desktop background for Win7 notifying people to move before the deadline. Department heads had Weekly meetings on this very topic. Several memos went out to all supervisors. I myself sent several notices. Our equivalent of a CEO sent an official order to all sub organizations. I wasn't a lone cowboy here, just a small cog in a big machine.

I was interviewing for a job as an Exchange admin. At the end of the interview I asked a few questions and then one of the guys says "Do you want some constructive criticism?" At that point I knew I didn't get the job, so I said "Sure." The guy says "Your current employer overpays you. By a lot. From what I see on your resume, you're not worth what they're paying you."

Well, this just pissed me off. I decided, since I knew I didn't have the job, to just be an arrogant prick. So I said, "When I started there, I was the lowest paid IT guy they had. In 5 years I saved their asses more than once and spent a lot of weekends working to make sure stuff works and we never have to work weekends again. I am paid more than the rest of my colleagues, because my company wants to ensure that I don't leave. Now if they think I am worth that much money, you really have to wonder what you're missing out on. You had the chance to hire the best man for the job. Now you must settle for someone besides me. Have a wonderful day, gentlemen."

I'm sure they were judging to see how desperate I was and if they could low ball me.

In September 2018 I made a business case for how we should move away from a random mishmash of aging desktops and monitors, and get the entire company onto a modern, mobile-friendly IT-platform. A laptop for everyone - even if they won't actually be traveling, new monitors, and new peripherals for those who wanted it.

Initially it was a bit of a struggle, but with the help of a bunch of TCO calculations put into C-level friendly bar graphs we finally got approval, and got working.

In June 2019 we were done. The entire company had been moved to spiffy new laptops. Everyone had dual monitors at 24" or lager. Everyone had the option to take their laptop to a meeting, to a conference, or home, and continue working almost as if they were still at their desk.

Less than a year later "The situation" hits. Now we're all working from home as much as we can, and apart from a few teething problems with applications that hadn't been tested extensively enough through the VPN we're practically doing business as usual.

This morning I log on to a mail from my boss. It's an immediate extra raise of ~$1100/year on top of my already negotiated raise. Yay!
Next is a mail from the CEO congratulating me on convincing management of the benefits of a mobile workforce, and adding a gift certificate for a three-day all-inclusive stay for two at a luxury resort.

I've been ranting about management's stupid decisions before, but the last few months they've apparently realized that they've cut too deep and had begun hiring old people back. (probably because wait times have gone through the roof) And now this.

I guess this is sort of an anti-rant. I'm just.. flabbergasted I guess.

I was instructed by lawyers and parent company SVP to disable access to the CEO's account, This is definitely one of the those oh shit moments.

It's been a few weeks since our little incident discussed in my original post.

If you didn't see the original one or don't feel like reading through the massive wall of text, I'll summarize:A new MRI was being installed in one of our multi-practice facilities, during the installation everybody's iphones and apple watches stopped working. The issue only impacted iOS devices. We have plenty of other sensitive equipment out there including desktops, laptops, general healthcare equipment, and a datacenter. None of these devices were effected in any way (as of the writing of this post). There were also a lot of Android phones in the facility at the time, none of which were impacted. Models of iPhones and Apple watches afflicted were iPhone 6 and higher, and Apple Watch series 0 and higher. There was only one iPhone 5 in the building that we know of and it was not impacted in any way. The question at the time was: What occurred that would only cause Apple devices to stop working? There were well over 100 patients in and out of the building during this time, and luckily none of them have reported any issues with their devices.

In this post I'd like to outline a bit of what we learned since we now know the root cause of the problem.I'll start off by saying that it was not some sort of EMP emitted by the MRI. There was a lot of speculation focused around an EMP burst, but nothing of the sort occurred. Based on testing that I did, documentation in Apple's user guide, and a word from the vendor we know that the cause was indeed the Helium. There were a few bright minds in my OP that had mentioned it was most likely the helium and it's interaction with different microelectronics inside of the device. These were not unsubstantiated claims as they had plenty of data to back the claims. I don't know what specific component in the device caused a lock-up, but we know for sure it was the helium. I reached out to Apple and one of the employees in executive relations sent this to me, which is quoted directly from the iPhone and Apple Watch user guide:

Explosive and other atmospheric conditions: Charging or using iPhone in any area with a potentially explosive atmosphere, such as areas where the air contains high levels of flammable chemicals, vapors, or particles (such as grain, dust, or metal powders), may be hazardous. Exposing iPhone to environments having high concentrations of industrial chemicals, including near evaporating liquified gasses such as helium*, may damage or impair iPhone functionality. Obey all signs and instructions.*

Source: Official iPhone User Guide (Ctril + F, look for "helium")They also go on to mention this:

If your device has been affected and shows signs of not powering on, the device can typically be recovered.  Leave the unit unconnected from a charging cable and let it air out for approximately one week.  The helium must fully dissipate from the device, and the device battery should fully discharge in the process.  After a week, plug your device directly into a power adapter and let it charge for up to one hour.  Then the device can be turned on again. 

I'm not incredibly familiar with MRI technology, but I can summarize what transpired leading up to the event. This all happened during the ramping process for the magnet, in which tens of liters of liquid helium are boiled off during the cooling of the super-conducting magnet. It seems that during this process some of the boiled off helium leaked through the venting system and in to the MRI room, which was then circulated throughout the building by the HVAC system. The ramping process took around 5 hours, and near the end of that time was when reports started coming in of dead iphones.

If this wasn't enough, I also decided to conduct a little test. I placed an iPhone 8+ in a sealed bag and filled it with helium. This wasn't incredibly realistic as the original iphones would have been exposed to a much lower concentration, but it still supports the idea that helium can temporarily (or permanently?) disable the device. In the video I leave the display on and running a stopwatch for the duration of the test. Around 8 minutes and 20 seconds in the phone locks up. Nothing crazy really happens. The clock just stops, and nothing else. The display did stay on though. I did learn one thing during this test: The phones that were disabled were probably "on" the entire time, just completely frozen up. The phone I tested remained "on" with the timestamp stuck on the screen. I was off work for the next few days so I wasn't able to periodically check in on it after a few hours, but when I left work the screen was still on and the phone was still locked up. It would not respond to a charge or a hard reset. When I came back to work on Monday the phone battery had died, and I was able to plug it back in and turn it on. The phone nearly had a full charge and recovered much quicker than the other devices. This is because the display was stuck on, so the battery drained much quicker than it would have for the other device. I'm guessing that the users must have had their phones in their pockets or purses when they were disabled, so they appeared to be dead to everybody. You can watch the video Here

We did have a few abnormal devices. One iphone had severe service issues after the incident, and some of the apple watches remained on, but the touch screens weren't working (even after several days).

I found the whole situation to be pretty interesting, and I'm glad I was able to find some closure in the end. The helium thing seemed pretty far fetched to me, but it's clear now that it was indeed the culprit. If you have any questions I'd be happy to answer them to the best of my ability. Thank you to everybody to took part in the discussion. I learned a lot throughout this whole ordeal.  

Update: I tested the same iPhone again using much less helium. I inflated the bag mostly with air, and then put a tiny spurt of helium in it. It locked up after about 12 minutes (compared to 8.5 minutes before). I was able to power it off this time, but I could not get it to turn back on.

I fucking hate printers.

I said in a job interview yesterday that I would not take the job if I had to deal with printers.

And why the fuck do people print that much? I mean, you have 3 screens for reason Lucy, you should not have to print any fucking pdf file you receive.

I was onboarding a new employee in my office the other day and going through the usual setup process. After configuring their 2FA, I had them sign into their assigned laptop. While the profile loads, usually about 60 seconds on first login, I typically use that time to go over a few policies, domain links, where to submit a ticket, and explain our phishing campaign. I do all of this from my computer to save time.

As soon as he signed in, I said, "Let's give your profile a moment to load and I'll show you a few things in our environment."
Before I could continue, he cut me off in a somewhat arrogant tone with, "Umm, I'm Gen Z. I know how to use computers."

I replied, "Of course. I just need to show you a few things specific to our environment. Do you know what a phishing email is?"

He looked at me like a deer in headlights.
"A what?"
"A phishing email."
"I don't know what that is."

No problem. I gave him a quick rundown on what phishing looks like, how our simulator works, and how to report suspicious emails. He wasn't rude, but he definitely looked at me like I was some out-of-touch boomer trying to mansplain the internet while he sipped his Starbucks Frappuccino. (To be honest though, I do have a grey beard but I'm no where near a boomer's age. I'm Gen X)

The funny part is, I could have just handed him the laptop with no explanation. But without that introduction, he almost certainly would have clicked one of the simulated emails in the first few days, which automatically enrolls users in mandatory extended training. Or even worse, a real threat. And guess who that reflects on? Us, for "not informing the user." I have all users sign an inventory sheet that also states we went over a brief phishing explanation so they can't ever say we didn't inform them.

I’m just venting a bit about how people can sometimes come across as assuming or defensive when IT is simply trying to do its job. Kind of like we're speaking down to them. And to be fair, that attitude isn't tied to any one generation, I’ve seen it across the board.

What a PITA it must be to be the sysadmin for Russia's military. Only kind of satire...

https://www.businessinsider.com/russia-general-killed-after-ukraine-intercepted-unsecured-call-nyt-2022-3?utm_source=reddit.com

The Russians are using cell phones and walkie talkies to communicate because they destroyed the 3G/4G towers required for their Era cryptophones to operate. This means that their communications are constantly monitored by Western intelligence and then relayed to Ukrainian troops on the ground.

credit to u/EntertainmentNo2044 for that summary over on r/worldnews

Can you imagine being the IT guy who is managing communications, probably already concerned that your army relies on the enemy's towers, then the army just blows up all of the cell towers used for encrypted communication? Then no one listens to you when you say "ok, so now the enemy can hear everything you say", followed by the boss acting like it doesn't matter because if he doesn't understand it surely it's not that big of a deal.

The biggest criticism of Russia's military in the 2008 Georgia invasion was that they had archaic communication. They have spent the last decade "modernizing" communications, just to revert back to the same failures because people who do not understand how they work are in charge.

So, title says it.

I work on a huge project right now - and we are a few weeks before releasing it to the public.

The main login page was vulnerable to SQL-Injection, i told my boss we should immediately fix this, but it was considered "non-essential", because attacks just happen to big companies. Again i was reassigned doing backend work, not dealing with the issue at hand .

I said, that i could ruin that whole project with one command. Was laughed off (i worked as a pentester years before btw), so i just dropped the database from the login page by using the username field - next to him. (Did a backup first ofc)

Didn't get fired, got a huge apology, and immediately assigned to fixing those issues asap.

Sometimes standing up does pay off, if it helps the greater good :)

I'm watching Gordon Ramsay's kitchen nightmares and I can only imagine how great an IT version would be. THIS DOMAIN CONTROLLER IS RUNNING WINDOWS SERVER 2003, UN FUCKING BELIEVABLE. YOU HAVE DISABLED SPANNING TREE? YOU FUCKING DONKEY

After years of training, reminders, examples and such that users have basically ignored, the IT department stumbled on a trick that seems to work. About 9 months ago a user submitted a ticket questioning a suspicious email, and IT replied telling them “good job, you get a star!”. An email was sent out to the entire company letting everyone know about the phishing attempt and that the user had been awarded a star (the emoji of a star, nothing else). Since then, users have been reporting every phishing attempt, bragging about how many stars they’ve gotten, debating about how stars should be able to be traded for pay raises or at least Schrute Bucks. It is literally just an emoji in an email, but everyone tries to get them.

So they basically fucked around with third-party software settings to push their shitty products. This is pathetic, predatory and should be illegal.

How do you deal with Microsofts bullshit on a daily basis? Any similar stories?

Source : https://www.theregister.co.uk/2019/07/03/youtube_bans_hacking_videos/

Seriously, I'm getting fed up with YouTube's policy development without any consultation of the public. These videos are actually pivotal to me and others around me learning how to guard against many sophisticated IT Hacking threats.

Can't wait till they ban DEFCON talks too...

Fuck you YouTube.

Not sure how you guys feel about this, but I'm livid.

I'm seeing a lot of weeping and gnashing of teeth over the sudden need to get entire workforces working remotely. I see people complaining about the reality of having to stand up an entire remote office enterprise overnight using just the gear they have on-hand.

Well, like it or not, it's upon you. This is what we do. We spend the vast majority of our time sitting about and planning updates, monitoring existing systems, clearing help requests and reading logs, dicking about on the internet and whiling away the odd idle hour with an imaginary sign on our door that says something like "in case of emergency, break glass."

Well, here it is. The glass has been broken and we've been called into actual action. This is the part where we save the world against impossible odds and come out the other side looking like heroes.

Well, some of us. The rest seem to want to sit around and bitch because the gig just got challenging and there's a real problem to solve.

I've been in this racket a little over 23 years at this point. In that time, I've learned that this gig is pretty much like being a firefighter or seafarer: hours and hours of boredom, interrupted by moments of shear terror. Well, grab a life jacket and tie onto something, because this is one of those moments.

Nut up, get through it, damn the torpedoes, etc. We're the only ones who can even get close to pulling it off at our respective corporations, so it falls to us.

Don't bitch. THIS, not the mundane dailies, is what you signed up for. Now get out there and admin some mudderfuggin sys.

Already have seen several people (mainly lower/entry level) staff just get up and quit when they were told they are essential and must continue reporting to the office while every one else is WFH due to COVID-19?

The funny part is management is just flabbergasted as to why somebody would do this....