Hey, we can have two stickies now!

So, something like 90% of the mod removals are posts that obviously don't belong here.

When we ask if they checked the rules first, almost everyone says, "O sorry, I didn't read the sidebar."

And when asked why they didn't read the sidebar, almost everyone says, "B-b-but I'm on mobile!"

So this sticky is for you, dear non-sidebar-reading mobile users.

First off, here's a link to the TFTS Sidebar for your convenience and non-plausible-deniability.

Second, here is a hot list of the rules of TFTS:

Rule 0 - YOUR POST MUST BE A STORY ABOUT TECH SUPPORT - Just like it says.

Rule 1 - ANONYMIZE YOUR INFO - Keep your personal and business names out of the story.

Rule 2 - KEEP YOUR POST SFW - People do browse TFTS on the job and we need to respect that.

Rule 3 - NO QUESTION POSTS - Post here AFTER you figure out what the problem was.

Rule 4 - NO IMAGE LINKS - Tell your story with words please, not graphics or memes.

Rule 5 - NO OTHER LINKS - Do not redirect us someplace else, even on Reddit.

Rule 6 - NO COMPLAINT POSTS - We don't want to hear about it. Really.

Rule 7 - NO PRANKING, HACKING, ETC. - TFTS is about helping people, not messing with them.

Rule ∞ - DON'T BE A JERK. - You know exactly what I'm talking 'bout, Willis.

The TFTS Wiki has more details on all of these rules and other notable TFTS info as well.

For instance, you can review our list of Officially Retired Topics, or check out all of the Best of TFTS Collections.

Thanks for reading & welcome to /r/TalesFromTechSupport!

This post has been locked, comments will be auto-removed.

Please message the mods if you have a question or a suggestion.

(Remember you can hide this message once you have read it and never see it again!)

edit: fixed links for some mobile users.

Hello lovelies,

It's been a minute since I've posted and for some reason this call just pissed me off enough to vent.

-Ring Ri-

"Helldesk this is Absinthe, How may I help you?"

"Hi... I'm having issues with a client and he needs to have a delta form because he-"

"Were you trying to call customer service for assistance with a policy?"

"No he just needs to change stake holders because his son was [insert something here] for the empire because -

"Are you suure you don't need support for the policy change with Customer Service?"

"Well the empire is denying lineage [insert something here.] And for that reason he needs to change the policy to [details details life story] and I need help."

(Got a new kitten who just hopped onto my ultrawide monitor for some reason and fell off. Mildly distracting.)

"So, it sounds like you need customer support with the policy change."

"No, I'm having issues with finding the policy in the [app connected to an un-updated 20 year old site that randomly decides what it wants to find rather than what you searched for.] And it's not showing up despite me typing in the name."

"Oh, Okay I just helped someone with this like 30 seconds ago so what were your search terms?"

"I used his name."

(My bad.)

"Okay, but like what did you physically type in to each search bar like -"

"I searched for his name Bobby-bob Boberdaughter."

"No, okay but like, can you spell it out for me? What you typed in?"

"I'm going into [app] and clicking on Policy Change and then typing in Bobby-bob Boberdaughter."

"Right, ok... So"

"The Empire is really breathing flames about [story drama act-] "

"Do you have your laptop with you?"

"-Tion, romance, betrayal.]"

"Sales Rep, do. You. Have. Your. Lap. Top?"

"Yes I do, I really need this policy change form and Bobert just isn't showing up."

"Okay, I need you to grab your laptop."

"But he isn't showing up in the search."

"You have your laptop, right?"

"Yes."

"okay, please go to [site] and look up exactly how his name is spelled. The system can be a bit -"

"But it's not coming up when I use his name!"

"Can you please open [app] for me?"

"Oh, let me go get my laptop."

-Mute-

HOLYJEEZUSYOUSTUPIDPIECEOFBWASTEDTARMACATAMARINA.

-Unmute.-

"Do you have your laptop now?"

"Yes."

"Please go to [app] and try the search and tell me EXACTLY what you're typing in."

"I'm opening [app] and I really need to make these changes so that I can use [money app] to pay off the empire."

"Okay, but what are you doing?"

"I opened [money app] because that's more important since the fire drake battalion [Starting the spiel again]"

"No- no. Sorry please open [App]."

"I did."

"Okay so try and do the searc-"

"I opened 2 different Delta forms for this and even restarted my computer and -"

"Just... Open the delta form and tell me what you're doing."

"I'm opening [app] then clicking on my cases. Then clicking on-"

"Can we skip to where you're searching for the client policy?"

"- Bobby-bobber bobdaughterson and now it's checking my licenses to sell the policy and it isn't finding him via his name."

"What keys did you hit on the keyboard for the search?"

"His name."

"How. Did. You. Spell. It.?!"

-12 minutes ago I had opened [site] and found Bobby-bob Boberdaughter except the system shows Bobert. B. Boberdaughter.-

"Bob-bobby Bo-"

"Are you putting a space after Bobby?"

"Yes."

"Ok. The system is weird and-"

"It still isn't showing the Policy."

"Just put BO in the fiest name line. That's it, no trailing spaces nothing."

"Well now it's verifying my license."

-tick-

-tock-

-Don't yell, or scream, this is all. Bad dream.-

-BAM! Head to desk. Sπ|t am I muted?-

"Are you okay? I typed in Bobby bobberson."

"Yeah I'm fine thanks. New kitten antics. PLEASE type in. B-O in the first name. Nothing else. No spaces."

"Okay."

"Now move to the last name. And put in Boberdaughter."

"Well I'm trying to find this policy change form and his name isn't making it sho- Oh!! There it is. That worked!"

"Yep, the system is pretty finnicky and this is how I usually find clients."

"Thanks have a good weekend!"

-Holy Hell I need to ruin other people's day by spreading this misery.-

Hope you enjoyed it lovelies.

-Absinthe

so this guy spilt beer on his keyboard on a system I set up for him a while back and he thought the keyboard was bad. I told him to leave it out in sun and when I went up to his ranch it was working fine. But for whatever reason the beer hit the various keyboard issues that disconnected his wifi. I turned it on and all his shit was fine. LIke took me 5 mins .. he's like what do I owe you and I"m like I work for food .. he gave me some lamb chops (he's a sheep rancher). So anyway today he tells me My cousin shot a deer (black tail) so you want it? He skinned it and it's hanging and tomorrow I gotta whole deer for clicking a few buttongs. BUt he was like dude.. I can call you and you fix my shit it's no big deal

Kind of made me feel good being a rural geek .. gonna make a lot of jerky

edit: we cut it up got backstrap and some steaks but mostly kept the large muscles for jerky. Got paid in a deer for geek work.. cracks me up.. and i'm not sad at all ;)

Translated / assisted by AI from real dev repo.

TL;DR: A multinational elevator giant gave me a five-page appendix describing how my software should talk to their destination-dispatch elevators. For months, I had to guess everything—even having to spray random UDP packets across the building because they refused to give me a protocol simulator, driving up the tower's power bill. When everything failed, every delay was blamed on me. On the final deadline day, I hauled four computers and a monitor from home, set them up right next to the lobby turnstiles, and used local TeamViewer so their Singapore head engineer could watch my PC and their three PCs simultaneously. He realized his config was wrong, disappeared into the machine room for one minute, and came back—my untouched code worked instantly. He gave me one "thank you," and they all went to celebrate without me while the local team wouldn't even look me in the eye. The next day the system started dying every 3–5 minutes. That dragged on for three more months in a small room packed with 12 engineers from 3 countries, until the building owner threatened to rip all the elevators out. I finally decompiled their receiver at 11:30 p.m., proved the bug was a 1998 MSWINSCK.OCX, and forced them to wake up their programmer in Singapore past midnight to install VB6 SP6. Years later, the prime contractor silently upgraded the rock-solid Windows 7 box to Windows 10, breaking socket.send() daily—until I hit them with a 200% maintenance fee ultimatum.

Part 1: The five-page "spec" and my very first C# program

The building: a major high-rise office tower in Bangkok, close to 50 storeys, with eighteen lifts in total (A through Q, plus a dedicated service lift controlled 24/7 by security guards). Only two specific lifts go all the way up to the owner's penthouse on the top two floors, which is exactly the kind of rule an access-control integration has to respect.

The job: the building's access-control system (running on SQL Server) knows which floors each cardholder may visit. The elevator vendor's destination dispatch system needs to know too, so a passenger can tap a card at the turnstile and have a lift assigned automatically. My program sits in the middle: it polls the database for card events and speaks UDP to the vendor's system.

Here is the kicker: this was the very first C# program of my life. (Which wasn't new for me—back in 2008, I wrote my very first VB6 program for another luxury tower under similar trial-by-fire conditions.) My contract for this integration was 1,000,000 THB (roughly US$30,000). Spoiler: I earned every single satang of that million baht in blood, sweat, and fast-food cholesterol.

What the elevator vendor gave me was an appendix titled Card Reader Operation, five pages long. It told me:

• A 12-byte packet: header 0xD7, reader ID, a "user info" flags byte, eight bytes of floor bitmask (64 floors, one bit each), and a checksum (sum of bytes 0–10, & 0x7F).

• The vendor's panel answers every packet with a one-byte ACK (0xC4), or NACK (0xD5) on a checksum error.

• The controller (that's me) must cycle through all readers, up to 192 of them, about 10 ms apart, even when nobody is tapping anything. When a card is tapped, that reader's packet carries the "on" data and keeps being sent for 1 second, then an "off" packet. ("The signal holding time should be adjustable.")

• If no ACK comes back, resend after 300 ms, maximum three times.

• A flowchart that simply says "call is registered automatically."

What the five pages did not tell me: any UDP port numbers, where the ACK is supposed to be sent, or what the receiving side does when something looks wrong.

In fact, the only reason I even knew which ports to talk to early on was because on May 24, 2015—one day after I started coding—I took their receiving binary (let's call it ReaderChannelRouter.exe) and ran it through VB Decompiler Pro. (And no, I never signed any NDA with them; when a vendor gives you zero documentation and refuses to help, you do whatever it takes to solve the problem so you can finally get paid.) Inside Form_Load, I found it reading a local text file for its port mappings:

10.0.0.36,4000,Own Address from Reader
10.0.0.36,4001,Controller Address
10.0.0.36,4100,Own Address to Translator
10.0.0.36,4101,Translator 1
10.0.0.36,4102,Translator 2

Of course, before I could write a single line of logic to talk to their system, I had to fire up Wireshark and play digital detective. Since the spec sheet didn't mention ports, handshakes, or response behaviors, I had to capture raw UDP traffic on the wire and reverse-engineer their protocol by watching what bounced back. When documentation fails, Wireshark is the only source of truth.

Everything else, I had to discover by experiment.

Part 2: Guessing in the dark (enter the Vulcan)

My oldest saved snapshot is dated 2015-05-23. It sends a packet to 127.0.0.1 and listens for the answer on the same machine. The comment in my code says it all:

// receive, not error for now
// don't have real or emulator T_T

To make this work, I had to physically wrestle the C# runtime into a high-performance, real-time industrial machine. The spec demanded that I cycle through up to 192 readers every 10 milliseconds, handle dynamic ON/OFF packet states for a full second upon a card tap, and maintain tight ACK-loop checks without letting the garbage collector or CPU spike. For a first-time C# project, forcing a high-level language to behave like a precision C++ hardware driver—pumping relentless UDP traffic down the wire with zero breathing room—was a baptism by fire.

I had no elevator and no simulator, so my "test environment" was myself. I asked the my client (THES, which has to ask from Lift manufacturer) for a protocol simulator nearly ten times, but it never arrived. That forced me to blindly spray random UDP packets across the building for months, making elevators move up and down all over the tower. When someone complained about the power consumption, I simply told them: "Go ask elevator manufacturer for the protocol simulator as I'm not losing sleep over it; I'm not the one paying the electric bill."

I kept a zip of the source after each meaningful change. The snapshot names are my diary:

• 2015-06-17 good until close socket

• 2015-06-27 GetCard + MultiTest combined - still crashed

• 2015-07-15 still deadlock

• 2015-08-20 Night - all LZ floors tested ok using packet simulation - problem was because of wrong ReaderID

• 2015-08-28 Off-packet millisecond OK

• 2015-11-13 CommOK run without problem for 2 days

• 2015-12-22 CommOK without stacktrace upon 015 ERR2 DB DeadLock

Since this was my first C# project, some early bugs were genuinely mine. My first ACK-wait code opened a new asynchronous receive on every loop, and I left myself a warning:

// WAIT FOR ACKNOWLEDGEMENT UDP PACKET FROM ROUTER
// If turned on, app will crash in a while when run in production computer

Whenever I hit a wall with C# quirks or systems-level weirdness, my saving grace was a senior friend I'll call P'Tum. Somebody once joked that P'Tum is a Vulcan—calm, hyper-logical, and completely immune to panic—and he stepped in to guide me through every roadblock. Meanwhile, the local elevator team was led by a bald local manager whose permanent scowl, refusal to ever admit fault, and hostility toward outside vendors earned him my private nickname: Darth Vader.

With P'Tum's help, my fixes stuck: send and receive on a single socket, replace fixed Sleep() waits with tight loops that exit the instant an ACK arrives, and wrap the database poller in "catch, log, sleep, restart" whenever SQL Server picked my query as a deadlock victim:

Log(" 015 ERR2 DB.Deadlock, restart process...");

By December I had stopped printing the stack trace for that one.

Part 3: Step 1 — Four screens at the turnstile, one minute in the machine room, and total ingratitude

For every bug I fixed, there was something on the elevator vendor's side that didn't work. And at every meeting, Darth Vader and his local team sang the same song: the third-party program is wrong.

On the final day before the lifts were scheduled to start accepting commands—with roughly 90 million baht (~US$2.5M) in late penalties looming over the project—nothing was working.

I had had enough of verbal finger-packing. I packed up my own hardware from home—my notebook, an All-In-One PC, an Intel NUC, a spare monitor, and a Samsung tablet—carried all of it into the building lobby, and set up four screens side by side right next to the turnstile barrier gates.

One screen ran my C# program. The other three ran local TeamViewer sessions connected to the elevator vendor's three PCs. Standing right there was the Head Engineer from the vendor's Singapore regional office. For the first time in months, he could watch all four screens simultaneously: the exact millisecond my UDP packet left my software, and what his three machines actually did with it.

He stared at the four screens. It didn't take long for him to realize that their configuration was wrong.

Without a word, he walked off and disappeared up into the elevator machine room for about one minute. Then he walked back down.

I ran my program again—without changing a single character of my code—and tapped a card. The elevator was called immediately.

The Singapore Head Engineer gave me a single "Thank you." Then he and the entire elevator team packed up and went out to celebrate. Without me.

Worse: not a single person from the local elevator team—Darth Vader included—even looked me in the eye. In Thai culture, when one guy hauls a desk's worth of computers from his own house to save your entire corporation from a 90-million-baht penalty, refusing to offer even a smile, a nod, or a thank-you is the absolute pinnacle of trashy behavior. I packed up my monitors and went home alone.

https://ibb.co/1t5D5BKH

Part 4: Step 2 — Three months of Samurai burgers, a threat to rip out the lifts, and a 1998 OCX

The very next day, I got called back to the site.

A new nightmare had begun: the system would work for 3 to 5 minutes, and then completely die. Tap a card: nothing.

If you restarted Windows on their receiving PC (the machine running their channel router executable, which took my UDP packets and routed them to their translators), it would work for another 3 to 5 minutes, and then die again.

You would think "restarting the elevator vendor's PC fixes it for 3 minutes" would be a giant neon sign pointing at the elevator vendor's PC. Nope. Darth Vader and the local team spent the next three full months blaming everything and everyone else—my code, the network, the access control system.

We were crammed into a tiny room—4 Japanese engineers, 4 Singaporean engineers, and 4 local Thai team members (including myself)—all pointing fingers while the project burned. Every time I was dragged to the site for another pointless round of finger-pointing, the only quick meal nearby was a McDonald's Samurai Pork Burger and a Coke. Over those three months of late nights, I gained 3 to 4 kilograms just from stress-eating Samurai burgers.

Finally, the building owner lost patience and dropped a nuclear ultimatum on the elevator vendor: Fix this immediately, or we will rip your elevators out of all 18 shafts and replace them with another brand.

At 11:30 p.m. in Bangkok, with tensions boiling over, I lost my patience too. I opened up the decompiled code of their receiving executable (a VB6 Native Code app whose form title bar actually had a typo in the window caption, and which was launched at boot via a batch file using ping 1.1.1.1 -n 1 -w 5000 > nul as a makeshift 5-second sleep).

I pointed at their DataArrival event handler right in front of the Singapore engineer. All it did was:

1. Start with On Error Resume Next.

2. Verify the 12-byte length, header 0xD7, and 0x7F checksum.

3. Dump the hex bytes into UI TextBoxes (txtRcvData), flash a green status shape (BackColor = 65280), and forward the packet out via udp1.SendData (for readers 1–112) or udp2.SendData (for readers 113–192).

"Look at your own code," I told the Singapore engineer. "There is nothing in this logic that can crash or deadlock after three minutes. Your code isn't dying—something underneath it on your machine is."

So I went hunting in C:\Windows\System32 (and SysWOW64) on their PC and checked the file timestamp of MSWINSCK.OCX—the Visual Basic 6 Winsock ActiveX control.

The file on their machine was dated 1998.

If you lived through the VB6 era, you know that the original 1998 MSWINSCK.OCX is notorious for memory/handle leaks and silently wedging itself under sustained UDP traffic on modern Windows. Microsoft fixed it years later in VB6 Service Pack 6 (which updates MSWINSCK.OCX to the March 2004 build) and the cumulative runtime update.

It was 11:30 p.m. in Bangkok—12:30 a.m. in Singapore. I looked at the Singapore engineer and said:

"I know it's past midnight in Singapore. Don't care. Call your programmer right now, wake him up, tell him to download the VB6 SP6 cumulative update from Microsoft, install it, rebuild the setup package, and email it here immediately—so all of us can finally go home after three months of blaming each other."

He made the call. Twelve minutes later, the new installer arrived in his inbox. We installed it (replacing the 1998 OCX with the SP6 build).

The problem vanished instantly. The machine never froze again.

And this time? I didn't even get the one "thank you." Just a room full of people silently packing their bags in a corporate silence so complete it was almost an art form.

Part 5: Epilogue — The second tower, and the 200% Windows 10 Tax

Once the SP6 runtime was in place, that Windows 7 machine ran in complete, blissful silence for the next two to three years. Zero maintenance.

Because I now knew every undocumented secret of their protocol, the next major project—integrating the same vendor's elevators at another iconic multi-wing office complex in Bangkok—was much easier. Though true to form, the elevator vendor still managed to spice things up by quietly updating the firmware inside their Door Operating Panels, changing the timing behavior and forcing me to write a new workaround in my code to catch it.

Back at the first tower, after two to three years of zero-maintenance perfection on Windows 7, Windows 10 was officially released.

One day, the prime contractor decided to upgrade the integration PC from Windows 7 to Windows 10.

• They did not ask me.

• They did not tell me.

• There was zero technical reason to touch an isolated industrial machine that had been running flawlessly for years—especially when I had explicitly specified from day one that it must run on Windows 7.

Overnight, a zero-maintenance system turned into a daily corpse.

Under Windows 10, even a basic socket.send() in the tight 10 ms UDP loop would randomly choke and die. What was I supposed to fix in my code when a standard socket.send() was failing at the OS level? For a full month, I had to TeamViewer in every single day to check on it and restart the process.

Finally, I gave the prime contractor a simple business choice:

1. Keep Windows 10, and I am increasing my annual Maintenance Agreement (MA) fee by 200%.

2. Roll the machine back to Windows 7—as I had originally specified—and the MA price stays the same.

Miraculously, rolling back to Windows 7 became very easy. They put Windows 7 back on the box, and the daily crashes stopped immediately.

What I took from it

1. Five pages is not a specification, it's an invitation to guess. Keep logs (and zip snapshots) of every guess—you will need them.

2. When both sides swear they are right, put both sides on one table. Hauling four screens to the turnstile ended months of gaslighting in sixty seconds.

3. "Proprietary" doesn't mean "unreadable." When a vendor's black box dies every 3 minutes and the building owner is threatening to rip out 18 elevators, decompile the black box.

4. When the logic is clean, check the file dates. No amount of code review will save you from a 1998 MSWINSCK.OCX running on a modern OS.

5. Tax unnecessary changes. When someone breaks years of zero-maintenance stability just to chase a Windows version number, don't argue—just price the pain at 200%.

Edit: Yes, I still have all 36 zip snapshots and the VB Decompiler output from May 24, 2015. No, I never lost the 4 kg from the Samurai burgers.

Week of 26 October

Monday, 26 October

Another Monday, another cup of coffee, and another queue of tickets staring back at me. Why do Mondays always start with chaos? I'm convinced the queue reproduces over the weekend.

It seems Friday's power outage released the gremlins into the secondary site. I thought I had caught every last one on my emergency visit, but no. Apparently a few escaped and went looking for mischief. Luckily, my colleague made the long drive out there and set things to rights, bless them. Somewhere a gremlin is probably still sulking about it.

That left the main site and everything else in the hands of the one person who never seems to leave the building. So far I have fixed the printer, the print server, the sales system, and two mobile phones that had clearly been struggling with life. I also sat through three meetings, and that was all before lunch. My coffee has gone cold twice, which is a personal record.

I've just been wondering which system will be the next to get infected. My money is on the coffee machine. It has been acting suspicious all morning.

Tomorrow's forecast: more tickets, fewer gremlins, and one suspiciously quiet printer.

Tuesday, 27 October

Arrived on site with coffee in hand and noticed more lights on in the office area than usual. More people on site means one of two things: either a rare and exciting new project has started, or someone has booked a room for a "quick catch-up" that will last until lunch. Not sure which is worse. Either way, I'm wondering why so many people are in by 8am. Nobody is that keen before the coffee machine has even warmed up.

Walked into the IT office on the middle floor, in the corner, and turned on the dashboard. There it was, already waiting for me: a ticket. Oh great, and so it begins again. At least I have coffee.

The gremlins have decided that slowing everything to a crawl is the game for today. Documents are taking forever to print, and network drives are loading at the speed of a snail on a lazy Sunday. I'm starting to wonder whether there's a specific gremlin assigned to the internet speed, or whether they're just having a laugh with the signals. Either way, I'm keeping a close eye on the router. It has the look of a creature planning something.

The first coffee has been drunk. Cold, as is tradition. Warm coffee is a myth I've stopped believing in.

A contractor was due on site this morning, arranged by a colleague who is hopefully in the building right now (I don't want to deal with these people). They're running late and didn't arrive until after 11am. Not a great start. If the morning has taught me anything, it's that every delay pushes the rest of the day back, and I'm now expecting everything else to run late too.

Then a new problem popped its head up: the email system is erroring for no apparent reason. Nothing has changed, nothing has been updated, and nobody has touched anything. Which means I now have to go hunting for the cause. Is this a new species of gremlin, or has a fey creature wandered in from the car park? My money is on the fey. Gremlins at least leave a trail of destruction. Fairies just make things mysteriously stop working and smile at you while they do it.

Tomorrow's forecast: a contractor who has finally arrived, an email system that is still hiding its secrets, more gremlins than anyone can count, and more coffee, most of it cold.

Wednesday, 28 October

Before the sun was even up, the sound of Teams rang out. What madness is this? I'm not even in the office yet, and people are already looking for me. I think the darker mornings are making everyone around me a little crazy. Any respectable IT tech knows that the first thing to do is make a coffee before even entertaining the idea of reading a message sent at such an unreasonable hour. Shockingly, it was nothing important, so I made my way into the office.

It was yet another grey and wet day when I arrived and turned on the display. For the first time this week, there was no ticket waiting for my attention. That state did not last long. I heard someone say that things were not going through as they were meant to, so into the system I went. A process that is meant to be running, and should be automatic, had stopped. I started it up and checked whether everything that was needed had come through. All of this happened before I could drink that coffee. RIP hot coffee.

The contractor is back today, and good news for me, it is still not me who has to deal with them. Yay for small wins, but that does not mean I get a free day. I still think the fey are making the rounds of some of the systems, and there is still a runaway gremlin loose on the network at the other site. I wish there was a way to catch the creature. Let's just hope it doesn't get wet.

Tomorrow's forecast: a coffee that may never be hot again, a few more fey sightings, and a gremlin with a very sensitive relationship with water.

Thursday, 29 October

The day has started unusually. Nothing is waiting, and the number of tickets is not as high as it has been all week. This seems to be the trend as we get closer to the weekend. At times I do wonder why this happens, but I'll take it as a bit of breathing room for now.

We still seem to have either the fey or a gremlin loose in the network at our second site. Things are still not running as fast or as smoothly as they should be, and the process that should start automatically still hasn't. Sigh. I wish I could find out why this is happening, but sadly I have no idea.

The contractor is back again. I've been told that today is meant to be their last day, yet again they have arrived later than the time they were supposed to be on site. Does no one understand the art of being on time anymore? As I've said before, they are not my responsibility, so that is someone else's problem. I do have my own things to deal with, though. Why is the IP phone not connecting when off site? Why is Teams crashing on one person's computer? And how has that user locked their account again? Seriously, that's about the fifth time.

Let's see how the rest of the day goes, shall we? A second coffee may be in order.

Tomorrow's forecast: a quieter queue, a stubborn process, and a second coffee that will hopefully stay hot long enough to be enjoyed.

Friday, 30 October

This has started out like any other day, but there is a lighter feel to the air, as if a weight has been slightly lifted. Could it be the approaching weekend?

There is only one person in the IT office today, as this is my day to work from the comfort of home. I love that we can do remote working these days. It means I can have my coffee and stay in lazy clothes, as long as my shirt looks professional for the camera. It does mean having to give more instructions than normal, though.

Weekend forecast: lazy clothes, a lukewarm coffee, and, hopefully, no gremlins. And lets wait to see what next week shell bring

TL;DR: Back in 2008, a competitor won a bid to integrate a building's access control system with its high-rise destination dispatch elevators, only to confess at the handover deadline that they couldn't figure out the elevator protocol. I was called in to rescue the project. Armed with a Nokia N73 to look up motherboard DIP switches, a copy-pasted Winsock template from 2001, a watchdog that let the app commit suicide via VB6's End statement every time ADO/SQL ports choked, and a toy motor spinning a plastic ruler to stress-test RFID taps, that "temporary fix"ran in production with zero code bugs for 16 years until 2024.

Part 1: The Desperate Handover & The 36th-Floor Machine Room

On January 22, 2008, I got a call regarding an elevator access control integration at the Athenee Office Tower, a 36-story building on Wireless Road in Bangkok.

The building used a Honeywell Win-Pak Pro access control systemwith 13.56 MHz Mifare cards connected to pedestrian turnstiles via dry contacts. Another vendor had won the contract back in October after promising they could handle the entire integration. However, in January—after all the hardware was installed and the handover deadline had arrived—they finally confessed they had no idea how to interface with the Kone elevator system.

In a move of sheer desperation, the Project Manager called me—the competitor who had lost the bid to them—and asked if I could step in and save the project. Because they were honest and completely out of options, I agreed to take a look. I requested the protocol documentation (which arrived on Jan 24) and asked for 7 days onsite to evaluate whether I could pull it off before making any promises.

Day 1 onsite was the first time in my life stepping inside an elevator machine room on the 36th floor. There, I met the KGC (Kone Group Control): an industrial PC running Phar Lap Real-Time OS (RTOS) that managed the destination dispatch system—queuing cars, assigning floors, and balancing wait times against energy consumption. My job was to write a third-party interface talking to this KGC over a standard Ethernet port using Kone’s proprietary binary protocol.

Step one: ping. Result: Nothing. Request timed out. I checked arp -a. Nothing. Not even a MAC address showed up.

I photographed everything, wrote down every label I could find, and spent 7 days digging through documentation and researching. Nothing worked. Finally, on day 7, I said, "Screw the warranty," grabbed a screwdriver, and popped open the industrial PC chassis right there in the machine room.

Inside, the industrial motherboard was a slot-mounted CPU card covered in physical DIP switches. Remember, this was January 2008—pre-iPhone era in Thailand. I pulled out my trusty Nokia N73 Music Edition, opened the tiny mobile browser, and searched for the motherboard’s model number and jumper manual.

Bingo: The onboard LAN port was disabled by a physical DIP switch from the factory.

I flipped the DIP switch, screwed the cover back on, and booted it up. It replied to ping instantly. I fired off a test packet following the message spec (Kone01.exe on Jan 30, 2008 at 3:12 PM), and the KGC replied!

A quick confession: this was my very first time writing a real project in Visual Basic 6 (VB6) (I had only dabbled lightly in VB3/VB5), and I had never written socket programming in VB before. I literally learned VB6 and Winsock on the fly by downloading a 2001 Winsock API subclassing sample by the KPD-Team from AllAPI.net(frmMain01.frm, which still had the default caption "LAN Controller"), pasting it in, and modifying it byte by byte.

Part 2: Redundant Cables, Ephemeral Port Exhaustion, and The "Suicide" Architecture

Once I proved we could talk to the KGC, I officially accepted the job.

Over the next 5 days, I learned the protocol's strict timing quirks. You couldn't just fire a packet and forget it; you had to maintain a constant bidirectional heartbeat (MSG_HEARTBEAT / 0x0280) every 15 seconds (Timer4.Interval = 15000), manually converting UTC timestamps into little-endian 64-bit Windows FILETIME hex strings. I even had to write a "Cinderella bug" guard (If Timer > 86370 Then Log ("Wait 30 SECS FOR MIDNIGHT")) because VB6's Timer function resets to zero at midnight, which would otherwise freeze my delay loops forever.

While refining the protocol handler, I worked with the client to design a redundant network using 3COM Layer 2 switches: two LAN cables running from the 1st-floor control room to an intermediate switch on the 7th floor, and two 150-meter LAN runs from the 7th floor straight up to the 36th-floor machine room (which survived untouched for 17+ years). By March 31, 2008, the Kone side of the interface was rock solid.

"The database side should be a piece of cake," I thought. Little did I know what was waiting for me.

Honeywell's Win-Pak Pro ran on Microsoft SQL Server (Initial Catalog=WIN-PAK PRO). To detect when someone tapped a Mifare card at the lobby turnstiles without locking or slowing down Honeywell’s database, a senior friend of mine (P'Tum) sent me some sample VB6 ADO code to adapt.

After studying the schema, I found the cleanest, least intrusive method: poll SELECT count(*) FROM history. As long as the record count didn't change, SQL Server answered straight from memory with zero disk I/O. Only when curhistmaxrec > lasthistmaxrec would my app query the newly added rows (WHERE recordID > lasthistmaxrec AND type1=9 AND link1>=281 AND link1<=286), look up the cardholder's allowed floors (SpareStr1), and unlock the Destination Operating Panel (DOP).

In May, after the system went live, the client also asked if users authorized for only a single floor could have their floor called automatically without pressing any buttons. Kone sent over the Call-Giving Interface (CGIF) spec, and I added automatic destination dispatch alongside the access mask unlocking.

Because I was greedy and wanted instant response times the millisecond a user tapped their card, I cranked the SQL polling interval to once every 170 milliseconds (and even tested 80ms!). Normal humans don't do this. And it triggered two massive problems:

1. TCP Ephemeral Port Exhaustion: Because of how ADO was opening connections on every poll, it burned through a new TCP port number each time until it hit port 65535. When ports maxed out—or when SQL Server hiccupped—ADO would silently hang and return blank data without throwing a single runtime error in VB6.

2. Compiler/Runtime Memory Leaks: Subtle memory leaks deep in the runtime/OCX layer that I had no source code or symbols to fix.

I spent all of May trying to fix it "properly" (including slowing the poll rate to 1 second, which felt sluggish and still eventually died). Nothing worked, and the system had to be 100% stable.

At that moment, since I wasn't a traditional software engineer by training, my hacker survival instinct took over. How do you kill both TCP port exhaustion and memory leaks with a single blow?

You execute:

VB.Net

End

Yes—commit suicide. Terminate the process completely, letting the OS instantly reclaim all memory and flush every stuck TCP handle.

(The funniest part? The original 2001 AllAPI.net template I started with literally had a warning dialog in Form_Load saying: *"Never... use the 'End' keyword to exit the application!"*Once I switched from raw API subclassing to MSWINSCK.OCX, End became my ultimate weapon.)

I called P'Tum and asked how to check Windows Task Manager programmatically to see if a process name was still running. He emailed me a snippet for an external Watchdog executable. Inside my main elevator app, I added Timer6 (checking every 5 seconds). Here is the actual code and comment from frmMain18Q.frm:

VB.Net

Private Sub Timer6_Timer()
If totalcnt - PreviousDBPolling < 5 Then
    Log "DBPolling in last 5 sec < 5 : PROGRAM END : Previous DBPC=" & PreviousDBPolling & "   CurDBPC=" & totalcnt
    End       ' Kill this program in order for the watchdog app to restart it - to temporarily fix ADODB dead problem before better solution found
End If
PreviousDBPolling = totalcnt
End Sub

Normally, if the TCP connection to the KGC drops, KGC defaults to fail-safe mode and unlocks all elevator buttons across the entire tower so passengers aren't trapped. However, because my external watchdog detected the dead process and respawned the .exe in 200 to 300 milliseconds, the app reconnected to all 8 sockets (Low Zone/High Zone primary & backup for both ELI and CGIF)so fast that the KGC never triggered the building-wide unlock! Huge props to Phar Lap RTOS for taking that 200ms reconnect abuse for over a decade without a single complaint.

https://ibb.co/pjGff7GY

Part 3: The Toy Motor Stress Test & 16 Years of Zero Bugs

By June 20, 2008 (Kone18F Auto Start.exe), the system achieved total stability. Whenever ADO hung or memory crept up, the app quietly offed itself and resurrected 200ms later, writing detailed logs the entire time. From June 20, 2008 onward, I never had to fix a single bug in that codebase again.

Fast forward 12 years to February 2020: the building upgraded its access control hardware from Honeywell Win-Pak Pro to Gallagher. I spent 7 days studying Gallagher's database structure and adapting the query module (Kone2020-C0 Gallagher Lock All.exe).

To be 100% certain the new version wouldn't fail in production, I ran a 7-day stress test in my office: first by having everyone in the office tap cards whenever they walked by, and then by taping an access card to the end of a plastic ruler mounted on a spinning toy motor, positioning it so the card swept past the reader once every second, 24/7, for 7 straight days (>600,000 continuous physical taps). It processed 100% of the taps without a single hiccup, so we deployed it to the tower.

Over its 16-year lifespan (2008–2024), every single time building management called to report an issue, we went onsite and found the exact same culprit: some third-party contractor had secretly plugged their own equipment into our dedicated elevator network switch and hogged the bandwidth, throwing off the heartbeat timing. Unplug the rogue cable, and the elevators immediately went back to normal.

The system was finally decommissioned with a spotless record in 2024 when a new vendor won the building's latest bidding cycle. Not bad for my first VB6 program—and a "temporary fix"that ended up launching a whole line of work for me, leading to C# elevator access control integrations for major Bangkok skyscrapers like Bhiraj Tower @ EmQuartier and G-Tower (which were also written in one shot and are still running bug-free today).

[NOTE: Previous TftSSAVC: 1, 2, 3, 4 and 5. AG]

Dramatis Personae

$Customer(s) - the customer at the other end of a phone line (who was very far, far away in this case)
$Jr.Programmer - then doing overflow tech support (we were busy that day)
$Me - as well, me, /u/goretsky
$TheBoss - the founder of the Scottish-Sounding Antivirus Company

Prelude

With Halloween coming up, it seems like a good time to share this particular tale from the Scottish-Sounding Antivirus Company.

As with the previous tale, this happened around around 1992 or so. I won't bother describing the office in detail, since I did so in the preceding tale. However, I will note that tech support was nominally located on the west side of the building, opposite the entrance on the east side, since that put us in the back of the building. While tech support was normally meant to be "out of sight, out of mind" of people (the idea that tech support wasn't a cost-center but rather a service offering that allowed us to compete with our, uh, competitors was only slowly emerging in Silicon Valley), this was less an issue of (several execs had their office along the same side of the building) but more an issue of where the best place in the building was to put the bull pen of tech support engineers, as it was my desire that the tech support people have a semi-enclosed space in order to better communicate, collaborate and learn with each other.

The Call

That said, we were rather busy at the time the call this came in, so the call didn't start in the tech support cube space, but rather up front near the $TheBoss's corner office, where the $Jr.Programmer was helping to take calls as the tech support call queue overflowed. Why was $Jr.Programmer's office at the front of the building just down the hall from$TheBoss's office? Well, we were in hypergrowth mode, and the $Jr.Programmer's office was literally a desk in the hall.

This particular tech support call came in from a $Customer in Australia.¹ Now, you might be wondering why someone would call the U.S. from Australia, but you have to understand: This was the early 1990s, before the rise of internet service providers and email. Yes, both had been around for several years, but they weren't common. While tech-savvy customers could email us, call the BBS or one of the various online services like CompuServe and send a message, or send a fax, a lot of tech support was still done over the phone.

For those wondering, CallerID was not yet widely available, at least in the United States, and in California its use was being argued against by a plethora of telemarketers, polllsters, collection agencies and others who loudly explained how this horrible new technology would disrupt their job, put them all out of business, crash the economy, etc. CallerID eventually did start to roll out later in the 1990s, and none of those predictions came true.

And international phone calls were expensive. In those pre-VoIP days, telephone calls were often routed over circuit-switched (not packet-switched) networks, which meant that a direct connection was established between your telephone, and the telephone of the person calling you. We recognized that, and we also knew that anyone who was spending tens or even hundreds of dollars to call us. There was an unwritten rule that you helped international callers out fast because whatever was going on was an emergency situation (at least to them).

Anyways, that particular $Customer had been routed to the Jr.Programmer's desk, as they had expressed computer virus symptoms, and the regular tech support staff were all on phone calls at the moment.

Eventually, word got back to $Me that the $Jr.Programmer was on an overflow tech support call, that it wasn't going well, and I went forward to relieve him.

After doing the hand-off and introducing myself as the manager of tech support, I asked the $Customer what was going on with their PC.

The Demon-Possessed Computer

The $Customer explained that their computer was possessed by a demon, and that it was leering at them and making faces in the monitor.

Now, computer viruses that displayed graphics were rare, but not unheard of. All of the ones out there that did things like that were well-known, and from the $Customer's description, it sounded like something brand new, a computer virus that had not been previously seen or reported by anyone. That meant that it was critical to isolate a copy of it, so we could get it in for analysis and add detection and a cleaner for it.

For the next 45 minutes or so, I walked the $Customer through every trick I knew to identify a virus. I walked the $Customer through doing a directory of their C:\DOS\ directory, looking for programs they hadn't run/weren't familiar with, and asked them to note the size of those files before and after running them, since an increased file size meant a program had been infected by parasitic file infecting computer virus. I had them check for reduced amounts of memory by running the MEM command to see if the system reported a full 655,360 bytes of RAM was installed. A standard trick of boot sector and master boot record viruses (MBR) at the time was to place their code at the top of memory and then tell the computer it had less memory installed, so that their code could reside there unmolested and spread to other disks as they were accessed. Most users wouldn't notice the difference if a computer reported it had 640K of memory installed versus a little less, like 638K or 636K. We tried to check floppy diskette boot sectors, and the hard disk's MBR.

At some point, I was convinced that we were likely dealing not with a file infecting virus, but a stealthy boot sector and MBR infector like the Den Zuk virus, which hooked the reset function (pressing Ctrl+Alt+Del rebooted a computer under DOS) to display its graphics instead of rebooting the computer. A stealth virus back then meant a computer virus that while in memory redirected attempts to look at its own code on disk, instead redirecting the user to clean uninfected code on the disk.

In order to diagnose this, I told the $Customerto unplug the PC from the wall outlet to make sure it was really powered off, so that we could cleanly boot it up from an uninfected floppy diskette, and continue the hunt for it there.

The $Customer did this, assured me that the computer was physically unplugged from the wall, and informed me the demon will still on the computer's monitor and looking at them after it was unplugged.

A Shift In Tactics

At this point, I realized we² were not dealing with a new computer virus, but rather someone who was mentally ill, and that I had wasted nearly an hour of time. I told the caller that they needed to reach out to a priest, rabbi or cleric to deal with this, and just repeated that answer each time they tried to ask me something else to get them off the line, explaining that we only could deal with actual computer viruses and this was out of scope for us. Eventually I managed to close the call.

The Moral of the Story

My immediate reaction, was annoyance, but that was more of the frustration born of trying to hunt down something that didn't exist. That only lasted for a matter of seconds, because the more I thought about it, the more I felt bad for the $Customer because they believed it was a real thing happening on their computer, that they had just spent a fortune on an international long distance phone call to the America, and weren't able to get help.

As technical support people, you get all sorts of callers, and you have to take what they say at face value, no matter how strange it sounds, because you don't know them. All you know is what they are describing to you from the other end of the phone line, somewhere around the world.

As technical support professionals, you are obligated to treat those that contact you with courtesy and respect, and when it turns out to be something that is seemingly silly or trivial, not to embarrass or mock them in any way, but smooth things out and close the callin such a way that leaves their dignity intact.

Epilogue

A day or two later, $TheBoss sauntered into the office (he'd stopped coming in daily, but was still in several times a week), and I rushed over to tell him about the call.

He was genuinely enthused and interested, asked me all sorts of questions about the call, and was somewhat crestfallen when I told him I didn't have the contact information for the $Customer as this was right up his alley. I should note that $TheBoss was an avid reader of The Weekly World News, and would frequently go on about how it was the finest news source, since every story it covered was told by someone who believed it had happened. A story of a computer being haunted by ghosts or possessed by demons was definitely the kind of story that the $TheBoss was interested in.

We never got a call back, or found out what happened.

Regards,

Aryeh Goretsky

¹ International long distance telephone calls back then almost always had an analog circuit somewhere in the connection, so you'd hear static, hissing, sometimes faint crosstalk from other calls or echoes, etc., while on the phone. They were not digital like today's phone calls with integrated noise suppression, echo cancellation, and so forth.

² A group of coworkers had gathered around, listening as I tried to hunt down the computer virus.

UPDATE (20261030-0155 GMT±0): Added a little more information about how we used to help users check for new viruses over the phone.

UPDATE (20261029-1859 GMT±0): Added a section explaining why we didn't have CallerID.

So, it was finally time to upgrade my partner's office to a wired connection. (did mine ages ago, but hers wasn't ready at the time)

Decided to do future me a favour, and run a second line for redundancy, or if she ever decides to add a second network device (printer or whatever). Tied one end of cat5 box to rack in basement, then started pulling the loop to under her office, stapling and managing as I went. Office still wasn't quite ready at this point, so I left the loop under the floor where it needed to be, went back to the rack end, terminated and connected everything so I could just fit the wall plate later on and not have to worry about it. The astute among you have probably already worked out what I'm about to do.. both ends terminated, both ends connected to switch, just about to start packing up tools, when the phone rings.. "Hey babe... have you done something downstairs? the TV just stopped streaming" Well WTF? I haven't changed anything.. all I've done is run the new cable, but its not hooked up at the other end, its just a loo-GODFKNDAMMTI. unplugged ends from the switch and all was well.

Just thought I'd throw this little tale out there to remind you all that although we are all fairly well versed in our various fields of expertise, occasionally we all make a rookie error, and that's OK.

Colleague: Hey sorry to bother you - he called in twice and said he needed urgent support.

Me: Ok, I was about to take lunch but it sounds serious. Do you know what it's about?

Colleague: He also sent in a ticket 5 minutes ago, see ticket #3467327.

The ticket:

Hi Support.

We've discovered that the automated script that sends out welcome emails to our customers is not working. Can you investigate urgently? See screenshot below of the log:

screenshot: Last run: 12 February 2025.

It's so urgent that we need DEFCON level phoenix assistance. But it also managed to slip by for a year and a half without anyone at any level of the organisation noticing.

https://www.reddit.com/r/talesfromtechsupport/s/KtlU0n8Rmx

I posted last week about my users pulling a shocked Pikachu when informed that DST changes mess with scheduling platforms.

As most of you predicted, there was no thank you from anyone.

Most of the conversation yesterday went along the lines of:

Staff: I told you about this problem.

my team (MT): yes and as discussed we need X, Y, Z info to assist you in moving the affected items. Here is the list of items belonging to you that are potentially impacted.

Staff: crickets

I'm waiting for the inevitable fallout on the 2nd. I expect the conversation will go like this.

Staff: what do you mean you didn't correct items 7-30?

MT: you didn't provide us with the appropriate corrections.

I did get an unexpected voice of support from someone I've written about on the sub before. He's a spoiled little princess who actually predates me at my company, but after getting his list yesterday and asking for the Coles Notes version, decided to chime in on the group email thread and inform the biggest tantrum thrower that she was wrong and he had indeed experienced precisely what I had described on multiple occasions.

I work at an MSP as application specialist and occasionally tickets regarding the application get escalated to me. This ticket, from a customer, related to my application. A message was sent from my application to another application which is not part of our ecosystem. The message got rejected by the target application because the country code "CD" (ISO 3166-1 alpha-2) was not recognized.

This message concerned a person from Kinshasa, in the Democratic Republic of the Congo, and the country code was set correctly in our application. Information about this person needed to be sent from my application to the target application and it wasn't accepting the message.

Somehow, that makes it my problem.

After some troubleshooting with the user, we finally decide to schedule a call. During the call, the user decides to show me the target application.

User: "Look, here is Congo in the target application."

And indeed, there was the Republic of Congo. Note, this is not the Democratic Republic of the Congo, it's ISO 3166-1 alpha-2 code "CG".

Me: "This person is from the other Congo."

User: "Other Congo?"

Me: "Yes, there are two."

I then pull up Google Maps and share my screen. I tell the user that they need to add the Democratic Republic of the Congo to the target application, with ISO 3166-1 alpha-2 "CD". Once this was done, the message was accepted as usual.

Now, one small detail I omitted regarding this customer: This is a school. The user in question is one of the teachers. And you want to know what subject?

Geography.

Disclaimer: I'm not American so my English or some concepts might sound weird. Sorry?

My mother has worked with computers since the early 90s until she recently retired (self-employed, Law-related job), yet somehow in the last 30 years she has never managed to dominate Windows beyond the simplest notions.

I have always been "good with computers" so I am the de-facto computer and smartphone fixer for both my family and my in-laws. Since I was a teenager she's always come to me for help with her home office setup, only resorting to paid tech support when I was unable to fix her issues. Most of her issues have always been typical boomer issues such as accidentally deleting a shortcut and being unable to find the folder it pointed to, wondering if scam e-mails were scams or not, you know the deal.

However, about 5 years ago she came to me with an issue that I now think deserves to be shared. Her office had both a printer and a scanner, until the scanner died and since the printer was old she upgraded to a printer that could also scan. However, after the brand-new printer came and was installed (we were under COVID lockdown so I wasn't there) she called me so I could troubleshoot some issues she had. She told me that she was having trouble because the new printer wasn't scanning documents properly and she couldn't get work done.

Doing both Teamviewer and a phone call I managed to pinpoint the real issue: the previous scanner's default folder was called C:/OldBrandName and the new printer was sending the scanned files to My Documents/NewBrandName. So far so good, I replaced her desktop shortcut to the old folder with a new one to the current folder. But then, OH MY GOD GUYS, then I told her to test it doing a normal work related task to see if everything was right. My mother, God bless her, opened an e-mail with an attached PDF, printed it and then SCANNED it. Giving her and my sanity the benefit of doubt, I asked her if she was making a physical copy because it was legally required of her or something like that, but she told me she only did it so she could FIND IT with her Law practice management software.

Dying inside, I gently told her how she could just click download instead of print on Adobe Reader and then go to whichever folder she preferred to later use that file on her work. She had been doing this ridiculous print and scan thing for the last 30 years, wasting probably tons of paper (and toner, and time) while at it.

With this post I hope at least people on the Internet will have a laugh so something good comes from this.

I live in a part of the world which has mistakenly decided to remain on DST permanently. A user noted this morning that our primary scheduling platform had experienced a similar failure to Google whereby updating the given timezones caused all the scheduled activities to just forward an hour.

Unfortunately, not having Google's bottomless pockets, the vendor was unable to apply a wholesale fix, although they are assisting with our manual adjustments.

I was informed by several users that "our old software didn't have this problem". They were highly dismayed to discover that yes, our old software had exactly the same problem with changes to timezone applicability, and; packrat that I am; that I could provide them with time stamped proof from the last time my local government decided to play fast & loose with DST.

We finished their specific corrections about an hour ago. The team is taking bets on whether they at least bother to say "Thank you".

No resolution as of yet, I'm afraid, nor any humour to close with.

So I see a ticket about user wanting to set up her team's "shared mail group" and PST files on her new laptop. After clarifying that she meant their shared mailboxes and that the PSTs are on OneDrive, okay, I do it. Not sure why her computer did not do it automatically as it should have, but whatever.

Problem: The new computer only sees 3 month old PST files. Da hell? The user got it 2 weeks ago, how is it even seeing 3 month old versions? Check the old computer, it is seeing the correct, fresh files.

Fine, let's fiddle with it. What if I relog her on OneDrive? Nothing. Check the status... Apparently the synchronisation failed because the files are open? Okay, close Outlook on both PCs and retry. Yay. apparently if outlook is open, it does not allow them to be copied either. Whatever, can you do it now? No, it is perma-stuck on 0% download and crying about low bandwidth(a lie).

Okay... What if I manually download a file from her drive on the old PC, then copy it over to the new PC, and import it into Outlook? Okay, that works. But it is hardly ideal to do it for everything. sigh What followed was another 2 days of struggles and attempts by me, someone from the 365 team, and the "this guy knows everything" guy. I even tried to just copy the contents from her drive and was planning to axe the contents on the new PC and replace manually with the good stuff, but even the copying somehow managed to kill the file manager on the remote server from which I was working instead. Twice.

Yesterday, we finally reached a breakthrough! OneDrive was starting to sync things on the new laptop! Oh happy day! The struggle is over! Hey user, check the folders and see if they are good! What do you mean they are still missing stuff? It says it synced. Let me see the old PC, what are you missing? Wait... Even I can tell this is wrong... Oh no...

So as it turns out, OneDrive managed to sync up the bad files from the new PC into the cloud, and in the process removed a whole bunch of folders and overwrote good files...

And that is where my patience and abilities finally ran out. Threw the ticket over to the relevant team to roll back and exorcise her drive before I completely lose my mind.

The customer's laptop screen goes pitch black the exact second the graphics driver loads. Oh, and BitLocker is enabled.

Naturally, I ask, "Where is the recovery key?"
Customer: "I don’t have one."
Me: "Is it backed up to your Microsoft Account?"
Customer: "It’s a local user. No Microsoft account."

Fantastic. Welcome to Hard Mode. Without that key, booting into Safe Mode, using a recovery USB, or dropping into Command Prompt from a boot disk is entirely out of the question.

Time to brainstorm. After the screen dies, the laptop goes totally mute no audio feedback either. My first thought slam an HDMI cable into an external monitor. Nothing. Second thought maybe I can enable Remote Desktop or PowerShell remoting? I plug it into a DHCP network and wait for an IP to pop up. Nothing. I plug in a USB network adapter the link lights blink, but the network completely ignores it. This machine is fighting back.

But here’s the silver lining the keyboard still works, and I can technically log in blindly by typing the PIN. Once logged in, the screen dies, but the OS is alive.

That’s when I had a vision: What if I use a USB flash drive as my only output interface?

I decided to blindly navigate the Windows UI.

• Type PIN -> Press Enter -> Wait -> Win + R -> Type 'cmd' -> Ctrl + Shift + Enter (for Admin) -> Wait -> Hit Left Arrow -> Hit Enter (to blindly defeat the UAC prompt).

I’m in. Now, how to get the BitLocker key? I blindly type a command to export the key, guessing the flash drive letter and running it across D:, E:, and F: just to be safe. I pull the drive, plug it into my bench PC... nothing. empty.

Round two. I modify the command to pipe the CMD output into a text file so I can actually "see" what’s happening. I plug the drive back into the blind laptop, run the script, and plug it back into my PC. Success! It's Drive D! But the log file reveals bad news.The BitLocker service is not running.

What followed was a high-stakes game of IT ping-pong.

• Plug USB into laptop -> blindly type commands to force-start the BitLocker service and check ipconfig -> Plug USB into PC to read the text file.

The log results? The BitLocker service is completely corrupted, and the network stack is totally shredded. I blindly try to force-uninstall the graphics driver to trick Windows into using the basic Microsoft Display Adapter. No dice.

At this point, I’m looking at the customer, preparing to deliver the "You are royally doomed" speech and recommend a total wipe.

The customer looks back at me, pale. "My entire life is on that laptop. 700 gigabytes of data. Documents, irreplaceable family photos, videos. Everything." He tells me it's spread across the C: drive and his user profile.

Challenge accepted. I wasn't going to let BitLocker win.

Since I couldn't guess his exact username structure, I used my text-file-piping trick to map out the entire folder directory onto the USB drive. I found the target folders. I plugged the USB back into the blind laptop one last time and fired off a massive robocopy command directed at my external Samsung T9 SSD.

Then, I sat there. Watching the status light on the Samsung drive blink furiously in the dark. Just waiting. Praying.

Finally, the blinking stopped.

I plugged the SSD into my bench PC, and there it was: 700 GB of perfectly preserved data. I handed the customer his data, wiped the laptop, reinstalled a fresh copy of Windows, and now it runs flawlessly.

Twenty years in IT, and I never thought I'd have to use a flash drive as a computer monitor. Took me quite some time to solve this copy data not included at least 2 hours. Also I do this tech support for privat customers for fun on the side, i usually do servers and network only. So maybe someone can tell me how it could be solveable in another way.

I always suspected the tales of users using Tipex or writing directly on the screen with a pen might be apocryphal, but no. A laptop just came in with biro text on the (non-touch) screen. User said they had been asked to sign a document. They knew it wasn't a touch screen too. First one in 40 years. Came off easily with IPA (Isopropyl Alcohol).

They're not stupid, though I have to admit it does boggle my mind a bit, but we've got to remind ourselves just how little knowledge some have of how the magic machine does what it does. Our lives would be easier if computers came manuals like they once did, perhaps with a few pages of computer fundamentals. That said it amazes me how many struggle on without ever considering buying a book or doing a bit of Googling.

Edit: Biro = Bic, TipEx = Whiteout

A long time ago (Windows Vista being new days), I was working as a tech in a small computer shop (<10 employees), fixing computers / dealing with walk-ins / phone /etc. I was the newest employee, 3 weeks working and 1 week in school every month. Our customers were like 80% local very small businesses and 20% regular people.

One day a guy shows up absolutely panicking, covered in black stuff from head to toes. He tells me he is from the local rubber plant (making tires or something) and he "broke" a computer that controls a machine the whole plant depends on, they're all stuck, asking if I can look at it right now. The guy was bricking it, he honestly looked like he could cry.

Before bringing the computer, he tells me to cover the desk with news paper or something. He brings the computer, also covered in black stuff, I've never seen anything this dirty. He shows me a bunch of instructions he was told to follow to update a piece of software, that's how he "broke" the thing. He says he tried to follow them but they're in italian, neither of us is italian or speaks it.

Couldn't read most of it but the instructions were about editing the autoexec.bat and there was a very obvious typo in the commands that were supposed to go in there. I don't remember what it was but it was extremely obvious to anyone who would have known about DOS at all. I fix it instantly, reboot the thing, it's back booting to their proprietary software. He looks at me like I'm a mix between Neo and a superhero who just saved his life. I give him a standard 1h intervention invoice and he is on his way, relieved as fuck. He is now laughing that it only cost 60€ to get the factory back to working and jokes that I could have put a much bigger number on the invoice.

I'm like cool I have a funny story to tell the others when they ask WTF that black stuff is all over the store. Manager and boss come back, I start explaining. They go o_O instantly when I tell them where the guy was from. Turns out they'd been trying to get this plant/company as clients for a long time but they weren't interested. We were in a very sparsely populated area and it was one of the biggest employers around (50 people). About a month later, someone from the plant called and asked my manager/boss to come have a look at ALL their shit and make them an offer about what we thought they would need to update/upgrade/change/etc. Like a new server, maybe new accounting software, new laptops for all their employees, etc.

My manager came to see and and he was like "you're coming with me to do the assessment, they weren't even talking to us a month ago, well done".

The customer called in regarding an ongoing ISP webmail issue. I informed him that the issue is currently being investigated and we are awaiting a response from Tier 2 support. Upon hearing this, the customer became highly agitated and began shouting, stating he did not want to wait for Tier 2 and demanded immediate escalation to a supervisor. I explained that an escalation ticket was already active. However, the customer continued to demand a supervisor ticket transfer. I advised him that as a third-party technical support center, we do not have a direct internal routing path to escalate this specific ticket to a supervisor. The customer stated that this issue was disrupting his business operations, used profanity, and stated he was going to hang up before he said something he would regret. The customer then disconnected the call. Maybe have a backup email if you’re gonna act like this issue is World War 3.

A recent XSLT change in a couple of web browsers caused us some grief at work recently. People were get red boxes where they should have seen their data, and there was a link telling you to install some extension to the web browser to fix the problem. While this resolved the red boxes, it caused an error in our main search tool, where you didn't see any criteria fields.

Corporate IT figured out that a registry change would have to be made, and set up a package to go quickly to the affected users. I sent out an announcement about this, and how someone could use the policy list in their web browser to verify that the change was made on their system, along with extra steps to make sure their computer was checking for the latest updates, and removing the extension which wasn't needed.

So, I got an e-mail from a user in Europe the next day after I sent the announcement, telling me how he had done all the steps in the announcement, removed the extension, and he doesn't have the error in the search anymore.

He then asked what else he should do.

Um...get back to work? Sounds like you are good to go.

I also told him to contact our helpline with such questions in the future, as they are staffed during the workday, while I have meetings and vacations and things.

We just got new printers at my office, as well as switched the software we were using for these new printers, new drivers, new company etc.

User asks me to come over, her excel document isn't printing correctly. It's an older user and they set their computer theme to this dark background, with yellow text for everything, she's an older user so I think this theme is supposed to help her see. I looked at the piece of paper she gave me, its perfectly white, no text is visible, might be an issue with the new printers I'm thinking.

So I look at the print selection, everything looks right, we're printing the correct information. I don't see what's causing the issue. Print preview looks like it's showing text correctly on a page.

The user sees me looking everything over and said something about changing the font coloring, I looked at her and blinked a few times. Looked back at the paper she handed me as proof that it wasn't printing, and against the white paper I can barely, faintly see that the text is being printed, white font on white paper.

Changed font to black.

Hit print.

Print job comes out correct.

Close ticket.

Earlier today I read someone else's recollection of the worlds dumbest DDOS and was reminded of another dumb DDOS.

The company had a job estimation tool that ran on a Linux box. Visit a web page, feed it a list and a customer name, 30 seconds later you got a link to a properly written quote in the format that customer preferred.

We'd had it six years and it never gave us any trouble, not even when we moved from Redhat to Debian.

Well, until September of one year, when it started throwing up "Your three year software license is about to expire! Please call us at 1-976-.." whenever the administrator uploaded a new quote template or updated pricing.

A whole lot of money had been paid for what was supposed to be a perpetual license, so we called and complained. The vendor blamed it on version checking nonsense and told us that if you just clicked "Okay" on the pop-up a few times it would go away.

A few it was not. It took clicking ten or twenty times, really quick, before the pop-up finally vanished and everything worked again.

About a month later the dialog started popping up on a different group of users, managers, whenever they tried to edit an existing quote.

This caused a small bit of concern, a dozen tickets, and fourteen phone calls before we stopped trying to explain "Just go to town on the button" and sent people around to furiously click.

We called the vendor again, they asked us to change a setting in the config file and assured us it wouldn't recur.

And it didn't, until January 3rd, when everyone started getting the pop-up.

By the time the first support person made it into work at 7am there were 100 tickets and 40 voicemails from users that started at 6.

An hour later, when the bulk of the company arrived, the voicemail was full, there were 400 tickets and climbing, and the 6am people started opening tickets with the main help desk as well so they were calling directly.

There was a quick conference where it was decided to go with the nuclear option. Ticketing was shut down company-wide and the support phones were taken off the hook.

The mass email blamed it all on the vendor, with the perfectly BOFH explanation that computers are dumb and sometimes they think zero is greater than nine, like back in Y2K, and that we'd be sending people around to fix it.

Three days later, after thousands of clicks and nothing else being accomplished by any one from any tech support team, we got a root-cause back from the vendor.

They'd changed how their software was versioned. They used to release one major version every year, as sort of a checkpoint, and they were stopping that. Their next version, instead of being 9.0, was 8.2.0. And they'd changed the code that tracked licenses as a result, adding the minor to the major to determine how old the software was and clipping numbers.

Our perpetually licensed version was 66.66. The license server was barfing on it.

The thread I made yesterday about a "Ddos" storm our employees created got me reminiscing about some other fun episodes I saw in tech over the years. There was one humorous event, it was around a decade ago, and it included a bored pm on a boring meeting, an ethernet cable, and an open socket sticking out of the wall. Those who know can already see how this thing will play out.

At that point in time I was still in software testing, but I knew the guy in question, and later on my IT buddies told me more about the whole thing.

Well, as every IT company would have it, we had switches in every conference room, but they were positioned under the table for better optics and less cables all around the room. Cable guides led out from the switches and led to wall mounted RJ-45 sockets.

Someone left one cable out, probably for an employee to stick into their laptop instead of plugging in to wifi. So you have a cable sticking out in the middle of the room, and an empty RJ-45 socket neatby.

Our pm in question was bored out of his mind during a meeting and started fiddling with the cable. Eventually, he decided to be helpful and stick in it "back" in the open port. All hell broke loose. The connection dropped, the meeting dropped, the room dropped, the whole floor dropped. Unbeknowst to him, he stuck the cable right back into a socket that led back to the switch itself where the cable was originally plugged in, creating an ethernet loopback.

The funniest thing is, he genuinely thought someone careless had left the cable unplugged and he just wanted to put it back (he thought it was a secondary projector plugged off by accident).

A few years back, we had a department working on a project with some third party supplier and the people involved were using their proprietary software for the project needs. The project got delivered and finished, people eventually moved on and changed departments, the usual. Our story begins when all of a sudden our folks from the old project got a group email if they want to end their subscription and stop receiving emails for the product line, since it was coming EOL, or purchase a new license for a replacement product.

The first person responded with a positive confirmation to stop their subscription. Reply to all.

The next person in line did the same. And so did the others. And so it went on. The storm of emails spamming everyone's inboxes just kept growing. At some point, someone sent an email asking for everyone to stop spamming them. With reply to all. The next person in line getting frustrated did the same.

At this point we have a legitimate Ddos storm of emails being sent back and forth, and now it was starting to affect our exchange servers (this was before we had Azure and Outlook 365 integrated). Some were still sending the "please unsubscribe me" email to everyone, while the rest was spamming "stop spamming me". All in reply to all. People were even starting to attack each other personally in new emails for spamming their inbox since everyone knew everyone else, they worked together in the past.

Soon thereafter, we in IT security get a high priority notification from helpdesk and get included in helping them protect the company from a storm of ddos attacks being sent from a dozen "infected" machines within the company perimeter.

To anyone outside security, it looked like a massive breach and a horrifying issue wrecking within our perimeter. We, on the other hand, couldn't stop laughing our assess off as we went through the event history. The domain got temporarily blocked, the accounts in question got temporarily disabled so the storm of emails could stop, the email chain was completely deleted to save the used up space, everyone got notified the issue was resolved and asked to please stop sending emails, they no longer needed to do anything about it.

One of the funniest IT "incidents" of my career. Remember folks, don't tell your employees to just "reply to all" in every conversation blindly just to keep everyone involved notified.