Friday, July 19, 2024

It Isn't Easy Being Blue

This isn't about the blue stain on city hall fueling unlimited spending and it isn't about the blue hysteria around Biden reverting to the 2020 campaign tactic of hidin'. 

Nope. This is about the Blue Screen of Death.

Remember those? Thought that was cured didn't you? Well, you'd be wrong. Not much, maybe nothing, has changed with regards to the robustness and stability of the Windows platform, except maybe that the scope has expanded in breadth and severity. The story, accepted for god knows what reason, is that with adequate add-ons Windows can be made just as reliable as any other platform. Or so it was sold to enterprises, enterprises who also adopted remote management of software including updates. Once again we've proven: to err is human but to really foul things up requires a computer. A computer [system] was used for massive, rapid, unstaged, and apparently untested, rollouts of CrowdStrike FalCon updates. In the dead of the night. And sometimes, like this time, you wake up to dead systems. Millions of them. Maybe more.

And that is what just happened. And while some remote updates can be rolled back, when you get a Blue Screen of Death, you have a bricked computer, a doorstop. And keep in mind, this is a Windows platform, where all too many software updates require a reboot, and when that reboot hits the Blue wall, your computers are automagically Out Of Service. Return To Service requires organic intervention. That's right, manual recovery. One. Computer. At. A. Time. So while the rollback image may be available in short order, actually getting back to normal is going to take a long, long time, or an army of techs, or both. This is gonna make Y2K look like a joke (it kinda was). 

This will not only cost billions of dollars in lost revenue, it will actually cost lives. In the UK, doctors are triaging, taking only the most serious patients, leaving the others without care. Globally, hospitals are cancelling elective surgeries, prioritizing the most urgent. Care deferred is lives lost, and the corporate decision makers have blood on their hands. Think of that when you cannot buy that latte. 

But thinking locally, are we really impacted? After all, aren't our voting systems running atop Windows? Yes, yes they are. And here's where irony kicks in. The version of Windows in these devices is no longer supported, as in no updates, automatic or otherwise. Furthermore, they are not likely to run the CrowdStrike package, and it seems that while source code control is somewhat lacking, actual updates to the system are treated seriously, almost fearfully. Not that this happy circumstance is a result diligence or foresight, but this is Georgia and we'll take what we can get. 

So let's hope we've paid our dues.