Winging It · Implausible Deniability
There is, in every organisation I have ever worked for, a system. Nobody is quite sure what it does. It lives in a rack in a data centre that nobody visits, on hardware that has not been current since BlackBerrys were aspirational. It runs something important, probably finance-related, and it has not been patched since approximately the financial crisis. Not the recent one. The one before that.
Everyone knows about the system. It comes up in meetings only as a small euphemism. "We do have some legacy considerations," someone will say, and everyone will nod, and nobody will say the system's name out loud, the way sailors won't whistle on a boat.
The system cannot be patched. This is not because nobody has tried. It is because the one person who understood it was a contractor who finished up in 2007 and has since either retired, died, or gone to live in a cabin in the woods, depending on which version of the story you hear. The application it runs was written in a language that universities stopped teaching before some of the security team were born. The vendor was acquired, then acquired again, then dissolved into a company that now sells something else entirely and has no record of the product ever existing.
And so the system sits there, unpatched and unknowable, while the organisation builds an entire security programme around the polite fiction that it doesn't exist.
I have watched a CISO, a grown adult with a mortgage and a firm handshake, explain to an auditor that the organisation's patching policy requires critical vulnerabilities to be remediated within fourteen days. I have then watched the same CISO, forty minutes later, in a different meeting, agree with the infrastructure team that the moment we apply a patch to it, three things will happen, none of them good, and the first one is that payroll stops.
Patching, in the way the auditor wants you to describe it, is a discipline. In practice it has become a hostage negotiation in which the hostage is your uptime, the kidnapper is your own infrastructure, and the ransom is paid in change windows that never quite arrive.
Pete, from a previous job, kept a spreadsheet of patch outcomes. The "applied successfully with no impact" column had four entries. The "broke something we did not know existed" column had 412. His conclusion, delivered without emotion over a beer, was that the patches were doing more damage to the organisation than the attackers were, on the grounds that the attackers had never once taken out the phone system.
This is the part that is funniest, in retrospect, and least funny at the time. The industry has spent two decades telling people to patch faster. Patch Tuesday. Exploit Wednesday. Patch or perish. Meanwhile, every practitioner knows the truth, which is that a patch is simply a change you didn't ask for, arriving on a schedule you don't control, to software you can't test properly, with release notes that say "miscellaneous security improvements" in the tone of a teenager explaining where they were last night.
The 9.8 arrives on a Thursday. It is always a Thursday, and it is always a 9.8, the number designed by humans to produce the maximum possible adrenaline response. The CVE lands, the vendor advisory is vague, the threat intel feeds light up, and somewhere a journalist is already typing the words "organisations are urged."
And then the great machinery grinds into motion. The vulnerability management team raises tickets. The tickets go to asset owners. Some of the asset owners exist. This is not a small point. Most organisations don't know what they have, which means the vulnerability scan is less an inventory of your risk and more an archaeological survey, turning up servers the way a farmer's plough turns up Roman coins. Each discovery triggers the same conversation. Whose is this. What does it do. Why is it talking to the internet. Why is it running Windows Server 2016. Who is "TEMP-DO-NOT-DELETE" and why do they have domain admin.
The patch itself, when it finally gets applied, is tested in an environment that resembles production the way a passport photo resembles a person. The test environment has one tenth of the load, none of the weird integrations, and was last refreshed when the office still had a fax machine. The patch passes testing. Of course it passes testing. Testing was designed for it to pass. It then goes to production, where it meets the undocumented Excel macro that the entire accounts payable process secretly depends on, and the helpdesk phones begin to ring.
The system in finance, meanwhile, sits serenely through all of this, untouched and untouchable, protected by the only security control that has never failed an audit it was never subjected to: being too frightening to change. There is a Post-it note on the rack, written by someone who left in 2016. It says "DO NOT REBOOT." It has the weight of a constitution.
I once suggested, in a planning meeting, that we simply turn it off for an hour and see who screamed. This is a legitimate discovery technique, and I maintain it would have worked. The room reacted as though I had proposed setting fire to the building to test the smoke detectors. The idea was minuted as "deferred pending further analysis," which is corporate for "never," and we moved on to the next item, which was a proposal to buy a tool that would tell us about more vulnerabilities we wouldn't fix.
That is the real catch. The tooling has become magnificent. We can now detect vulnerabilities in minutes that we will not remediate in years. The dashboard shows 4,000 criticals and the dashboard is technically correct, and everyone has privately agreed to treat the number as weather. Cloudy today, 4,000 criticals, chance of ransomware.
This is, I'm afraid, what patching has come to mean in security: a series of compromises, deferments, and quiet agreements between adults to leave specific things alone. The patching policy says fourteen days because the auditor needs it to say fourteen days, and the reality says "when the change board next meets, assuming the change board approves it, assuming the maintenance window holds, assuming nothing else is on fire," which rounds to a quarter.
The new CISO started last month. She is bright, and ambitious, and she has announced a programme to achieve full patch compliance within eighteen months. The infrastructure team wished her well with the warmth of people who have attended this exact meeting before, under three previous CISOs, two of whom now work in real estate.
She has not asked about the system in finance. Not once.
— Chris