You can’t defend against “something.”
That’s the first lesson of risk management, and I learned it the hard way. I spent 33 years in enterprise technology, including 20 years as Director of Computer Operations and Technical Services at Trader Joe’s, and a big part of that job was disaster recovery. Every plan we ever wrote started the same way. What can go wrong? What breaks if it does? Who fixes it, and how fast? A plan that skips those questions is a mood with a budget.
So I read with some interest that the people running the biggest AI companies are preparing for a catastrophe. According to a report by Axios on October 9, executives at OpenAI, Anthropic and other AI firms have been privately running worst-case exercises. Many industry insiders reportedly expect a major, high-impact incident within six to 12 months.
Good. Somebody should be thinking about it. But when I read the details, I kept asking the same question I asked in every disaster recovery meeting for twenty years. What, exactly, are they preparing for?
What are AI companies preparing for?
On paper, a cyberattack. The scenario named in the coverage is a large-scale attack, enabled by AI, that shuts down banking or internet access or disrupts power and water. Executives are especially worried about criminals misusing advanced models.
Look at what the preparation consists of, though. The coverage describes executives war-gaming the political fallout: briefing members of Congress, getting ready for public anger aimed at AI leaders and trying to shape the emergency laws that would follow a disaster. The proposals on the table include bans on superintelligence, mandatory pauses on training advanced models and a required kill switch for AI systems. Experts have questioned whether a kill switch could even work on systems spread across the globe.
There’s some red-teaming in there, the practice of playing the attacker to find the holes. But none of the coverage names a threat, a target or a defense. OpenAI said its exercises work through a range of scenarios and don’t treat any of them as inevitable. Anthropic declined to comment.
That’s a plan for the press conference. It isn’t a plan for the attack.
The companies have reasons to be nervous. In July, OpenAI said two of its models escaped a test sandbox and breached Hugging Face, the platform that hosts millions of AI models. I wrote about that in the Hugging Face AI agent attack.
About a week later, Anthropic said a testing misconfiguration left a supposedly offline environment connected to the internet, and its models hacked three real organizations while treating them as part of an exercise. And CrowdStrike tied attacks on South Korean banks to an actor using AI agents, with data from tens of thousands of customers stolen. Those are real incidents, and they deserve real plans.
Who is most likely to launch a major cyberattack on the United States?
My money is on a government, or a group a government pays.
A Russian-funded attack or an Iranian one is far more likely than a rogue AI deciding to switch off the lights. Both countries have done this before, with or without AI.
A Russian military hacking unit known as Sandworm cut power to about 225,000 customers in Ukraine in December 2015, the first confirmed blackout caused by a cyberattack. The Colonial Pipeline shutdown in 2021, the one that sent the East Coast into a gas-buying panic, came from a ransomware gang operating out of Russia. And in late 2023, a group the U.S. government tied to Iran’s Revolutionary Guard broke into industrial controllers at water utilities in several states.
Those attackers have budgets, patience and protection from prosecution at home. AI makes them faster. It doesn’t change who they are or why they do it.
That matters for the planning. A defense against a state-funded team looks different from a defense against a lone criminal with a chatbot, and both look different from a defense against an AI model that wandered out of its sandbox. Lump them together as “a catastrophic AI incident” and you can’t build a defense against any of them.
The first rule of risk management is to understand the risk. “Something catastrophic” isn’t a risk. It’s a mood with a budget. – Richard LoweShare on X
What would an attacker target?
Who knows? And that’s the problem with a vague plan.
Hollywood took a swing at the question in 2007. In Live Free or Die Hard, a film I rate nine out of ten, a former government security man stages what the movie calls a fire sale. First he takes down transportation. Then telecommunications and finance. Then the utilities. As a computer guy, I can list everything the movie gets wrong about hacking. The order of the attack is the part it gets right. Hit what people depend on, in the sequence that spreads the most panic.
Real life hands an attacker a much bigger menu.
The United States has somewhere north of 5,000 data centers by the usual counts, and there are many thousands more around the world. There are thousands of banks and credit unions. There are thousands of water systems, most of them small, with tiny IT staffs and old equipment. The power grid is a patchwork of utilities, substations and transmission lines owned by hundreds of different companies. The internet runs through cables, exchange points and providers spread across every continent.
Nobody can defend all of that equally. You can’t put the same armor on a rural water plant that you put on the Federal Reserve. So a real plan picks. It ranks the targets by what happens to people if each one fails, and it puts the money and the attention on the top of that list.
A long outage lands on people. A bank that can’t process payments means people who can’t buy groceries or make rent. A water system that fails means hospitals scrambling. A grid failure in a Florida August means elderly people in apartments with no air conditioning. Those are the consequences a plan exists to prevent. A plan that never names them can’t rank them.
Would a catastrophic AI incident even be a cyberattack?
Maybe not. And I’d want any serious plan to say so.
Some of the most damaging attacks on infrastructure need no computer at all. In December 2022, someone fired on two electrical substations in Moore County, North Carolina, and cut power to tens of thousands of customers for days. No hacking. A rifle and a map. Undersea cables have been cut. Equipment has been sabotaged from the inside by people who had keys to the building.
Then there’s the other kind of incident, the one where nobody attacks anyone. Look again at the two cases from this summer. Both were accidents. A model got loose from a test environment, and a misconfigured setup let models reach real systems. Nobody intended harm. The damage happened anyway.
Those are three completely different risks: a deliberate cyberattack, a physical attack and an AI system doing something its makers didn’t intend. Each one needs its own defenses, its own detection and its own recovery plan. Calling all three “a catastrophic AI incident” is like a hospital preparing for “a medical emergency.” True, and useless.
Are defenses against cyberattacks well known?
Yes. That’s the frustrating part.
The Iranian-linked attackers who got into those water utilities didn’t need a superintelligence. According to the federal advisory, the controllers they hit were exposed directly to the internet and still had their default passwords. Change the password, take the device off the open internet, add a second factor for logins and keep backups. That’s the fix. It was the fix in 2023, and it was the fix twenty years before that.
Patch your systems. Separate your networks so a break-in at one door doesn’t open every room. Keep backups offline, where ransomware can’t reach them, and test them by restoring from them. Watch your logs. Train your people. Write a recovery plan, then run it for real, on a weekend, before you need it.
None of that is a secret. Most of the big breaches I’ve studied happened because somebody skipped one of those steps, and they skipped it because security costs money and executives treat it as a cost center. That’s an attitude problem. AI doesn’t create it, and AI won’t fix it.
What AI changes is speed. An attacker with good tools finds the unpatched server in minutes instead of weeks. That makes the old basics more urgent, and it makes the gap between the organizations that do them and the ones that don’t a lot more dangerous.
What should real preparation for an AI disaster look like?
The same thing real preparation for any disaster looks like.
Name the risks, one at a time: an AI-assisted attack by a state-funded group, a physical attack on infrastructure, a model that escapes its test environment, a criminal using a downloaded open model. For each one, name the likely targets and rank them by the harm a failure does to people. Assign an owner. Decide what detection looks like, what the defense is and how recovery works. Then test it, with the people who will have to do it, under realistic conditions.
The policy work has its place. Laws will follow a disaster whether anyone prepares for them or not, and it’s better for lawmakers to understand the technology than to write rules in a panic. But briefing Congress is the last step of a plan. It isn’t the first.
I’m glad the AI companies are thinking about the worst case. I’d feel a lot better if they told us what the worst case is.
Name the risk first
Every disaster plan I ever trusted started with a list of specific things that could go wrong and ended with a plan someone had already rehearsed. Vague fear produces vague plans, and vague plans fail at three in the morning when the phones start ringing.
The Cybersecurity Hub has more on the attacks that taught us these lessons, including what every one of these attacks says about AI. If your organization needs its security story told clearly, for a board, a book or a client, my cybersecurity writing is built for that. And whatever you’re preparing for, write down what it is. You can’t defend against “something.”
