Table of Contents
TL;DR: In 1980, a water district in a snow-locked region had a problem nobody could solve with the technology of the day. The remote pumping stations were cut off from central command for weeks at a time every winter, and the water had to keep flowing. The team built a system of distributed autonomous controllers in Pascal that could think for themselves, knew their limits, and failed safely when conditions exceeded what they had ever seen how I apply this to AI on books. The system ran for decades. Every successful AI rollout since has followed the same shape. Every failed one has broken the same rule. The buzzword changes. The rule doesn’t.
Northern California, winter 1980. A small control room in a concrete building somewhere in the snowbelt. Outside, three feet of fresh snow on the ground, more coming down, the kind of weather that closes roads and shuts down the cell network nobody had yet.
Inside the building, a young engineer sits in front of a CRT screen running green monochrome text. The screen is showing flow rates from a pumping station forty miles away. The station is currently inaccessible. The road is closed. The phone line is down, because in 1980 these stations communicated by leased phone lines and the lines fell with the trees.
The pumping station is running anyway. It’s running because somebody three years earlier had been told that the existing system, the one that required a human operator at central command to make every routine decision, was about to fail in exactly this kind of weather, and they had time to fix it before winter.
The problem
The water district covered hundreds of square miles. Tanks, dams, pumping stations, sluice gates, all scattered across mountain terrain that nobody could reach in winter. The job of central command was to monitor flow, adjust pump rates based on demand, open and close gates to balance the system, and respond to any equipment that misbehaved.
For nine months of the year, this worked. For the other three months, central command lost contact with the remote stations one by one as roads closed and phone lines fell, and the system had to keep operating without the humans who normally ran it.
The old solution had been to set each remote station to a winter mode before the snow came. A fixed schedule. No adjustments. If demand changed, if a pump failed, if a tank started overflowing, the station kept doing what it had been told in October until somebody could physically reach it in March.
This worked badly. Tanks overflowed. Pumps ran when they didn’t need to. Equipment failed because nobody was watching it. By spring there was always damage to repair and water that had been wasted or lost.
The new solution had to be something different. The remote stations had to be able to make decisions on their own when they couldn’t reach a human, but they couldn’t make every decision, because some decisions required information only central command had.
The system they built
They wrote it in Pascal. Each remote station got its own controller, its own program, its own job. The program could read every sensor at the station. It knew the normal flow rate, the seasonal variations, the historical patterns from years of data. It knew what to do in dozens of routine situations because the team had walked through every routine situation with the human operators before writing the code.
And, critically, the system knew its own limits.
When conditions at a station went outside what the controller had ever seen, it didn’t guess. It didn’t extrapolate. It did one of three things: it reverted to a safe default that wouldn’t damage equipment, it sent an alert to central command if the lines were up, or it logged the situation and waited. The system was designed to fail safe, not to fake competence.
The engineer watching the green screen that winter night could see flow rates from the station forty miles away because the controller at the station had decided the current conditions were within the normal envelope and was running on its own. If the station had encountered something unfamiliar, the screen would have shown a flag instead of a number, and the engineer would have known a decision had been deferred until a human could reach it.
What the system never tried to do
It never tried to replace the human operators.
The operators still did the planning, still set the seasonal envelopes, still made the calls on anything that required judgment about the district as a whole. The system handled the routine, predictable, repetitive parts of the job when the operators physically couldn’t be in the room. The operators handled everything else, all year round.
This was not the cheap solution. The cheap solution would have been to write a controller that tried to do everything the human operators did, in their absence, and call it a day. That controller would have failed the first time conditions at a remote station went outside its training, because it would have generated a confident response to a situation it had no business responding to, and the response would have been wrong, and the wrong response at a water district can flood a town or empty a reservoir.
So the system was built around what the technology could actually do, not what it could pretend to do. The technology could run a known pattern reliably. It couldn’t make judgment calls about anything it hadn’t seen before. The design respected that limit and built the rest of the system around it.
What happened next
The system ran. It ran that winter. It ran the next winter. It ran every winter for the next two decades, with periodic upgrades, until the entire industry moved to newer hardware and protocols that did the same job with more elegance.
The engineer who wrote it didn’t think of any of this as artificial intelligence. The phrase wasn’t in common use. It was just a control system. A program that could handle the boring routine work of running a remote station and knew when to ask for help.
Forty years later, every successful AI rollout I’ve seen has followed the same shape. The system handles the routine. The humans handle the exception. The system fails safe when it’s outside what it knows. The humans stay in charge of judgment. The technology gets used for what the technology can actually do, not for what an executive press release wishes it could do.
And every failed rollout has broken the same rule
The doctor’s office chatbot that traps the patient for five minutes was designed by people who thought the system could handle every call. It can’t. It can handle the routine. The patient who knows what they want and just needs a human got trapped because nobody built the off-ramp.
The fintech company that fired 700 customer service workers and replaced them with an OpenAI chatbot was run by an executive who thought the system could do every job. It can’t. It can handle the routine 70 percent. The remaining 30 percent destroyed the brand, and the company is now hiring humans back at a cost that exceeds the savings several times over. I walked through that story in detail in Why Most AI Rollouts Fail.
Every Klarna in 2026 is making the mistake the water district team didn’t make in 1980. They’re confusing “the system handles the routine” with “the system can replace the humans.” The water district team, working with technology that today’s executives would consider primitive, understood the difference. Today’s executives, with technology those engineers could only have dreamed about, do not.
The lesson
The technology has gotten more powerful. The lesson has not changed.
Any system that runs without human judgment in the loop will eventually face a situation outside what it was trained on. Whether that system is a Pascal controller at a remote pumping station or a large language model handling customer service for a Fortune 500 company, the failure mode is the same. The system has no signal that it’s out of its depth. It produces a confident response to a situation it has no business responding to. Somebody pays.
The fix is the same in both centuries. Design the system to know its limits. Design the workflow to keep humans in the loop wherever the cost of a single failure exceeds the convenience of automating the routine. Build for what the technology can actually do, not for what the press release wants it to do.
The water district team solved this in 1980 in Pascal. The buzzword changes. The rule doesn’t.
For the deeper version of the story, told as a first-person memoir about what the engineer was actually thinking in that control room, see What I Learned in Pascal in 1980 That Every AI Executive Should Know.
Frequently Asked Questions