For about six years, the polite way to describe the inside of an AI model was “a black box.” The less polite way was “a mess.” Researchers would open a model up, look at a single neuron, and find it firing for cats and for cars, two things with nothing in common. The conclusion they drew was gloomy. The model had crammed more ideas than it had room for into the same parts, and a human would need another AI to untangle the first one.
This week a researcher who helped build that gloomy picture said he thinks it was mostly wrong.
Nick Cammarata worked on the early OpenAI research that made those mixed-up neurons famous. Now at Goodfire, he’s published a long essay called “Neural Networks Are Already Clean.” His argument is that the mess was in the observers. The neuron that seemed to like cats and cars responds to pointed, V-shaped forms: ears, flames, the nose of a fox, the angles of a car. Once you know the concept, the neuron reads cleanly.
I like that argument a lot, because I’ve lived a version of it.
I also think a few people are running a lot further with it than the essay does.
What does the “Neural Networks Are Already Clean” essay claim?
The old theory has a name, superposition. A network has more concepts to represent than it has neurons, so it packs unrelated concepts into the same neurons and trusts that they rarely collide. If that’s true, single neurons are close to meaningless, and you have to decode them with a second trained network called a sparse autoencoder.
Cammarata’s replacement idea is what he calls recursive generalization. A network learns a useful pattern, then builds new patterns on top of it, then builds on those. A neuron that starts out spotting the letters “iy” ends up responding to “diy” and then to “do it yourself,” a phrase that doesn’t contain “iy” at all. From the outside that looks like noise. From the inside it’s a family tree, every branch growing out of the one before it.
The numbers are what got everyone’s attention. In preliminary audits of Qwen3-32B, an open model, he reports that about 90% of neurons have one main job, about 7% have two related jobs, and somewhere around 2 to 4% aren’t understood yet. On an old image model, InceptionV1, the share of readable neurons went from roughly 5% with 2010s tools to about 50% with better tools.
Most neurons in the audited model do one main job. The small remainder is where I'd look first.
He’s careful about all of this. He calls the essay a working hypothesis, warns that the percentages depend on thresholds and definitions, and admits models with trillions of parameters “aren’t close to solved due to sheer size.”
The careful version is a good piece of science. The loud version is getting shared with a lot more enthusiasm than the caveats.
Being able to read a system has never stopped it from failing. You still have to watch it run.Share on X
Why does messiness mean you’re missing a concept?
Patient people reading logs have solved more outages than any dashboard ever did.
One line in the essay stuck with me. A tangle tells you the observer lacks the right concept, while the system had structure the whole time.
I spent twenty years running computer operations for a retailer. Every system there produced logs, and plenty of those logs looked like garbage until you understood the subsystem that wrote them. A new hire would stare at a screen of error codes and see chaos. Somebody who’d been there five years would glance at the same screen and say “the backup job is fighting the database again.” I was the new hire once, and I stared at those screens feeling stupid until the vocabulary clicked.
Same text. Different concepts in the head reading it.
So I’m not surprised a neuron that seemed to like cats and cars turned out to like a shape. Complicated systems look that way to anybody who hasn’t picked up the vocabulary yet. Biology went through the same thing. “Whales and bats” sounds like a random pair until somebody tells you what a mammal is.
That part of the essay should cheer anybody who wants these systems understood. Interpretability turns into ordinary science. Look at a neuron, form a guess, test it, repeat. Cammarata says as much, and he suggests scaling that plain investigation will beat clever new math. I’d bet on that too. Patient people reading logs have solved more outages than any dashboard ever did. Why would AI be different?
Does readable AI mean safe AI?
No.
Some of the LinkedIn reaction loses me right here.
One chief AI officer called it a landmark result that should “put catastrophic existential fears to rest.” Another post said humanity’s chances of survival just improved. Put to rest by whom, exactly? Those claims run well past the evidence. The essay is about whether we can read neurons. It says nothing about whether what we read is good news.
Every system I ever ran was readable. We had the source code, the manuals, the vendor support lines and the logs. They still failed.
The worst outage of my career came from a routine disk replacement that triggered a bug nobody knew existed, and it kept us down for three days. Every component in that chain was documented. We understood every piece, and it didn’t save us. People watching production and ready to fix it did.
Readability gives you a flashlight. It doesn’t do the walking. Somebody still has to point it at the right corner, every day, and the corners multiply as the systems grow. A trillion-parameter model has a lot of corners.
The hype camp loves a result like this for the same reason the doom camp hates it: each side wants one study to settle the argument. I’ve written before about why the doomers and the hypers are both wrong, and this is a clean example. A real step forward gets dragged into a fight it was never part of.
What about the neurons nobody can explain yet?
If 90% of neurons have one clear job, what’s in the remaining slice? I’d want that answered before anybody declares victory.
In operations, the boring 90% never kept me up at night. The batch jobs that ran the same way every night, the servers that did one thing, the reports that printed on schedule: those almost never caused trouble. The outages came from the rare cases. The job that only failed on the last day of a quarter. The interaction between two systems that only happened when both were under load. The weird path through the code that ran once a year.
The 2 to 4% of neurons that resist a label sound a lot like that. They’re the rare, context-dependent cases, and rare, context-dependent behavior is exactly where you’d expect the surprises in an AI system to live: the strange answer to an unusual prompt, the behavior that shows up only under one combination of conditions. A clean majority tells you the system is mostly understandable. It doesn’t tell you the dangerous part is in the majority.
None of that knocks the research. Cammarata says himself that some neurons remain unexplained. My complaint is with anybody who reads “90% clean” and hears “safe.”
What should people take from the AI black box research?
The first rule of risk management is to name the risk. I made that argument when AI companies started war-gaming a catastrophe without saying which one, and it applies here too. “AI is dangerous” is a mood. “This model behaves badly under these conditions, and this circuit causes it” is a risk you can manage. Readable neurons make the second kind of statement possible. That’s the real win.
The next step is the one nobody can skip. Read the neurons, then test what you think you read, then keep watching the model after it ships. Researchers have shown they can find and push on internal directions in these models that change behavior well outside the task they studied, and I wrote about that work in AI consciousness research.
Finding a dial is great. Learning what else moves when you turn it takes far longer.
For writers and business owners, the practical lesson hasn’t changed. The tools are machinery, built by people, and people can understand, test and fix machinery. They aren’t magic and they aren’t monsters. I don’t think they’re curious or sentient, and nothing in this essay changes my mind on that. If anything, it makes them look more like what I always assumed they were. They’re very large, very clever pattern machines, and now we have some evidence we can read their patterns.
Use them the way you’d use any powerful tool you didn’t build yourself. Check what they give you, especially names, numbers and facts, because a readable network can still distort your own data with total confidence.
My take
Good research, bad victory laps. Cammarata showed that most neurons in a big model can be read, and I believe him, because complicated systems always look like garbage until you learn the vocabulary. Readable has never meant safe. Every system I ran was documented down to the last cable, and they still went down. Celebrate the flashlight, then keep somebody walking the halls with it, starting with the neurons nobody can explain yet.
Will we ever fully understand how AI models work?
I think so, eventually, and this essay moves my estimate closer. Not because one study cracked it. Because it suggests the job is ordinary, patient science with a huge workload. Ordinary jobs with huge workloads get done. They get done faster once AI tools do the reading, and the field is heading there.
Full understanding of a system and safe operation of it are still two different jobs, though, and the second one never ends.
Ask anybody who’s carried a pager.
The AI and Writing Hub has more on what these tools can and can’t do, from the people side and the technical side. If you’re an engineer or researcher with work like this worth putting in a book, I’ve spent my career translating technical work for people outside the field, and a technical ghostwriter who understands the work makes that book a lot easier to write.
The black box is getting a window. Somebody still has to stand at it and look.
I wrote this book about using AI so it amplifies your thinking. It covers how to produce more and make better decisions while you stay the one doing the judging.
This one covers the other side, what happens when people hand their judgment to a machine and stop checking. You don’t feel that loss until the day you need the skill.
