Eight Years of PCI Audits: The Most Thankless Heroics in IT

For at least eight years straight, I lived an annual ritual: the PCI audit. I ran the infrastructure group at a major national retailer.

The networking group ran the network. Between us, with development pulled in wherever the audit touched their code, we carried the company through an assessment that decided whether it could keep accepting credit cards.

Most of our business came through cards. If we failed, we’d lose the ability to process certain brands, and the revenue impact would have been immediate and severe. Every audit season carried that weight, and every audit season ended the same way: we passed, the business kept running, and nobody outside IT ever knew anything had happened.

What does a PCI audit take?

The audit was IT-wide. Not company-wide; the rest of the business was untouched. But inside IT it reached everything: my entire infrastructure staff, the full networking group, and slices of development wherever cardholder data flowed through their systems.

We usually got a couple of months of notice. Two months sounds like time until you understand what has to happen inside it: every system scanned, every finding triaged, every fix implemented and verified, every document current, all while the day job of keeping a retailer’s systems running continues at full speed.

The load wasn’t evenly distributed. My team handled a stream of specific, farmed-out fixes: firewall this machine, correct that SQL configuration, patch this server. Manageable, discrete work. The crushing weight landed on me and on the network side, especially the network manager. The network was where cardholder data lived and moved, so the network was where the audit lived too. Those were tense months, every year, for eight years.

What does preparing for a PCI audit involve?

From the outside, audit prep sounds like paperwork. From the inside, the two months broke down roughly like this. The first stretch was discovery: scanning every system in scope, which for us meant everything, because we scanned everything instead of sampling. That thoroughness wasn’t bureaucratic zeal; it once turned up a password-cracking tool a consultant had planted on a server, a story I tell elsewhere in this series, and we’d never have found it by sampling.

The middle stretch was triage and remediation. We sorted every finding from the scans: quick fixes farmed out to the teams, structural problems escalated to me and the network manager, and the stuck cases worked into mitigating controls. The last stretch was verification and documentation, proving that we fixed what we said we fixed, because an auditor who grades on the spirit of the standard also checks the letter of your claims.

Running underneath all of it was the day job.

A retailer’s infrastructure doesn’t pause for audit season. Systems still failed, projects still shipped, and the same people doing the audit work carried their normal load. The two-month figure hides that overlap, nobody ever budgets for it, and it’s the part that wears people down. Anyone who has lived through an audit season will recognize that particular exhaustion instantly.

The people problem inside the audit

My team wasn’t trained in security. Most infrastructure teams aren’t. They saw the audit as something they shouldn’t have to worry about, an interruption to their real work, a specialty that belonged to someone else. Getting them engaged was a yearly negotiation.

I wasn’t trained in security either. When PCI was handed to me, I had no security training at all, and the answer was, “You’ll get it.” That first audit was not fun. We had a good consulting firm helping us, so it went pretty well, and its head got us through all eight of our audits. I still keep in touch with him.

They did fine, every time. But the lesson stuck with me: in most IT organizations, security is a thing that happens to the staff once a year instead of a thing they practice. The audit passed because of a few people. The organization never absorbed it. If I were advising that company today, that’s the first thing I’d change.

Part of that problem is structural, and I see it almost everywhere. Executives lump cybersecurity in with IT, so security reports up the same chain as the people it’s supposed to keep in check. Security sat inside IT for us too, and it caused real problems, because people put their own IT issues ahead of security. I think it belongs in its own department, possibly reporting to the COO while IT reports to the CFO, so it isn’t biased by IT’s priorities.

Furniture

When you pass, nothing happens. No stage, no bonus announcement, no all-hands recognition. You’re furniture. The systems keep taking credit cards, which they were already doing, so from the outside it looks like nothing occurred.

What occurred was two months of near-heroic effort by people under real pressure, delivered on time and on budget, eight years running. The only version of the audit anyone outside IT would ever have noticed is the version where we failed. Success was designed to be invisible. That’s a shame, because the work matters enormously, and the people who do it burn out on exactly this asymmetry. If you run a company that passes its audits every year, the correct response isn’t silence. Somebody worked hard to give you that silence.

Why should executives care about PCI audit stories?

Because the audit shows exactly where security depends on a handful of people doing thankless work, and the company is most exposed at those points. When those people burn out or leave, what they know walks out with them, and the executives who never thanked them usually find out what they did the year the audit fails.

Heard elsewhere

The recurring problem in cybersecurity is communication before technology, leaders seeing it as a cost center because the people explaining it speak in bits and bytes instead of risk. Richard discussed the reframe on Tequila and Tech Talk.

When leaders write books about security and compliance, they tend to write about structures, and most of those books put me to sleep. The story worth telling is the network manager under two months of pressure, the team that thinks security is someone else’s job, the pass nobody celebrates. Readers who have lived an audit recognize it instantly, and readers who haven’t learn what their own IT departments never tell them.

And if I could send one instruction back to myself at the start of those eight years, it would be this: don’t let security live only in audit season. The yearly scramble existed because security was an annual event instead of a standing practice. Teams that patch, scan, and verify continuously walk into audits with weeks of work instead of months, and their audits stop being heroics. Heroics are what you need when the system is wrong. We were very good at heroics.

For more from this series, see The Cybersecurity Hub: breaches, audits, and hard-won security lessons from decades in the trenches.

Frequently Asked Questions