45 minutes, $460 million: when technology governance is missing

Governance has a reputation for bureaucracy: committees, policies, documents nobody reads. But when I look at the technology disasters that actually shut companies down, I almost never find a brilliant bug. I find one simple control that nobody had.

Two cases explain this better than any presentation on risk.

Knight Capital: 45 minutes, $460 million

On 1 August 2012, Knight Capital was one of the largest electronic brokerages in the United States. To take part in a new program from the New York Stock Exchange, it needed to update the order router on eight servers.

Seven got the new code. One didn't.

The detail that turns a deploy error into bankruptcy: the team had reused a flag that, years earlier, switched on a system called Power Peg — code that had sat dormant since 2003 and was never removed. On the forgotten server, the flag switched that dead code back on.

In the first 45 minutes of trading, the router fired more than 4 million orders trying to fill 212 customer orders. 397 million shares changed hands, for a loss of roughly $460 million.

Two details hurt more than the loss itself:

• The alerts existed. Before the market opened, an internal system had already fired off 97 automated emails flagging the problem. Nobody acted on them.
• The response made it worse. Technicians uninstalled the new code from the seven servers that had received it. The bad orders kept going out.

The SEC concluded the company lacked adequate controls, including mandatory code review and a defined deployment process. Knight only survived that week thanks to an emergency capital injection. Less than a year later, it was sold in a merger and stopped existing as an independent company.

Code Spaces: 12 hours

In June 2014, Code Spaces hosted Git and SVN repositories and project-management tools. It had backups. It even had what it called an offsite backup.

An attacker gained access to the company's AWS control panel, launched a denial-of-service attack, and demanded payment. When Code Spaces tried to regain control of the account, the attacker deleted everything they could reach: snapshots, buckets, machine images, and instances. In the company's own words, "most of our data, backups, machine configurations and offsite backups."

The problem wasn't the lack of a backup. It was that the backup lived inside the same account the attacker controlled.

That same day, the notice on the company's site said it would not be in a position to continue operating. An entire company shut down in hours, because one credential granted access to everything — including the backup copy.

The pattern behind both

Different decades, different industries, different technologies. The pattern is the same:

• A process that depends on someone remembering. Copying a file to eight servers by hand works fine — until the day it doesn't.
• A control that was never tested. A backup nobody has restored is hope, not a control. Same goes for a rollback nobody has rehearsed.
• An alert with no owner. 97 emails aren't an alert, they're noise. An alert is something that reaches a specific person with a defined action.
• Concentrated power. A credential that can wipe out production and its backup together isn't convenience — it's existential risk.
• Dead code that stuck around. Whatever isn't removed stays executable, and someday somebody reuses the switch that turns it on.

None of these is a hard technical decision. They're all governance decisions: who decides, who checks, who owns it, how often it gets tested.

The minimum governance that would have prevented both

If your company runs software in production, this is the floor. Each item maps to the disaster it would have prevented.

1. Automated deploys, identical everywhere. No server gets updated by hand, and the deploy only finishes after confirming every server is on the same version.
2. No dead code, no reused flags. Removal is part of the job. A kill switch has its own name and does exactly one thing.
3. Rehearsed rollback. Before the deploy window, someone runs the rollback in an identical environment. "We'll roll back if it breaks" is not a plan.
4. Every alert has an owner and an action. If nobody's going to act on it at six in the morning, that alert shouldn't exist. Whatever's left needs to reach a person, not an inbox.
5. Minimal, separated access. Mandatory MFA, production credentials kept separate from development ones, and nobody able to delete everything on their own.
6. Backups out of reach. A different account, a different provider, or immutable storage, with defined retention — and restores tested on a schedule, not in an emergency.
7. An owner per system and an incident runbook. Who decides to pull the plug? How is it pulled? In writing, before the incident happens.

The restore test is the easiest thing to automate and the most expensive thing to go without:

# .github/workflows/restore-test.yml — runs every Monday and fails loudly
name: restore-test
on:
  schedule:
    - cron: "0 6 * * 1"
jobs:
  restore:
    runs-on: ubuntu-latest
    steps:
      - name: Restore the latest backup into a disposable database
        run: ./scripts/restore-latest.sh --target ephemeral
      - name: Check business invariants (counts, balances, integrity)
        run: ./scripts/check-invariants.sh

The value isn't in the file. It's in having, every single week, an objective answer to the most expensive question there is: is yesterday's backup any good?

What this has to do with your company

Neither of these companies was amateur. Knight moved billions of dollars a day. Code Spaces sold infrastructure to development teams. Both had good people. What was missing was agreeing, in writing and before the incident, who decides what and what gets tested how often.

Technology governance isn't what stops the team from working. It's what lets the company survive its worst day. Putting the seven items above in place costs a few weeks of work. Not having them cost, in both of these cases, the entire company.

Further reading

• The seven items above turn into work in Platform Engineering: pipeline, observability, runbook and a restore test on the calendar.
• The other side of the same coin, a number almost nobody calculates: the most uncomfortable number in your customer base.

Sources

• SEC administrative order against Knight Capital Americas (16 October 2013) and the SEC press release about the case.
• The Register's report on the end of Code Spaces (18 June 2014).