A single incorrect parameter was enough to trigger one of the most memorable outages in the history of Amazon Web Services (AWS) back in 2017. Amazon S3 lost a significant part of its capacity in us-east-1, needed two subsystems restarted, and dragged other services down with it. The episode is especially useful for sysadmins and developers because it wasn’t an isolated case: GitLab, Meta, Cloudflare, Fastly, and Atlassian have all documented incidents where perfectly authorized commands, scripts, configurations, or code ended up causing massive outages. Zoom out and it’s easy to see why: unplanned downtime at large companies already costs $600 billion a year.
The key facts about errors that broke major platforms in 30 seconds
- AWS removed too many S3 servers in 2017 because of an incorrectly entered parameter and had to restart two subsystems.
- GitLab accidentally deleted its primary database directory that same year and lost several hours of data.
- A regular expression maxed out Cloudflare’s CPUs in 2019.
- Meta disconnected its global backbone in 2021 during a maintenance operation.
- Fastly and Atlassian show that even a valid configuration or an authorized script can have an unexpectedly large blast radius.
The S3 case is particularly instructive because AWS explained what happened in considerable detail. At 9:37 a.m. Pacific time on February 28, 2017, an authorized member of the team was running a routine procedure to remove a small number of servers while investigating an unrelated billing system issue. One of the values entered into the command was incorrect, and far more servers were removed than intended.
The problem reached S3’s index and placement subsystems. The index subsystem held the metadata and information about where objects were stored; the placement subsystem managed the allocation of new storage. Both needed a full restart.
And that’s where a second problem showed up, one that feels familiar in infrastructure that has been growing for years: those large subsystems hadn’t been fully restarted in a very long time. S3 had grown enormously, and the checks needed to bring the whole platform back took longer than expected.
AWS S3 and the cost of letting a single command hold too much power
AWS started recovering GET, LIST, and DELETE operations at 12:26 p.m. The index subsystem was fully recovered by 1:18 p.m., and PUT operations, which also depend on the placement system, returned to normal at 1:54 p.m. Other services took longer to work through the backlog.
The $150 million figure usually attached to the incident needs an important caveat. It wasn’t a loss disclosed by Amazon.
Risk-modeling firm Cyence estimated roughly $150 million in losses for S&P 500 companies and another $160 million for U.S. financial services firms. Those are estimates of economic impact, not a bill paid by AWS or an audited figure of Amazon’s own losses.
What’s more interesting for a technical team is what AWS changed afterward.
The tool used to remove capacity could take out too many servers too quickly. Amazon changed how it worked to slow those operations down and introduced limits that prevented any subsystem from being pushed below the minimum capacity it needed to keep running.
It’s a near-perfect example of a guardrail.
The fix wasn’t simply telling the operator to double-check the command next time. AWS changed the system so that a similar human mistake couldn’t produce the same result.
It also accelerated splitting S3 into smaller units called cells. The goal was to reduce the blast radius: if one part fails, recovery and impact can be limited to a fraction of the system.
Even the system used to report the incident was affected. The Service Health Dashboard’s admin console depended on S3, so AWS couldn’t update individual status entries normally until 11:37 a.m. After the incident, Amazon moved that administration to an architecture distributed across multiple regions.
The human error had uncovered several dependencies that had seemed reasonable in isolation.
And AWS is far from the only example.
Six incidents every sysadmin should know about
The postmortems published by large companies make it possible to build a small collection of failures with something in common: the initial action looked far less dangerous than its consequences.
| Incident | Trigger | Documented impact | Technical lesson |
|---|---|---|---|
| AWS S3, 2017 | Incorrect parameter in a server-removal tool | S3 us-east-1 and dependent services affected for hours | Hard limits on destructive operations and a smaller blast radius |
| GitLab, 2017 | Accidental deletion of the primary database directory | GitLab.com was down for about 18 hours and lost several hours of database changes | A backup that hasn’t been restored and tested isn’t a recovery plan |
| Cloudflare, 2019 | A WAF regular expression triggered massive backtracking | CPU usage near 100% globally and 502 errors; traffic dropped as much as 82% | CI/CD needs to test resource consumption too, and roll out gradually |
| Meta, 2021 | A maintenance command unintentionally disconnected the backbone | Facebook and other services became globally inaccessible; DNS and internal tools were also affected | The systems that validate commands can fail too |
| Fastly, 2021 | A valid customer configuration triggered a latent bug | 85% of the network started returning errors; 95% was working again within 49 minutes | A valid input can trigger a state that testing never covered |
| Atlassian, 2022 | A maintenance script received an incorrect list of identifiers | 883 customer sites were deleted; 775 customers affected, some taking up to 14 days to recover service | Automating a destructive operation requires validation, observability, and tested recovery |
The table also dismantles the comfortable explanation that these outages simply happen because “someone made a mistake.”
At GitLab, for instance, an engineer accidentally deleted the data directory of the primary PostgreSQL server while intending to rebuild the secondary one. But the postmortem uncovered a much longer chain of problems.
GitLab estimated it lost changes tied to roughly 5,000 projects, 5,000 comments, and 700 new accounts. Git repositories and wikis weren’t lost because they were stored separately. The service remained down for about 18 hours.
The incident also put its backup and recovery procedures to the test. The human error was the last visible link in a much longer operational chain.
Cloudflare offers an example that hits even closer to home for software development.
In July 2019, a new rule was deployed for its Web Application Firewall (WAF). It contained a regular expression that generated an enormous amount of backtracking. The result was that the CPUs handling HTTP and HTTPS traffic approached 100% utilization across its entire network.
At the worst point, Cloudflare recorded an 82% drop in traffic.
What’s interesting is that the change had followed the process: pull request, approval, continuous integration, tests, and deployment.
The tests checked that the WAF correctly detected certain requests, but they didn’t check whether a rule could spike CPU consumption. On top of that, rules could be distributed globally within seconds.
The pipeline was green, and production went down.
Cloudflare responded by adding performance profiling to its tests, rolling out changes progressively, and modifying its regular-expression engine, among other measures.
When the failure also destroys the tools needed to fix it
Meta took this problem to a different scale on October 4, 2021.
During routine maintenance, a command was issued to assess available capacity on its backbone network. The command ended up unintentionally disconnecting that global network’s connections.
There was a system specifically designed to audit commands of this kind and block dangerous errors. But a bug in that auditing tool failed to stop the command.
The backbone failure then caused Meta’s facilities to withdraw their Border Gateway Protocol (BGP) announcements for DNS. The servers could keep running, but the rest of the internet stopped being able to find the services.
The problem got even harder because the internal tools used to diagnose the incident also disappeared. Engineers had to travel physically to data centers to recover certain systems.
It’s the same circular dependency AWS had uncovered four years earlier with its status page.
A recovery system shouldn’t depend exclusively on the system it’s trying to recover.
Fastly showed another variant in June 2021.
The company had introduced a bug on May 12 that stayed latent for weeks. On June 8, a customer made a perfectly valid configuration change that brought together the conditions needed to trigger it.
85% of Fastly’s network started returning errors.
Monitoring detected the disruption in about a minute. Fastly later identified the triggering configuration, and 49 minutes after it began, 95% of its network was operating normally again.
Here there wasn’t even necessarily an operator who had written something incorrectly.
The configuration was valid.
The software wasn’t ready for it.
Atlassian offers perhaps the most unsettling case for anyone who maintains internal scripts.
In April 2022, the company was running a script meant to delete certain legacy applications. Because of a problem in the data fed into the process, it ended up deleting 883 customer sites. Atlassian later confirmed that 775 customers were affected.
The script deleted the sites sequentially without triggering an immediate alert.
Jira, Confluence, Atlassian Access, Opsgenie, and Statuspage became inaccessible for those customers. Some took up to 14 days to fully recover service.
The lesson isn’t to lock down access to production
These incidents point to a conclusion far more useful than “humans make mistakes.”
Systems need to be designed on the assumption that an administrator will one day enter the wrong parameter, a developer will write a problematic regular expression, a script will receive the wrong data, and a seemingly valid configuration will trigger a combination that was never tested.
That changes how critical operations get designed.
An rm, a BGP change, or a mass deletion shouldn’t follow the same execution path as a routine operation. Destructive actions can include hard caps, dry-run modes, independent confirmations, waiting windows, and automated mechanisms that block states incompatible with operational minimums.
Deployments don’t need to jump straight from zero to 100% either.
Canary deployments, phased rollouts, and feature flags make it possible to watch a small slice of traffic before extending a change globally. Cloudflare found out in 2019 just how much that difference matters, when its ability to distribute rules worldwide within seconds turned a technological advantage into an incident multiplier.
Backups deserve the same distrust.
Having a job that reports backup completed successfully doesn’t prove the system can recover within the expected RTO (Recovery Time Objective). GitLab learned the hard way that restoration is also part of the backup.
And observability needs to survive the service it’s observing.
Status pages, emergency channels, authentication systems, and admin tools should have independent paths when the cost of losing them during an outage is high.
AWS, Meta, and Cloudflare have all documented incidents where their own dependencies complicated diagnosis or recovery.
The useful question before running a critical operation isn’t only whether the command is correct.
It’s also worth asking what would happen if it were wrong, and why the system would let that error reach all of production.
That shift in thinking turns human error into an anticipated condition of the design itself.
Frequently asked questions
Was the 2017 Amazon S3 outage caused by a mistyped command?
AWS confirmed that an authorized team member incorrectly entered one of the parameters for a command meant to remove a few servers. The tool ended up removing far more capacity than intended.
Did Amazon lose $150 million because of that outage?
That’s not an accurate way to put it. Cyence estimated about $150 million in losses for S&P 500 companies and another $160 million for affected U.S. financial services firms — those weren’t losses disclosed by Amazon.
What was the most important thing AWS changed after the incident?
AWS added limits to its tool to prevent a command from removing too much capacity or leaving a subsystem below its operational minimum. It also accelerated splitting S3 components into smaller cells and changed the administrative infrastructure behind its status page.
How do you keep a human error from causing a global outage?
There’s no single fix. The cases examined here point to combining guardrails, operational minimums, dry-run modes, limited permissions, progressive rollouts, tested recovery procedures, independent observability, and blast radius reduction.
