How did one maintenance command take Facebook off the internet for hours?
The internet's routing runs on a handshake of trust between networks, first sketched on two napkins in 1989. In October 2021 Facebook withdrew its own address from the map.
▶ Start the storyA routine maintenance command accidentally disconnected all of Facebook's data centres, and Facebook's DNS servers then did exactly what they were designed to do: they withdrew their own routes from the internet. On 4 October 2021 at 15:39 UTC, Facebook and its subsidiaries became globally unavailable for six to seven hours. Experts identified the problem as a Border Gateway Protocol withdrawal of the address prefixes in which Facebook's DNS servers were hosted, making it impossible to resolve Facebook's domain names.
To see why that works as an off switch, you need the idea of an autonomous system: a group of networks operating under a single external routing policy. Sprint, Verizon and AT&T are each one. Each autonomous system uses BGP, the Border Gateway Protocol, to advertise the prefixes, the blocks of IP addresses, that it can deliver traffic to. The rest of the internet learns where to send packets from those advertisements. Withdraw the advertisement and, as far as everyone else can tell, the addresses have gone.
That is exactly what happened. The DNS servers ran on a separate network, but they were designed to withdraw their BGP routes if they could not connect to Facebook's data centres. When the maintenance command cut the data centres off, they pulled their routes, making it impossible for the rest of the internet to connect to Facebook. By 15:50 UTC Facebook's domains had expired from the caches of all major public resolvers.
Step 1: Maintenance command
Meant to assess backbone capacity
Step 2: Data centres disconnected
All of them, by accident
Step 3: DNS servers withdraw BGP routes
By design, when they cannot reach the data centres
Step 4: Names stop resolving
Caches expire within minutes
Step 5: Physical reset
Engineers reach servers in Santa Clara
Quiz me
0/3
Recap
BGP is the internet's trusting signpost system: networks say where they are, and everyone else believes them.
💡 A trick to remember it · BGP is gossip between networks: say you are gone and everyone believes you.
Surprising fact · Facebook's own DNS servers withdrew their BGP routes when they lost contact with its data centres.
Connects to
- 📖 How does your browser find a website from just its name?
- 🔒 How does your browser know that the padlock is not a fake?
- 🧱 Why can one company's bad day take down so much of the web?
- 🔓 Why did an NBA team hand over every employee's tax forms to a scammer?
- 📦 How does the internet deliver your data without ever giving you a line of your own?
- 🌊 How does the internet cross the oceans?
- Content delivery network
Sources (3)
No source, no claim. Every fact in this lesson (15 claims) cites at least one of these.