Whenever we see a major internet outage like this past week’s multi-billion dollar Amazon Web Services debacle, I like to come into work and crack wise: “Bumped into a server rack with my big ass and unplugged it this morning, hope nobody notices!” I may have been closer to the mark this time: As reported by Ars Technica, Monday’s outage was shockingly caused by a single software glitch whose effects cascaded through AWS’ systems, though no one’s tuchus appears involved.
The culprit was a software bug in DynamoDB, AWS’ DNS management system. DNS or “Domain Name System” is often analogized as the phone book of the internet, translating domain names preferred by human users into the IP addresses required by automated systems. On the modern internet, many services, such as cloud computing or streaming, need to be able to assign the same domain to multiple IP addresses in a network, allowing efficient use of geographically distributed servers.
DNS Enactor, a component of DynamoDB responsible for updating these tables, “experienced unusually high delays needing to retry its update on several of the DNS endpoints.” While Enactor played catch-up, DynamoDB continued generating new plans for it to follow, which another, on-time DNS Enactor helpfully tried to implement.
The fresh DNS configuration that resulted was overwritten by a significantly outdated one when the delayed Enactor finally caught up, bypassing a safeguard meant to prevent just such an error which was also experiencing delays. The second, on-time Enactor then deleted the outdated plan altogether, detecting that it was stale. This state of affairs resulted in knock on effects across AWS, requiring engineers to manually diagnose the problem and resolve it.
It’s another reminder of how fragile the internet is, susceptible to strange internal logic failures and imperiled by deep sea sharks going “mmm, I love fiber optic cables.” Being able to jump out of a fighter jet, headshot someone, then get back into your plane and keep flying in Battlefield 6 is truly an audacious feat of engineering on par with putting a man on the Moon.
