When DNS Ate the Cloud

The Moment It Started

It was 11:48 PM PDT on October 19th when the first customer calls started coming in. I was on the AWS support team that night, and what began as isolated DynamoDB connection errors quickly escalated into something we'd never seen before.

"My DynamoDB calls are timing out," the first customer reported. Then another: "EC2 instances won't launch." Within minutes: "Lambda functions failing," "ECS tasks stuck," "My entire application is down."

The pattern that emerged was both subtle and devastating: everything in us-east-1 that touched DynamoDB was failing, but each service was failing in its own unique way. Customers were seeing their own infrastructure as the problem when the actual failure was completely invisible to them.

The Root Cause: A Race Condition in DNS Management

The failure originated in DynamoDB's internal automated DNS management system, which was designed with two independent components for high availability:

The race condition unfolded like this:

This wasn't a Route 53 bug. Route 53 faithfully executed what it was told to do. The bug was in the system that told Route 53 what to do. Once the race condition fired, the system was left in an inconsistent state that prevented any further automated DNS updates.

The Cascade: How One DNS Failure Became a Multi-Hour Outage

What made this incident particularly devastating wasn't just the initial DNS failure—it was how that failure cascaded through AWS's interconnected systems:

What Customers Experienced vs. What Was Actually Happening

The most challenging aspect of this incident was that customers' own monitoring showed their infrastructure as the problem:

DNS Deep Dive: The Technical Lessons

Why DNS Is the Most Dangerous Single Point of Failure

DNS failures are uniquely catastrophic in distributed systems because:

• Negative Caching: NXDOMAIN responses get cached for the TTL duration
• Universal Dependency: Every network connection starts with DNS resolution
• Invisible Failure Mode: Applications see connection timeouts, not DNS failures

The Thundering Herd Problem at AWS Scale

When DynamoDB recovered, DWFM tried to re-establish leases for the entire EC2 fleet simultaneously. This is a textbook example of congestive collapse:

Recovery Load > System Capacity
→ Requests timeout faster than they can be processed
→ Clients retry, increasing load further
→ System becomes less available during recovery than during outage

The "Cleanup as a Weapon" Anti-Pattern

The cleanup process that deleted "stale" DNS plans was doing the right thing locally but the wrong thing globally. In distributed systems, cleanup operations that can't be rolled back are dangerous.

What Good Disaster Recovery Looked Like

During this incident, some customers were completely unaffected. Here's what worked:

• Multi-region active-active architectures: No single region dependency
• DynamoDB Global Tables: Could fail over reads to other regions
• Route 53 health checks + failover routing: Automatically routed to healthy regions
• ElastiCache in front of DynamoDB: Cache absorbed reads during outage
• Circuit breakers in application code: Failed fast instead of hanging
• Hardcoded IPs (ironically): Some customers with this bad practice worked during DNS failure

AWS's Response and What Changed

AWS's immediate response was decisive but drastic:

• Disabled DynamoDB DNS Planner and DNS Enactor automation worldwide
• Switched to manual DNS management until safeguards could be implemented
• Conducted a comprehensive review of all automated DNS management systems

In November 2025, AWS introduced the DNS failover feature:

• Services can specify backup DNS records that activate automatically
• Invariant checking prevents DNS records from ever being set to empty
• Staggered recovery mechanisms prevent thundering herd problems

Personal Takeaways

Working customer-facing triage during this incident taught me several critical lessons:

The Hardest Incidents Are Invisible Ones

When customers can see the failure (server down, network partition), they understand what's happening. When the failure is invisible to them but their applications are broken, trust erodes quickly.

DNS Failures Are Uniquely Bad

Unlike most failures, DNS failures are cached. Fixing the root cause doesn't immediately fix the symptom. This creates a secondary wave of customer confusion when their "fixes" don't work.

At AWS Scale, Recovery Is a Distributed Systems Problem

The DWFM congestive collapse showed that at sufficient scale, even recovery becomes a distributed systems challenge requiring careful orchestration.

The Value of True Multi-Region Architecture

This incident reinforced that multi-region isn't just about protecting against natural disasters or AZ failures—it's about protecting against any single-region dependency, including ones inside the cloud provider itself.