The outage progressed in distinct phases. AWS\u2019s official postmortem and independent analyses detail the sequence of events:<\/p>\n\n\n
<\/span>How the AWS outage unfolded<\/strong><\/span><\/h3>\n\n\n![\"\"](\"https:\/\/www.getpanto.ai\/blog\/wp-content\/uploads\/2025\/10\/aws-outage-timeline.png\")
<\/figure>\n\n\n\n\n- Oct 19, 11:48 PM PDT (Oct 20, 6:48 AM UTC)<\/strong> \u2013 DynamoDB DNS failure begins.<\/em> Customers in us-east-1 and internal AWS systems started seeing increased DynamoDB API error rates. The root trigger was that the DynamoDB service\u2019s regional endpoint (dynamodb.us-east-1.amazonaws.com) could no longer be resolved via DNS. All requests that needed DynamoDB began failing immediately.
<\/li>\n\n\n\n- Oct 20, ~12:38 AM PDT<\/strong> \u2013 Root cause identified.<\/em> AWS engineers traced the problem to DynamoDB\u2019s DNS records by about 12:38 AM. They realized the public endpoint had been assigned an empty or invalid DNS record.
<\/li>\n\n\n\n- Oct 20, 1:15 AM PDT<\/strong> \u2013 Mitigation applied.<\/em> By ~1:15 AM, AWS applied temporary fixes that restored some internal access to DynamoDB and key tooling, easing the investigation. This involved manually<\/a> pushing correct DNS entries so that internal AWS services could again reach DynamoDB.
<\/li>\n\n\n\n- Oct 20, 2:25 AM PDT<\/strong> \u2013 DNS fully restored.<\/em> At 2:25 AM, AWS engineers re-established the correct DNS records for the DynamoDB endpoint. Over the next ~15 minutes, as customers\u2019 DNS caches expired, external applications regained the ability to resolve and connect to DynamoDB. By 2:40 AM, most clients could successfully connect again, marking primary recovery of the database service.
<\/li>\n\n\n\n- Oct 20, 2:25 AM onward<\/strong> \u2013 Cascading EC2 failures.<\/em> Even after DynamoDB was reachable, related services remained impaired. The EC2 instance launch system had partially broken down (see below). From this point until late morning, new EC2 instances in us-east-1 failed with \u201cinsufficient capacity\u201d or limit errors.
<\/li>\n\n\n\n- Oct 20, 5:28 AM PDT<\/strong> \u2013 EC2 control-plane recovery.<\/em> By 5:28 AM, AWS completed manual resets and throttling of the EC2 Droplet Workflow <\/a>Manager (DWFM), which allowed instance launches to start succeeding again. However, a backlog of pending tasks and networking updates remained.
<\/li>\n\n\n\n- Oct 20, 5:30 AM \u2013 2:09 PM PDT<\/strong> \u2013 NLB and network issues.<\/em> Between ~5:30 AM and ~2:09 PM, customers saw increased errors on Network Load Balancers (NLBs) in us-east-1. This was due to a flood of delayed network state updates for newly launched instances, causing health-check failures. AWS introduced temporary fixes (like disabling automatic failovers) to stabilize NLBs.
<\/li>\n\n\n\n- Oct 20, 1:50 PM PDT<\/strong> \u2013 Service fully restored.<\/em> By early Monday afternoon, EC2 and most other services reported normal operation. AWS noted that by 1:50 PM PDT, all EC2 APIs and instance launches were back to normal. Other dependent services finished processing backlogs over the next day.
<\/li>\n<\/ol>\n\n\n\nIn summary, the initial DynamoDB DNS problem lasted only a few hours, but its side-effects on EC2, networking, and higher-level services prolonged the full recovery into the next day.<\/p>\n\n\n\n![\"AWS](\"https:\/\/www.getpanto.ai\/blog\/wp-content\/uploads\/2025\/10\/image-47.png\")
<\/figure>\n\n\n<\/span>Root Cause of AWS outage: DNS Race Condition in DynamoDB<\/strong><\/span><\/h2>\n\n\nAWS confirmed that the outage originated in DynamoDB\u2019s DNS management system. DynamoDB scales by dynamically managing hundreds of thousands of DNS records (via Amazon Route 53) for its load balancers. Automation must frequently update these records to add capacity or remove failed nodes. Crucially, AWS uses two distributed components: a DNS Planner<\/em> that generates updated DNS \u201cplans\u201d for each endpoint, and multiple DNS Enactors<\/em> in different Availability Zones that apply those plans via atomic Route 53 transactions.<\/p>\n\n\n<\/span>Official AWS Postmortem<\/strong><\/span><\/h3>\n\n\nAccording to the official AWS postmortem, the AWS DNS outage<\/strong> in 2025 was caused by a latent race condition between two DNS Enactor processes. One Enactor became unusually slow because it had to retry multiple transactions and continued using an outdated DNS plan for a regional endpoint. At the same time, the Planner created new DNS plans that a second, faster Enactor quickly applied across all AWS endpoints. After finishing its updates, the fast Enactor initiated a cleanup routine that deleted older DNS plans across the system.<\/p>\n\n\n\nUnfortunately, the timing caused a major failure. Just as the slow Enactor finally applied its outdated plan to the DynamoDB endpoint<\/strong>, it was immediately removed by the cleanup process for being several generations behind. This deletion removed all IP addresses for dynamodb.us-east-1.amazonaws.com<\/strong>, leaving the DNS entry completely empty. As a result, the automated<\/a> system stopped accepting new updates, forcing AWS engineers to perform manual intervention to rebuild and restore the correct DNS record.<\/p>\n\n\n<\/span>What does this mean?<\/strong><\/span><\/h3>\n\n\nIn practical terms, this meant that when the outage began, no client could resolve the DynamoDB hostname. Every DynamoDB API call in us-east-1 immediately failed with resolution errors. In short, a software bug in AWS\u2019s internal DNS automation left the service invisible on the network.<\/p>\n\n\n\n
\n\n\n<\/span>Cascade: How the Failure Spun Out Across AWS<\/strong><\/span><\/h2>\n\n\nWith DynamoDB unreachable, dependent systems began to break down in turn. The chain of failures unfolded as follows:<\/p>\n\n\n\n
\n- EC2 Droplet Workflow Manager (DWFM):<\/strong> EC2 uses a subsystem called DWFM to track leases on physical servers (\u201cdroplets\u201d) that host virtual machines. DWFM regularly checks in with DynamoDB to maintain the state of each droplet. When DynamoDB went offline, those state checks started timing out. Although running EC2 instances remained up, any droplet that needed to renew its lease or change state couldn\u2019t do so. Over a couple of hours, thousands of droplet leases expired. Once DynamoDB was restored, DWFM attempted to re-establish all leases across the fleet. Because the accumulated backlog was huge, the system entered a \u201ccongestive collapse\u201d and was unable to make forward progress. Engineers eventually had to throttle incoming work and restart many DWFM hosts around 5:28 AM. This cleared the queues and allowed leases to be re-established. Only then did new EC2 instance launches start to succeed again.
<\/li>\n\n\n\n- Network Configuration Propagation:<\/strong> Separately, AWS\u2019s Network Manager subsystem had built up a backlog of pending network configuration<\/a> changes (to ensure new instances can communicate on the VPC network). After the DynamoDB fix, Network Manager began propagating all the delayed updates to newly launched EC2 instances and virtual networking appliances. Because so many instances had been launched out-of-order, this propagation took time. Until it caught up, many new instances lacked full connectivity.
<\/li>\n\n\n\n- Network Load Balancers (NLBs):<\/strong> The delayed network configuration in turn caused NLBs to misinterpret instance health. NLBs constantly perform health checks on their backend targets (typically EC2 instances) and automatically remove any instance that fails health checks. In this event, new EC2 instances that had just started up were not fully network-enabled, so some health checks were failing. The NLB health-check system temporarily pulled these instances out of service, then put them back when the checks eventually passed. As a result, between about 5:30 AM and early afternoon, customers saw a surge in \u201cconnection errors\u201d on their NLBs in us-east-1. AWS disabled automatic health-check failovers and limited the rate of NLB capacity removal to stabilize the situation.
<\/li>\n\n\n\n- Other AWS Services:<\/strong> The EC2 and networking disruptions affected many higher-level services. For example, container and orchestration platforms (ECS\/EKS), AWS Lambda, and serverless operations all rely on EC2 resources and DynamoDB in the background, so they experienced failures<\/a> or throttle limits. Amazon Redshift (data warehousing) and AWS Security Token Service (STS) faced delays too, because they had to retry workflows when underlying EC2 instance launches initially failed. In practice, many AWS teams spent Monday night working through backlogs of failed operations: for some services, the final cleanup extended into the next day.
<\/li>\n<\/ul>\n\n\n\nIn summary, a single DNS failure in DynamoDB caused a domino effect: it broke EC2\u2019s orchestration, which in turn broke networking and higher-level services. AWS had to recover in phases, first fixing the DNS, then clearing the dependent system backlogs, then processing queued work, and finally allowing customer applications to fully recover.<\/p>\n\n\n\n
\n\n\n<\/span>Global Impact of AWS outage<\/strong> on Applications<\/strong><\/span><\/h2>\n\n\nThe outage <\/a>in us-east-1 had widespread customer impact. DynamoDB powers many AWS-hosted applications, so its outage disrupted thousands of websites and apps. During the incident, over four million users reported errors, and the downtime impacted at least a thousand companies across various industries.<\/p>\n\n\n<\/span>Regions<\/strong><\/span><\/h3>\n\n\nIn the US and abroad, the outage knocked workers from London to Tokyo offline. Users complained of failures in everyday services \u2013 for example, digital payment app Venmo and video-call service Zoom both had outages or errors reported by Monday afternoon.<\/p>\n\n\n\n![\"Major](\"https:\/\/www.getpanto.ai\/blog\/wp-content\/uploads\/2025\/10\/image-45.png\")
<\/figure>\n\n\n<\/span>Sectors<\/strong><\/span><\/h3>\n\n\nHigh-profile consumer services spanning social media, gaming, finance, and enterprise all reported disruptions. Apps like Reddit, Snapchat, Roblox, and Duolingo\u00a0had all been affected. Gaming and crypto were hit too \u2013 Fortnite, Clash Royale\/Clans, Perplexity AI<\/a>, Coinbase, and Robinhood all cited AWS outages. <\/p>\n\n\n<\/span>Amazon’s own services<\/strong><\/span><\/h3>\n\n\nEven Amazon\u2019s own properties \u2013 its retail website, Prime Video streaming, and Alexa voice service \u2013 saw interruptions. In effect, platforms \u201cfrom Snapchat to Venmo\u201d went dark for some users during the outage. Analysts noted that for major businesses, hours of cloud downtime translate to millions in lost productivity <\/a>and revenue, highlighting the economic scale of the disruption.<\/p>\n\n\n\nAWS acknowledged the broad effects. This was the third major AWS US-EAST-1 outage in five years, reflecting how a problem in this central region can trigger a global knock-on effect. The outage underscored the interdependence of modern online services on a few cloud providers. Experts noted that it highlights the dependency <\/a>we have on relatively fragile infrastructures.<\/p>\n\n\n\n
\n\n\n<\/span>AWS Response and Mitigation<\/strong><\/span><\/h2>\n\n\nAWS released an official post-incident report describing the cause and its remediation steps. In brief, AWS has disabled the faulty DNS automation <\/a>and is deploying multiple safeguards. The DynamoDB DNS Planner and Enactor processes have been turned off worldwide until the bug is fixed. AWS plans to correct the race condition in the system and add protections to block any \u201cincorrect DNS plans\u201d from being applied.<\/p>\n\n\n\nIn addition, AWS will strengthen testing <\/a>and throttles in related systems: for EC2, it is adding tests of the DWFM recovery path and improving throttling of instance-launch requests; for NLB, it will limit how rapidly an individual balancer can shed capacity when health checks fail. These measures aim to prevent a similar bug from bringing down a critical endpoint again and to better contain cascading failures if one occurs.<\/p>\n\n\n\n![\"AWS](\"https:\/\/www.getpanto.ai\/blog\/wp-content\/uploads\/2025\/10\/image-48.png\")
<\/figure>\n\n\n\nAWS also apologized to customers and said it is learning from this event to improve availability. The company emphasized its track record of high availability but acknowledged the severity of this outage. The immediate fixes included temporarily disabling the DynamoDB DNS Planner, adding additional protections to prevent incorrect DNS entries, and improving testing metrics <\/a>to catch such errors before they can cause extended outages.<\/p>\n\n\n\n
\n\n\n<\/span>Conclusion<\/strong><\/span><\/h2>\n\n\nThe AWS DynamoDB outage in October 2025<\/strong> showed how a small software bug can trigger massive cloud disruption. A hidden race condition in AWS\u2019s DNS automation deleted all addresses for the DynamoDB endpoint, instantly cutting off apps that depended on the database. The failure spread quickly through AWS infrastructure<\/strong>, halting EC2 provisioning, disrupting load balancers, and freezing higher-level services. Engineers had to manually rebuild DNS records and throttle requests before restoring normal operations.<\/p>\n\n\n\nThis AWS outage<\/strong> revealed how fragile large-scale cloud systems can be when core dependencies fail. AWS\u2019s postmortem detailed the root cause and outlined fixes to prevent future incidents. While services recovered by Monday afternoon, many organizations spent hours clearing backlogs of failed requests. The AWS outage<\/strong> reminded businesses worldwide that even mature cloud platforms can experience cascading failures \u2014 and why ongoing resilience testing<\/a> is essential in modern cloud operations.<\/p>\n","protected":false},"excerpt":{"rendered":"On Oct-19-2025, a serious outage struck Amazon Web Services\u2019 DynamoDB in the US-EAST-1 (Northern Virginia) region. According to AWS, the AWS outage began at 11:48 PM PDT on October 19 and unfolded over roughly 15 hours. During this time, the DynamoDB database in N. Virginia became unreachable, triggering cascading failures across many AWS services and […]<\/p>\n","protected":false},"author":3,"featured_media":2148,"comment_status":"open","ping_status":"open","sticky":false,"template":"wp-custom-template-test-blog","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2139","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-coding"],"_links":{"self":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/posts\/2139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/comments?post=2139"}],"version-history":[{"count":0,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/posts\/2139\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/media\/2148"}],"wp:attachment":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/media?parent=2139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/categories?post=2139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/tags?post=2139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
<\/figure>\n\n\n\n- \n
- Oct 19, 11:48 PM PDT (Oct 20, 6:48 AM UTC)<\/strong> \u2013 DynamoDB DNS failure begins.<\/em> Customers in us-east-1 and internal AWS systems started seeing increased DynamoDB API error rates. The root trigger was that the DynamoDB service\u2019s regional endpoint (dynamodb.us-east-1.amazonaws.com) could no longer be resolved via DNS. All requests that needed DynamoDB began failing immediately.
<\/li>\n\n\n\n- Oct 20, ~12:38 AM PDT<\/strong> \u2013 Root cause identified.<\/em> AWS engineers traced the problem to DynamoDB\u2019s DNS records by about 12:38 AM. They realized the public endpoint had been assigned an empty or invalid DNS record.
<\/li>\n\n\n\n- Oct 20, 1:15 AM PDT<\/strong> \u2013 Mitigation applied.<\/em> By ~1:15 AM, AWS applied temporary fixes that restored some internal access to DynamoDB and key tooling, easing the investigation. This involved manually<\/a> pushing correct DNS entries so that internal AWS services could again reach DynamoDB.
<\/li>\n\n\n\n- Oct 20, 2:25 AM PDT<\/strong> \u2013 DNS fully restored.<\/em> At 2:25 AM, AWS engineers re-established the correct DNS records for the DynamoDB endpoint. Over the next ~15 minutes, as customers\u2019 DNS caches expired, external applications regained the ability to resolve and connect to DynamoDB. By 2:40 AM, most clients could successfully connect again, marking primary recovery of the database service.
<\/li>\n\n\n\n- Oct 20, 2:25 AM onward<\/strong> \u2013 Cascading EC2 failures.<\/em> Even after DynamoDB was reachable, related services remained impaired. The EC2 instance launch system had partially broken down (see below). From this point until late morning, new EC2 instances in us-east-1 failed with \u201cinsufficient capacity\u201d or limit errors.
<\/li>\n\n\n\n- Oct 20, 5:28 AM PDT<\/strong> \u2013 EC2 control-plane recovery.<\/em> By 5:28 AM, AWS completed manual resets and throttling of the EC2 Droplet Workflow <\/a>Manager (DWFM), which allowed instance launches to start succeeding again. However, a backlog of pending tasks and networking updates remained.
<\/li>\n\n\n\n- Oct 20, 5:30 AM \u2013 2:09 PM PDT<\/strong> \u2013 NLB and network issues.<\/em> Between ~5:30 AM and ~2:09 PM, customers saw increased errors on Network Load Balancers (NLBs) in us-east-1. This was due to a flood of delayed network state updates for newly launched instances, causing health-check failures. AWS introduced temporary fixes (like disabling automatic failovers) to stabilize NLBs.
<\/li>\n\n\n\n- Oct 20, 1:50 PM PDT<\/strong> \u2013 Service fully restored.<\/em> By early Monday afternoon, EC2 and most other services reported normal operation. AWS noted that by 1:50 PM PDT, all EC2 APIs and instance launches were back to normal. Other dependent services finished processing backlogs over the next day.
<\/li>\n<\/ol>\n\n\n\nIn summary, the initial DynamoDB DNS problem lasted only a few hours, but its side-effects on EC2, networking, and higher-level services prolonged the full recovery into the next day.<\/p>\n\n\n\n
<\/figure>\n\n\n<\/span>Root Cause of AWS outage: DNS Race Condition in DynamoDB<\/strong><\/span><\/h2>\n\n\n
AWS confirmed that the outage originated in DynamoDB\u2019s DNS management system. DynamoDB scales by dynamically managing hundreds of thousands of DNS records (via Amazon Route 53) for its load balancers. Automation must frequently update these records to add capacity or remove failed nodes. Crucially, AWS uses two distributed components: a DNS Planner<\/em> that generates updated DNS \u201cplans\u201d for each endpoint, and multiple DNS Enactors<\/em> in different Availability Zones that apply those plans via atomic Route 53 transactions.<\/p>\n\n\n
<\/span>Official AWS Postmortem<\/strong><\/span><\/h3>\n\n\n
According to the official AWS postmortem, the AWS DNS outage<\/strong> in 2025 was caused by a latent race condition between two DNS Enactor processes. One Enactor became unusually slow because it had to retry multiple transactions and continued using an outdated DNS plan for a regional endpoint. At the same time, the Planner created new DNS plans that a second, faster Enactor quickly applied across all AWS endpoints. After finishing its updates, the fast Enactor initiated a cleanup routine that deleted older DNS plans across the system.<\/p>\n\n\n\n
Unfortunately, the timing caused a major failure. Just as the slow Enactor finally applied its outdated plan to the DynamoDB endpoint<\/strong>, it was immediately removed by the cleanup process for being several generations behind. This deletion removed all IP addresses for dynamodb.us-east-1.amazonaws.com<\/strong>, leaving the DNS entry completely empty. As a result, the automated<\/a> system stopped accepting new updates, forcing AWS engineers to perform manual intervention to rebuild and restore the correct DNS record.<\/p>\n\n\n
<\/span>What does this mean?<\/strong><\/span><\/h3>\n\n\n
In practical terms, this meant that when the outage began, no client could resolve the DynamoDB hostname. Every DynamoDB API call in us-east-1 immediately failed with resolution errors. In short, a software bug in AWS\u2019s internal DNS automation left the service invisible on the network.<\/p>\n\n\n\n
\n\n\n<\/span>Cascade: How the Failure Spun Out Across AWS<\/strong><\/span><\/h2>\n\n\n
With DynamoDB unreachable, dependent systems began to break down in turn. The chain of failures unfolded as follows:<\/p>\n\n\n\n
- \n
- EC2 Droplet Workflow Manager (DWFM):<\/strong> EC2 uses a subsystem called DWFM to track leases on physical servers (\u201cdroplets\u201d) that host virtual machines. DWFM regularly checks in with DynamoDB to maintain the state of each droplet. When DynamoDB went offline, those state checks started timing out. Although running EC2 instances remained up, any droplet that needed to renew its lease or change state couldn\u2019t do so. Over a couple of hours, thousands of droplet leases expired. Once DynamoDB was restored, DWFM attempted to re-establish all leases across the fleet. Because the accumulated backlog was huge, the system entered a \u201ccongestive collapse\u201d and was unable to make forward progress. Engineers eventually had to throttle incoming work and restart many DWFM hosts around 5:28 AM. This cleared the queues and allowed leases to be re-established. Only then did new EC2 instance launches start to succeed again.
<\/li>\n\n\n\n- Network Configuration Propagation:<\/strong> Separately, AWS\u2019s Network Manager subsystem had built up a backlog of pending network configuration<\/a> changes (to ensure new instances can communicate on the VPC network). After the DynamoDB fix, Network Manager began propagating all the delayed updates to newly launched EC2 instances and virtual networking appliances. Because so many instances had been launched out-of-order, this propagation took time. Until it caught up, many new instances lacked full connectivity.
<\/li>\n\n\n\n- Network Load Balancers (NLBs):<\/strong> The delayed network configuration in turn caused NLBs to misinterpret instance health. NLBs constantly perform health checks on their backend targets (typically EC2 instances) and automatically remove any instance that fails health checks. In this event, new EC2 instances that had just started up were not fully network-enabled, so some health checks were failing. The NLB health-check system temporarily pulled these instances out of service, then put them back when the checks eventually passed. As a result, between about 5:30 AM and early afternoon, customers saw a surge in \u201cconnection errors\u201d on their NLBs in us-east-1. AWS disabled automatic health-check failovers and limited the rate of NLB capacity removal to stabilize the situation.
<\/li>\n\n\n\n- Other AWS Services:<\/strong> The EC2 and networking disruptions affected many higher-level services. For example, container and orchestration platforms (ECS\/EKS), AWS Lambda, and serverless operations all rely on EC2 resources and DynamoDB in the background, so they experienced failures<\/a> or throttle limits. Amazon Redshift (data warehousing) and AWS Security Token Service (STS) faced delays too, because they had to retry workflows when underlying EC2 instance launches initially failed. In practice, many AWS teams spent Monday night working through backlogs of failed operations: for some services, the final cleanup extended into the next day.
<\/li>\n<\/ul>\n\n\n\nIn summary, a single DNS failure in DynamoDB caused a domino effect: it broke EC2\u2019s orchestration, which in turn broke networking and higher-level services. AWS had to recover in phases, first fixing the DNS, then clearing the dependent system backlogs, then processing queued work, and finally allowing customer applications to fully recover.<\/p>\n\n\n\n
\n\n\n<\/span>Global Impact of AWS outage<\/strong> on Applications<\/strong><\/span><\/h2>\n\n\n
The outage <\/a>in us-east-1 had widespread customer impact. DynamoDB powers many AWS-hosted applications, so its outage disrupted thousands of websites and apps. During the incident, over four million users reported errors, and the downtime impacted at least a thousand companies across various industries.<\/p>\n\n\n
<\/span>Regions<\/strong><\/span><\/h3>\n\n\n
In the US and abroad, the outage knocked workers from London to Tokyo offline. Users complained of failures in everyday services \u2013 for example, digital payment app Venmo and video-call service Zoom both had outages or errors reported by Monday afternoon.<\/p>\n\n\n\n
<\/figure>\n\n\n<\/span>Sectors<\/strong><\/span><\/h3>\n\n\n
High-profile consumer services spanning social media, gaming, finance, and enterprise all reported disruptions. Apps like Reddit, Snapchat, Roblox, and Duolingo\u00a0had all been affected. Gaming and crypto were hit too \u2013 Fortnite, Clash Royale\/Clans, Perplexity AI<\/a>, Coinbase, and Robinhood all cited AWS outages. <\/p>\n\n\n
<\/span>Amazon’s own services<\/strong><\/span><\/h3>\n\n\n
Even Amazon\u2019s own properties \u2013 its retail website, Prime Video streaming, and Alexa voice service \u2013 saw interruptions. In effect, platforms \u201cfrom Snapchat to Venmo\u201d went dark for some users during the outage. Analysts noted that for major businesses, hours of cloud downtime translate to millions in lost productivity <\/a>and revenue, highlighting the economic scale of the disruption.<\/p>\n\n\n\n
AWS acknowledged the broad effects. This was the third major AWS US-EAST-1 outage in five years, reflecting how a problem in this central region can trigger a global knock-on effect. The outage underscored the interdependence of modern online services on a few cloud providers. Experts noted that it highlights the dependency <\/a>we have on relatively fragile infrastructures.<\/p>\n\n\n\n
\n\n\n<\/span>AWS Response and Mitigation<\/strong><\/span><\/h2>\n\n\n
AWS released an official post-incident report describing the cause and its remediation steps. In brief, AWS has disabled the faulty DNS automation <\/a>and is deploying multiple safeguards. The DynamoDB DNS Planner and Enactor processes have been turned off worldwide until the bug is fixed. AWS plans to correct the race condition in the system and add protections to block any \u201cincorrect DNS plans\u201d from being applied.<\/p>\n\n\n\n
In addition, AWS will strengthen testing <\/a>and throttles in related systems: for EC2, it is adding tests of the DWFM recovery path and improving throttling of instance-launch requests; for NLB, it will limit how rapidly an individual balancer can shed capacity when health checks fail. These measures aim to prevent a similar bug from bringing down a critical endpoint again and to better contain cascading failures if one occurs.<\/p>\n\n\n\n
<\/figure>\n\n\n\nAWS also apologized to customers and said it is learning from this event to improve availability. The company emphasized its track record of high availability but acknowledged the severity of this outage. The immediate fixes included temporarily disabling the DynamoDB DNS Planner, adding additional protections to prevent incorrect DNS entries, and improving testing metrics <\/a>to catch such errors before they can cause extended outages.<\/p>\n\n\n\n
\n\n\n<\/span>Conclusion<\/strong><\/span><\/h2>\n\n\n
The AWS DynamoDB outage in October 2025<\/strong> showed how a small software bug can trigger massive cloud disruption. A hidden race condition in AWS\u2019s DNS automation deleted all addresses for the DynamoDB endpoint, instantly cutting off apps that depended on the database. The failure spread quickly through AWS infrastructure<\/strong>, halting EC2 provisioning, disrupting load balancers, and freezing higher-level services. Engineers had to manually rebuild DNS records and throttle requests before restoring normal operations.<\/p>\n\n\n\n
This AWS outage<\/strong> revealed how fragile large-scale cloud systems can be when core dependencies fail. AWS\u2019s postmortem detailed the root cause and outlined fixes to prevent future incidents. While services recovered by Monday afternoon, many organizations spent hours clearing backlogs of failed requests. The AWS outage<\/strong> reminded businesses worldwide that even mature cloud platforms can experience cascading failures \u2014 and why ongoing resilience testing<\/a> is essential in modern cloud operations.<\/p>\n","protected":false},"excerpt":{"rendered":"
On Oct-19-2025, a serious outage struck Amazon Web Services\u2019 DynamoDB in the US-EAST-1 (Northern Virginia) region. According to AWS, the AWS outage began at 11:48 PM PDT on October 19 and unfolded over roughly 15 hours. During this time, the DynamoDB database in N. Virginia became unreachable, triggering cascading failures across many AWS services and […]<\/p>\n","protected":false},"author":3,"featured_media":2148,"comment_status":"open","ping_status":"open","sticky":false,"template":"wp-custom-template-test-blog","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2139","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-coding"],"_links":{"self":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/posts\/2139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/comments?post=2139"}],"version-history":[{"count":0,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/posts\/2139\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/media\/2148"}],"wp:attachment":[{"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/media?parent=2139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/categories?post=2139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.getpanto.ai\/blog\/wp-json\/wp\/v2\/tags?post=2139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Network Configuration Propagation:<\/strong> Separately, AWS\u2019s Network Manager subsystem had built up a backlog of pending network configuration<\/a> changes (to ensure new instances can communicate on the VPC network). After the DynamoDB fix, Network Manager began propagating all the delayed updates to newly launched EC2 instances and virtual networking appliances. Because so many instances had been launched out-of-order, this propagation took time. Until it caught up, many new instances lacked full connectivity.
- Oct 20, ~12:38 AM PDT<\/strong> \u2013 Root cause identified.<\/em> AWS engineers traced the problem to DynamoDB\u2019s DNS records by about 12:38 AM. They realized the public endpoint had been assigned an empty or invalid DNS record.