{"id":2879,"date":"2025-11-26T01:16:05","date_gmt":"2025-11-25T19:46:05","guid":{"rendered":"https:\/\/www.getpanto.ai\/blog\/?p=2879"},"modified":"2025-11-26T01:17:19","modified_gmt":"2025-11-25T19:47:19","slug":"cloudflare-outage","status":"publish","type":"post","link":"https:\/\/www.getpanto.ai\/blog\/cloudflare-outage","title":{"rendered":"Cloudflare Outage: How One Feature File Took Down the Network"},"content":{"rendered":"\n

<\/p>\n\n\n\n

Just weeks after AWS disclosed a major DynamoDB outage<\/a> in late October 2025 (triggered by a latent DNS-management race condition), Cloudflare suffered its own large-scale outage on November 18, 2025. Both incidents underscore how fragile modern cloud systems can become when small bugs combine with high deployment velocity. <\/p>\n\n\n\n

Some experts have dubbed today\u2019s fast, AI-assisted \u201cvibe coding<\/a>\u201d culture, which emphasizes shipping new features quickly as trading away robustness for speed. The Cloudflare outage postmortem makes this clear: a seemingly innocuous database permission change cascaded into a network-wide failure<\/strong>. <\/p>\n\n\n\n

\"Cloudflare<\/figure>\n\n\n\n

Cloudflare\u2019s report <\/a>confirms no malicious attack was involved. Instead, on Nov 18 at 11:20\u202fUTC<\/strong>, our network began seeing \u201csignificant failures to deliver core network traffic,\u201d with users getting the above error. <\/p>\n\n\n\n

The culprit was a database change: a ClickHouse permission update at 11:05<\/strong> caused the Bot Management feature-file query to double its output (via duplicate rows). That oversized file violated an internal 200-feature limit (set for performance), causing a Rust \u201cunwrap\u201d panic in Cloudflare<\/a>\u2019s new FL2 proxy code and triggering HTTP 5xx errors across the CDN.<\/p>\n\n\n

<\/span>Cloudflare Outage Timeline and Sequence of Events<\/strong><\/span><\/h2>\n\n\n

Key events unfolded as follows:<\/p>\n\n\n\n

\"Timeline<\/figure>\n\n\n\n

<\/p>\n\n\n\n

    \n
  1. 11:05\u202fUTC<\/strong> \u2013 Cloudflare deploys a ClickHouse access control change<\/em> to expose underlying shard metadata.
    <\/li>\n\n\n\n
  2. 11:28<\/strong> \u2013 The first errors appear on customer HTTP traffic as the new Bot Management feature file starts propagating. (Earlier, system metrics <\/a>were normal.)
    <\/li>\n\n\n\n
  3. 11:31\u201311:35<\/strong> \u2013 An internal monitoring test triggers an alert at ~11:31, and an incident call is convened by 11:35. Engineers initially see degraded response rates in Workers KV and attempt mitigations (rate limiting, traffic shaping).
    <\/li>\n\n\n\n
  4. 13:05<\/strong> \u2013 To reduce pressure, Teams bypass the core proxy for Workers KV and Access, falling back to an older proxy version. This lessens the downstream impact (the bug<\/a> was still present, but older code masked it).
    <\/li>\n\n\n\n
  5. 13:37<\/strong> \u2013 The Bot Management configuration file is identified as the trigger. Multiple workstreams begin. The fastest path: restore the last-known-good version of the file.
    <\/li>\n\n\n\n
  6. 14:24<\/strong> \u2013 New (bad) feature-file generation is halted. A tested, correct feature file is manually pushed out to all nodes.
    <\/li>\n\n\n\n
  7. 14:30<\/strong> \u2013 Core services begin recovering. A correct Bot Management file is deployed globally; HTTP 5xx error rates drop sharply. (Cloudflare reports \u201ccore traffic was largely flowing as normal by 14:30″.)
    <\/li>\n\n\n\n
  8. By 17:06<\/strong> \u2013 All remaining systems have been restarted or recovered, and traffic returns to normal levels. (The long tail on the errors chart reflects final service restarts.)
    <\/li>\n<\/ol>\n\n\n\n

    This timeline shows the unusual \u201czig-zag\u201d spike in errors every ~5 minutes as good and bad feature files alternated. Eventually the bad file prevailed and errors stabilized until remediation took effect.<\/p>\n\n\n

    <\/span>Deep Dive into the CloudFlare Outage<\/strong><\/span><\/h2>\n\n

    <\/span>Root Cause: Bot-Management Feature File and Query Bug<\/strong><\/span><\/h3>\n\n\n

    Cloudflare\u2019s Bot Management module embeds a reinforcement-learning<\/a> model that scores each request as bot or human. This ML model consumes a \u201cfeature\u201d configuration file (a list of request-trait parameters). The feature file is regenerated every few minutes via a ClickHouse query so that it stays up-to-date.<\/p>\n\n\n\n

    On Nov 18, the scheduled ClickHouse update at 11:05 changed query semantics: it granted explicit access to underlying shard tables in the r0 schema. In prior behavior, querying metadata returned only the default schema\u2019s columns by assumption; after the change, the same query now returned columns from r0 too.<\/p>\n\n\n\n

    In practice, the Bot feature-generation query (a system.columns lookup) suddenly returned all<\/strong> columns from both default and r0 tables. Concretely, the report notes that “the response now contained all the metadata of the r0 schema, effectively more than doubling the rows”<\/em> in the resulting feature file. <\/p>\n\n\n\n

    In other words, a permission tweak caused duplicate<\/a> feature entries. The new feature file exceeded Cloudflare\u2019s built-in limit of 200 features. As soon as the bloated file hit production (in the FL2 proxy code path), the proxy panicked:<\/p>\n\n\n\n

    thread fl2_worker_thread panicked: called Result::unwrap() on an Err value<\/code><\/pre>\n\n\n\n
    This unhandled Rust panic (during feature-file parsing) turned into HTTP 500 errors for any request going through the bot-management path. In summary, a simple DB query change made the ML feature config file twice as large, which triggered an internal error and caused core requests to fail.<\/p>\n\n\n
    <\/span>Failure Propagation and Impact Across Systems<\/strong><\/span><\/h3>\n\n\n
    Once the error occurred, it rippled through most Cloudflare services that depended on the core proxy or bot logic. The effect on customers varied depending on their proxy version: Cloudflare was midway through migrating traffic to a new proxy engine (FL2). <\/p>\n\n\n\n
    Per the report, customers on FL2 all<\/em> saw 5xx errors, while customers still on the legacy proxy (FL) saw no crashes \u2013 but all bot scores defaulted to zero. This meant any customer using bot scores to block<\/em> bots would suddenly drop all traffic (false positives). In short:<\/p>\n\n\n\n
    <\/p>\n\n\n\n