GitHub · Distributed Systems · 31 May 2026
\nBetween May 2025 and April 2026, GitHub experienced 257 incidents — 48 of them major outages. That's roughly one significant disruption every single week. The culprit wasn't a security breach, a botched deployment, or a rogue engineer. It was the thing GitHub had spent years celebrating: AI. Specifically, agentic AI workflows that turned one human developer's footprint into hundreds of commits, thousands of CI minutes, and a dozen simultaneous PR operations — all at once, across millions of accounts. GitHub had been built for humans. Agents are not human.
\n\nWe started executing our plan to increase GitHub's capacity by 10X in October 2025, with a goal of substantially improving reliability and failover. By February 2026, it was clear that we needed to design for a future that requires 30X today's scale. The main driver is a rapid change in how software is being built. Since the second half of December 2025, agentic development workflows have accelerated sharply.
— — Vlad Fedorov, CTO of GitHub — GitHub Engineering Blog, April 28, 2026
For most of its existence, GitHub has been one of the most reliable platforms on the internet. Developers took it for granted the way they take electricity for granted — always on, always there, a utility so dependable it disappeared into the background. That changed in 2025. Not because GitHub's engineers got worse. Not because the codebase got sloppier. But because something fundamental changed about who — or more precisely, what — was using GitHub. AI coding agents arrived at scale, and they didn't behave anything like the human developers the platform was built for.
\nThe numbers tell the story with uncomfortable clarity. In 2024, GitHub logged 119 service incidents, including 26 major ones — frustrating, but manageable, with an average resolution time of roughly 106 minutes. Then, between May 2025 and April 2026, incident monitoring service IncidentHub tracked 257 separate incidents, of which 48 were classified as major outages. February 2026 alone produced 37 incidents — the worst month on record. GitHub Actions, the platform's CI/CD automation backbone, suffered 57 outages in the same 12-month stretch. On May 15, 2026, a single Actions degradation caused 42% of all Actions runs to fail at peak impact. Developers worldwide woke up to red CI pipelines and spent hours debugging their own code — before eventually realizing it wasn't their code at all.
\n\n\n\nTHE CORE PROBLEM: AGENTS DON'T BEHAVE LIKE HUMANS
\nA human developer on a free GitHub account might generate a few commits and a handful of CI runs in a working day. An AI agent on the same account can generate hundreds of commits, dozens of PRs, and thousands of Actions minutes in a single afternoon. GitHub's 2025 Octoverse report celebrated nearly 1 billion commits. By early 2026, GitHub COO Kyle Daigle shared a more alarming figure: the platform was handling 275 million commits every single week — on pace for 14 billion in 2026 if growth held linear. That's a 14x annual increase. It wasn't 14x more developers. It was agents treating GitHub's API like a utility and consuming at machine speed.
\n
GitHub's architecture was designed for a world where developers work at human speed: open a PR, push commits over hours or days, wait for CI to run, merge when green. The platform's capacity planning, its database schemas, its job queues, its rate limits — all calibrated for a workflow where one human generates a bounded amount of activity per session. That assumption held for 17 years.
\nGitHub CTO Vlad Fedorov identified the mechanism: a single pull request can simultaneously touch Git storage, mergeability checks, branch protection, GitHub Actions, search, notifications, permissions, webhooks, APIs, background jobs, caches, and databases. A human merging one PR triggers this chain once. An AI agent framework running hundreds of concurrent sessions triggers it thousands of times simultaneously. AI-agent PRs alone jumped from 4 million in September 2025 to 17 million in March 2026 — a 325% increase in six months. Actions usage went from 500 million minutes per week in 2023 to 2.1 billion minutes in a single week in early 2026.
\nGitHub began a 10x capacity scaling initiative in October 2025. By February 2026, that plan was already obsolete — the real demand required 30x. Simultaneously, GitHub was running a migration to Azure, with 12.5% of all traffic on Azure Central US and a target of 50% by July 2026. Running a platform migration alongside an AI-driven traffic explosion is the engineering equivalent of rebuilding an airplane's engines at 35,000 feet.
\nThe pressure produced not just performance degradation but engineering failures. On April 23, 2026, an incomplete feature flag silently reverted commits across 658 repositories and 2,092 pull requests — the UI showed green checkmarks while code was being rewritten underneath. On April 27, a botnet attack overwhelmed the Elasticsearch backend and took GitHub Search offline for hours. On April 28, Mitchell Hashimoto — GitHub user #1299, co-founder of HashiCorp, joined February 2008 — announced that Ghostty was leaving GitHub. The Zig programming language project also migrated away.
\n\n\n\n🤖
\nPeak monthly metrics by early 2026: 90 million merged PRs, 1.4 billion commits, 20 million new repositories. These are not the metrics of a platform being used by developers. These are the metrics of a platform being consumed by machines.
\n
Two incidents in late April 2026 crystallized the reliability crisis into something personal for every developer who experienced them. The first, on April 23, was a merge queue regression — a bug caused by an incomplete feature flag deployment — that silently reverted commits across 658 repositories and 2,092 pull requests. The terrifying part was not the scope. It was the silence. The UI continued to show green checkmarks and merge confirmations while the system was actively undoing work underneath. Developers had no idea their code had been reverted until they dug into the diff. A platform's most sacred contract with its users is that when it shows a green checkmark, the operation succeeded. GitHub broke that contract.
\nThe second incident, on April 27, was a Search outage triggered by what GitHub described as a likely botnet overwhelming the Elasticsearch backend. Search went down for hours across the platform. Taken individually, either incident could be explained away. Together, in the same week, they were the signal that the accumulated reliability debt had become impossible to ignore.
\n\n\n\n⚠️
\nThe Silent Revert: Why the Merge Bug Was So Damaging
The April 23 merge queue bug was technically a data integrity issue — code that had been merged was reverted without notification. But its deeper damage was psychological. Developers depend on version control's fundamental promise: what you commit is preserved, what you merge is recorded. A bug that silently breaks this promise doesn't just cause data loss. It causes a loss of confidence in every operation the platform reports as successful. How many other green checkmarks weren't telling the whole story? This is the question that made Mitchell Hashimoto's departure feel less like frustration and more like a considered assessment of risk.
\n
On April 28, 2026, Mitchell Hashimoto — GitHub user number 1299, co-founder of HashiCorp, creator of Vagrant, Packer, Consul, Terraform, and Vault, one of the most prolific infrastructure engineers in the industry — posted that Ghostty was leaving GitHub. He had visited GitHub almost every day for over 18 years. His post described the decision as 'irrationally sad' but said the platform was no longer a place where he could 'get work done' and 'ship software.' He made a point that resonated across the developer community: the problem was not Git itself — the distributed version control system remained excellent. The problem was the surrounding infrastructure: issues, pull requests, GitHub Actions. The platform built around Git was failing.
\n\n\n\nGITHUB USER #1299: WHY THE NUMBER MATTERS
\nMitchell Hashimoto is GitHub user #1299 — one of the earliest accounts on the platform, joined February 2008, three years after GitHub's 2005 founding. His departure is not symbolically significant because he is famous. It is symbolically significant because he is the kind of developer GitHub was built for: a serious, high-output infrastructure engineer who had chosen GitHub without question for 18 years. When the person who never had reason to question the platform starts questioning it, something has fundamentally changed.
\n
GitHub's CTO Vlad Fedorov publicly outlined the engineering response in his April 28 blog post. The plan has three interlocking components, each targeting a different layer of the scaling problem. Together they represent one of the most significant architectural transformations GitHub has undertaken since its founding.
\nGitHub's five-layer engineering response to the agentic AI scaling crisis, as outlined by CTO Vlad Fedorov in April 2026
| Problem Layer | Root Cause | GitHub's Fix |
|---|---|---|
| Language / Runtime | Ruby monolith has GIL (Global Interpreter Lock), limiting CPU parallelism — cannot efficiently use all available cores under high concurrency | Rewriting performance-critical services from Ruby to Go — Go's goroutine model handles massive concurrency without the GIL bottleneck |
| Infrastructure | Single-cloud (Microsoft Azure-only migration) creates concentrated failure risk and limits horizontal scaling options | Multi-cloud deployment strategy — 12.5% on Azure Central US in early 2026, targeting 50% by July 2026 with additional cloud providers |
| Service Isolation | A single PR cascades through 10+ interconnected subsystems — Git storage, Actions, search, notifications, permissions, webhooks — so any bottleneck in one propagates to all | Isolating critical services (Git and Actions especially) into independent failure domains so a degradation in one subsystem cannot cascade to others |
| Capacity Planning | 10x scaling plan (October 2025) became obsolete by February 2026 as AI agent traffic doubled the required headroom | 30x capacity design — automated scaling for agent-driven burst load patterns rather than human-paced steady-state assumptions |
| Feature Safety | April 23 merge queue regression was caused by an incomplete feature flag that allowed a new code path to activate without full safeguards | Strengthened feature flag discipline — no code path that affects data integrity ships without complete flag protection and staged rollout verification |
\n\n\nℹ️
\nWhy the Ruby-to-Go Rewrite Is the Right Call
Ruby served GitHub extraordinarily well for 18 years. It enabled rapid product development, contributed to GitHub's culture, and its Rails framework made the web application layer elegant and maintainable. But Ruby's Global Interpreter Lock (GIL) is a fundamental constraint: even on a 64-core server, a Ruby process can only execute one thread of Ruby code at a time. For human-paced web traffic, this limitation is manageable. For AI agent workflows that generate thousands of concurrent operations, the GIL is a hard ceiling. Go's goroutine model — lightweight threads managed by the Go runtime that can run across all available CPU cores without a GIL — is architecturally suited for exactly the concurrency profile that AI agents create. The rewrite is not about language preference. It is about physics.
\n
A deeper structural problem underlies the engineering crisis: GitHub's business model was designed for humans, and its pricing reflects human-scale consumption. A developer on a free GitHub account generates some commits, a few CI runs, and a handful of API calls per day. An AI agent on the same account can generate hundreds of commits, dozens of PRs, thousands of Actions minutes, and tens of thousands of API calls in a single afternoon. The infrastructure cost per 'user' has fundamentally changed, but the pricing model has not yet caught up. As one engineer put it plainly: GitHub's Octoverse 2025 report celebrated nearly 1 billion commits and 36 million new developers. But the 2026 numbers aren't being driven by 36 million new developers showing up. They're being driven by agents that treat GitHub's API like a utility — which it basically is, except utilities charge for consumption.
\n\n\n\n83 INCIDENTS FROM CAPACITY FAILURES ALONE
\n83 of GitHub's 257 incidents between May 2025 and April 2026 were caused by load and capacity problems — with indications that many services did not have automatic scaling configured, requiring manual intervention to add capacity during surges. This means that dozens of times, engineers had to notice the problem, escalate it, and manually provision resources before the platform could recover. Automated capacity scaling for burst load is not optional infrastructure. For a platform being consumed by AI agents, it is the minimum viable reliability architecture.
\n
\n\n\n⚠️
\nThe CVE-2026-3854 Problem: Reliability and Security Compounding
The April week that prompted Hashimoto's departure also included a critical security disclosure: CVE-2026-3854, a CVSS 8.7 remote code execution vulnerability in GitHub's internal Git layer. The flaw allowed an attacker to inject extra header fields via a malformed git push and execute code as the Git service user. GitHub.com was patched within six hours of disclosure. GitHub Enterprise Server patches were released. But Wiz reported that 88% of self-hosted GitHub Enterprise Server instances remained unpatched at time of publication. A platform under reliability stress is also a platform whose administrators are too busy managing incidents to maintain their security posture — the two crises compound each other.
\n
GitHub's architecture evolved over 18 years around a core assumption: the unit of load is a human developer. A human opens a PR, waits for review, pushes a few commits, and merges. The platform's service graph — its Git storage layer, its mergeability computation engine, its branch protection evaluation system, its Actions job dispatch queue, its search indexer, its notification fan-out, its webhook delivery pipeline, its permission evaluation layer, its API gateway — was sized and coupled around this human-paced access pattern. Every service in the chain was both a dependency and a dependency of every other service. This architecture was efficient and made GitHub easy to reason about for years.
AI agents broke the architecture's fundamental assumption. An agent doesn't open a PR and wait. An agent opens 50 PRs in parallel, each triggering the full service chain simultaneously. At scale, this creates a concurrency storm that amplifies through every layer of the graph. GitHub CTO Vlad Fedorov described it precisely: a single PR touches Git storage, mergeability checks, branch protection, Actions, search, notifications, permissions, webhooks, APIs, background jobs, caches, and databases. When the number of concurrent PRs scales 4x in six months, the pressure on every one of those systems scales accordingly — and the interconnected failures begin.
\r\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\r\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\r\n\n\n\r\nTHE RUBY GIL: WHY THE MONOLITH COULDN'T SCALE
\nRuby's Global Interpreter Lock (GIL) is a mutex that prevents multiple threads from executing Ruby code simultaneously in the same process. For human-paced web traffic — where a request comes in, does some database work, and returns a response — the GIL is rarely the bottleneck. For AI agent traffic — where thousands of operations arrive concurrently and each one fans out across dozens of internal services — the GIL becomes a hard ceiling. Even on a 64-core server, a Ruby process can use exactly one core at a time for Ruby execution. The fix isn't optimization. It's a different runtime. Go's goroutine scheduler runs across all available CPU cores without a GIL, making it architecturally suited for the concurrency profile that AI agent workflows generate. GitHub's Ruby-to-Go migration for performance-critical services is the right move — not as a language preference, but as a physics constraint.
\n
\n\n\nℹ️
\nThe Azure Migration Timing Problem
GitHub began migrating traffic to Azure as part of its Microsoft integration — 12.5% of all traffic on Azure Central US in early 2026, with a target of 50% by July 2026. Running this migration simultaneously with a 30x capacity redesign and a Ruby-to-Go rewrite is an extraordinary amount of concurrent infrastructure transformation. Each of these projects is a multi-year undertaking at GitHub's scale. Running all three in parallel reduces the blast radius of each individual change — but increases the cognitive load and coordination complexity for the engineering teams managing them. The timing was not chosen; it was forced by the speed of the AI agent traffic explosion.
\n
GitHub's reliability crisis is not a story about a company making engineering mistakes. It's a story about a platform being asked to do something it was never designed for — at a speed that outpaced any reasonable capacity planning horizon. The lessons are as much about the nature of AI agent infrastructure demands as they are about reliability engineering practice.
\r\n\n\n\r\n✅
\nThe Availability-First Mandate
GitHub's leadership response to the crisis was to shift from a growth-at-all-costs philosophy to an availability-first mandate. This means engineering prioritization changes: new feature work is deprioritized relative to stability improvements, scaling infrastructure, and incident remediation. The availability-first mandate is the organizational signal that GitHub recognizes the seriousness of the reliability debt it has accumulated. Whether the engineering plans — Ruby-to-Go, service isolation, 30x capacity, multi-cloud — can be executed faster than the AI agent traffic continues to grow is the open question that will define GitHub's next two years.
\n
\n\n\r\nWHAT EVERY DEVELOPER SHOULD DO RIGHT NOW
\nGitHub's instability has a practical implication for every engineering team: treat GitHub as important infrastructure, not invisible infrastructure. Map your team's GitHub dependency surface — CI/CD pipelines, registry mirrors, source-of-truth, identity flows, Actions runners. Know which of your deployments would be blocked if GitHub Actions was degraded for four hours. Have a runbook for 'GitHub is down' that doesn't end with 'wait for GitHub to come back.' Independent CI mirroring, artifact registries with fallback paths, and local Git mirrors of critical dependencies are not paranoia — they are the appropriate response to a platform that has demonstrated it will have five major incidents per week for a year.
\n
\n\nGitHub spent 18 years building the platform where the world's code lives, survived Microsoft's acquisition, launched Copilot, and made developers 10x more productive — and then the thing that broke it was all those developers becoming 100x more productive using Copilot. The platform that accelerated AI-assisted development got outrun by AI-assisted development. There is probably a lesson in there somewhere about building infrastructure for the world you are creating, not the world you came from.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-31T00:00:00+00:00", "date_modified": "2026-06-13T19:03:06.257453+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Distributed Systems", "GitHub"]}, {"id": "https://techlogstack.com/explore/aws-dynamodb-dns-outage-2025/", "url": "https://techlogstack.com/explore/aws-dynamodb-dns-outage-2025/", "title": "A Race Condition in DynamoDB's DNS Took Down Snapchat, Fortnite, Ring, and Half the Internet for 15 Hours", "summary": "On October 19–20, 2025, a race condition between two DNS Enactor processes wiped all DynamoDB DNS records in US-EAST-1, cascading into a 15-hour outage that took dow", "content_html": "AWS · Distributed Systems · 31 May 2026
\nIt was 11:48 PM PDT on October 19, 2025. Two automation processes inside AWS's DynamoDB DNS management system were doing the same job simultaneously — one fast, one painfully slow. The slow one was just finishing up when the fast one, having already completed, triggered a cleanup job that deleted the slow one's work. In that moment, every DNS record for DynamoDB in the world's busiest cloud region vanished. Snapchat went dark for 375 million daily users. Fortnite lobbies dissolved mid-match. Ring cameras stopped recording. The UK's HMRC tax authority went offline. For 15 hours, the internet's largest database service had no address.
\n\nWhen this issue occurred at 11:48 PM PDT, all systems needing to connect to the DynamoDB service in the N. Virginia (us-east-1) Region via the public endpoint immediately began experiencing DNS failures and failed to connect to DynamoDB. This included customer traffic as well as traffic from internal AWS services that rely on DynamoDB.
— — Amazon Web Services — Official Post-Incident Summary, October 2025
DynamoDB is not just a database. Inside AWS's infrastructure, it is the connective tissue — the system that EC2, IAM, Lambda, STS, Redshift, and dozens of other control-plane services rely on to store metadata, track state, and coordinate operations. When DynamoDB becomes unreachable, it doesn't just take databases offline. It takes down the systems that manage everything else. This is why a DNS failure that lasted roughly three hours for DynamoDB itself cascaded into a 15-hour platform-wide crisis. The control plane broke. And when the control plane breaks, recovery is not a matter of fixing the root cause — it is a matter of stabilizing everything that lost its footing when the ground disappeared.
\nTo understand what happened, you need to understand how DynamoDB manages its DNS. At AWS's scale, DynamoDB maintains hundreds of thousands of DNS records to operate the massive fleet of load balancers that route traffic across each region. These records are updated constantly — as capacity is added, as hardware fails, as traffic is redistributed. AWS built a two-component system to manage this at scale: a DNS Planner and multiple DNS Enactors.
\n\n\n\nTHE TWO-COMPONENT DNS ARCHITECTURE: PLANNER AND ENACTOR
\nThe DNS management system had two independent components: The DNS Planner monitors load balancer health and capacity and periodically creates new DNS plans — essentially a specification of which load balancers should receive traffic and with what weight distribution. The DNS Enactors are the workers — multiple independent processes running across three different Availability Zones — that pick up the plans and apply them to Route53 (AWS's DNS service). Multiple Enactors running in parallel provide redundancy: if one Enactor fails, others continue. In theory.
\n
DNS Enactor A began applying an older DNS plan but encountered unusual delays — it kept getting blocked trying to update records and moved painfully slowly through the list of endpoints. Crucially, Enactor A performed a staleness check early in its process: 'Is my plan newer than what's currently active?' At the time of that check, it was. But by the time Enactor A actually finished applying the plan, the world had moved on — newer plans had been created and applied. The staleness check was now stale itself.
\nWhile Enactor A was slowly working through its updates, Enactor B picked up one of the newer plans and rapidly applied it across all endpoints. When Enactor B completed, it triggered the cleanup process: identify plans that are significantly older than the one just applied, and delete them. At that exact moment — T+45 seconds after the race began — Enactor A finally finished applying its old plan, overwriting Enactor B's newer records. The cleanup job identified Enactor A's newly-applied old plan as many generations old, and deleted it. All DynamoDB DNS records for the US-EAST-1 regional endpoint were gone.
\nAt 11:48 PM PDT on October 19, every system trying to connect to DynamoDB in US-EAST-1 via the public endpoint received DNS failures. No IP address. No connection. The failure was immediate and total — not a degradation, not elevated error rates, but a complete inability to resolve the DynamoDB endpoint. Internal AWS services relying on DynamoDB for control-plane operations went down alongside customer traffic. EC2's Droplet Workflow Manager lost its ability to track instance lease state. IAM couldn't validate credentials. Lambda couldn't execute. The cascade was underway.
\nEngineers identified the DNS issue by 12:38 AM UTC and began temporary mitigations by 1:15 AM UTC. DynamoDB itself recovered by approximately 2:25 AM UTC — roughly three hours after the incident began. But the cascade had already overwhelmed EC2's Droplet Workflow Manager with a backlog of expired instance leases it couldn't process. The DWFM entered congestive collapse, requiring 12+ more hours for network state to fully stabilize. The fix for the automation itself was brutal in its simplicity: engineers had to manually disable the automatic failover system entirely to stop it from flip-flopping between states and allow the platform to stabilize. Full recovery across all services wasn't complete until late afternoon on October 20 — roughly 15 hours after the cascade began.
\n\n\n\n🌐
\nOokla, the network intelligence company behind Speedtest.net, recorded over 17 million outage reports across more than 3,000 organizations during the incident. Independent measurements showed 20 to 30 percent of all internet-facing services experienced disruptions at peak impact. The US, UK, and Germany were hit hardest.
\n
The list of affected services illustrates something important about how the modern internet is structured. US-EAST-1 is AWS's default region — the one developers reach for first, the one that has the most mature service availability, the one that decades of 'just deploy to us-east-1' decisions have concentrated critical infrastructure in. Even services claiming multi-region redundancy often still rely on US-EAST-1 for authentication, metadata stores, or database calls — dependencies that only become visible when US-EAST-1 goes dark.
\nMajor services and platforms affected by the October 19–20, 2025 AWS US-EAST-1 outage
| Category | Affected Services |
|---|---|
| Social & Entertainment | Snapchat (375M daily users), Discord, Reddit, Roblox, Fortnite, Disney+, Hulu, Twitch |
| Finance & Payments | Coinbase, Venmo, several UK banks (Lloyds, Halifax) |
| Smart Home & IoT | Amazon Ring cameras, Amazon Alexa, Eight Sleep |
| Communications | Signal, several enterprise communication platforms |
| Government | UK HMRC tax authority |
| Travel | United Airlines app, Delta app |
| AI & Developer Tools | Perplexity AI, Pokémon GO |
| AWS Services (internal) | EC2, IAM, STS, Lambda, S3, SQS, Amazon Connect, Redshift (140+ services total) |
\n\n\n⚠️
\nThe Control Plane Problem: Why DynamoDB's Failure Was Uniquely Catastrophic
A typical service outage takes down the service that failed. The October 2025 DynamoDB outage was different because DynamoDB is infrastructure for infrastructure. EC2 uses DynamoDB to track compute instance state. IAM uses DynamoDB to store and retrieve access policies. Lambda uses DynamoDB for execution state. STS uses DynamoDB to validate tokens. When DynamoDB became unreachable, these services couldn't perform their core functions — not because they had their own bugs, but because the foundation they relied on had disappeared. This is a control-plane failure, and control-plane failures cascade differently from data-plane failures: they don't just take down what failed, they take down the ability to manage everything else.
\n
DynamoDB's DNS was restored in approximately three hours. But the outage lasted 15. The reason was EC2's Droplet Workflow Manager (DWFM) — the system responsible for managing EC2 instance lifecycle events, including lease renewals. When DynamoDB became unavailable, DWFM couldn't process instance state updates and began accumulating a backlog of expired leases. By the time DynamoDB recovered, DWFM was facing an enormous queue of backlogged lease management tasks — all trying to execute simultaneously. The system entered congestive collapse: the more it tried to process, the more it overwhelmed the now-recovered DynamoDB, which slowed processing, which lengthened the queue, which increased the pressure. Network state recovery from this congestive collapse took more than five additional hours after DynamoDB was fixed.
\n\n\n\nTHE ANTI-PATTERN: WHEN AUTOMATION PREVENTS RECOVERY
\nThe most counterintuitive part of the recovery was that engineers had to disable automatic failover to stabilize the system. The automatic failover mechanisms — designed to move traffic to healthy systems when failures are detected — were themselves contributing to the instability. With DNS records in an inconsistent state, the failover systems were flip-flopping: detecting failures, triggering failovers, detecting those failovers as failures, triggering more failovers. The automation designed to speed recovery was making recovery impossible. Engineers had to manually turn it off, let the system reach a stable state, and then re-enable it with the correct DNS records in place. This is one of the most instructive moments in the incident: sometimes, the recovery automation has to be stopped before recovery can begin.
\n
AWS published its official post-incident report three days after the October 20 event. The fixes address four distinct failure layers: the race condition itself, the cleanup automation that deleted active records, the EC2 cascade, and the inadequate test coverage for the recovery workflow. Each fix targets a specific mechanism that allowed the failure to happen or to propagate.
\nAWS's five-layer post-incident fix plan, derived from the official post-incident summary published October 23, 2025
| Failure Layer | What Went Wrong | AWS's Fix |
|---|---|---|
| DNS Enactor race condition | Two Enactors ran concurrently; Enactor A's stale staleness check allowed it to overwrite Enactor B's newer plan | Stronger staleness validation in the Enactor before applying plans — the check must reflect the current state of the world at time of application, not at time of plan pickup |
| Cleanup automation | The cleanup job deleted Enactor A's just-applied old plan because it appeared many generations old — wiping all DNS records in the process | Safeguards to ensure no automated process can delete or remove an active DNS plan — any plan being actively used as the live record is protected from cleanup regardless of its generation number |
| NLB failover velocity | Network Load Balancers moved large amounts of capacity during AZ failover triggered by the DNS failure, amplifying the cascade | Velocity control mechanism for NLBs to limit how much capacity a single NLB can remove when health check failures cause AZ failover — preventing AZ-level failures from creating region-level capacity evaporation |
| EC2 recovery workflow | EC2's DWFM entered congestive collapse when DynamoDB recovered and the backlogged lease queue overwhelmed the system — a failure mode that had not been tested | Additional test suite to exercise the DWFM recovery workflow at scale — catching congestive collapse scenarios before they happen in production rather than discovering them during outage recovery |
| Automatic failover during recovery | Failover automation flip-flopped during recovery, requiring manual disabling before stabilization could occur | Review of failover automation behavior during degraded DNS states — automation must detect the difference between 'service is down' and 'DNS is inconsistent during recovery' and respond differently to each |
\n\n\n⚠️
\nThe Unstated Root Cause: The Architecture of Trust in US-EAST-1
AWS's post-mortem addressed the technical race condition correctly. But the incident exposed a deeper architectural problem that no single fix resolves: the internet's implicit trust in US-EAST-1. AWS designed US-EAST-1 as a region — a geographic cluster of data centers meant to be one of many independently redundant deployment targets. Over 20 years, it became something else: the default region for millions of applications, the region where foundational services were first available, and the region that even 'multi-region' architectures often quietly depend on for authentication, metadata, or coordination. Ring cameras depend on it for authentication. Venmo depends on it for payment processing. UK government services depend on it for API calls. None of these dependencies were meant to create a single point of failure. But that's what they became.
\n
One of the most honest admissions in AWS's post-incident report is about test coverage. The DWFM recovery workflow — the path EC2's Droplet Workflow Manager follows when it needs to process a massive backlog of expired leases after a DynamoDB outage — had not been adequately tested at the scale required to discover the congestive collapse failure mode. AWS's response is to build additional test suites specifically for this recovery workflow. But the admission surfaces a fundamental challenge of hyperscale infrastructure: the failure conditions that matter most are the ones that only occur at scale, and at scale, test environments are approximations of production, not replicas of it. The only complete test of how AWS's systems behave during a DynamoDB outage recovery is an actual DynamoDB outage. This is the same insight that drove Netflix to build Chaos Monkey — except that for a cloud provider, you cannot deliberately cause a DynamoDB outage to test the recovery path.
\n\n\n\nTHE HIDDEN CROSS-REGION DEPENDENCY PROBLEM
\nThe October 2025 outage adds to a body of evidence about a specific architectural anti-pattern: regions that are called independent but aren't. AWS regions were designed with the premise that a failure in US-EAST-1 should not affect services running in EU-WEST-1 or AP-SOUTHEAST-1. But control-plane dependencies — authentication services, metadata stores, quota management systems — create invisible cross-region ties. When the control plane fails in one region, services in other regions that depend on that control plane for any operation fail with it. True regional independence requires not just deploying application code in multiple regions, but ensuring that every control-plane dependency is also independently redundant per region. For most organizations building on cloud infrastructure, this is not the architecture they have — it is the architecture they think they have.
\n
The October 2025 DynamoDB outage is a case study in what distributed systems engineers call a control-plane failure — a class of failure that is categorically more damaging than a data-plane failure because it removes the ability to manage and coordinate infrastructure rather than just disrupting one service. To understand why the failure cascaded so far and recovered so slowly, you need to understand the three layers of the failure: the DNS automation race condition, the DynamoDB control-plane dependency web, and EC2's Droplet Workflow Manager congestive collapse.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nWHY US-EAST-1 BECAME A SINGLE POINT OF FAILURE FOR THE INTERNET
\nAWS designed its regions to be independently operable — a failure in US-EAST-1 should not affect EU-WEST-1. This design intention is correct, but the reality that emerged over 20 years is different. US-EAST-1 is where AWS first launched most services, so it accumulated the most mature feature sets. It became the default — the region developers reach for first, the region enterprises trust most deeply. Over time, even architectures claiming multi-region resilience often retain quiet dependencies on US-EAST-1 for authentication flows, control-plane coordination, or foundational database calls. The technical independence of regions is real. The operational independence, as experienced during the October 2025 outage, is not.
\n
\n\n\n⚠️
\nThe Automatic Failover Anti-Pattern
One of the most practically instructive moments of the October 2025 outage was the decision to manually disable automatic failover to allow recovery to proceed. The automatic failover systems — designed to improve availability — were detecting the DNS inconsistency as failures and triggering failovers, which created new inconsistencies, which triggered more failovers. The automation was creating a feedback loop that prevented stabilization. Engineers had to turn it off to let the system reach a stable state. The lesson: automatic recovery systems need to distinguish between 'the service is down' (trigger failover) and 'the DNS state is inconsistent during manual recovery' (pause failover until DNS is stable). Automation that cannot make this distinction can prevent recovery faster than it enables it.
\n
The October 2025 DynamoDB outage is one of the most technically instructive incidents in cloud computing history — not because the root cause was complex, but because it was so simple, and yet it cascaded so far. A race condition in a cleanup job. The most consequential bug is often the one you're sure you've already solved.
\n\n\n\n✅
\nWhat Good Regional Independence Actually Looks Like
The October 2025 outage drew a clear line between companies that had genuine multi-region independence and those that believed they did. Genuine independence requires: application code deployed in at least two regions; authentication, authorization, and metadata services independently operational per region; no synchronous cross-region API calls in the critical path; tested failover that has been exercised under real load; and runbooks that don't assume a specific region is available. The companies whose services stayed up during the October 2025 outage weren't lucky. They had made specific architectural decisions years earlier — decisions that cost money and engineering time — that happened to be exactly the right decisions.
\n
\n\n\nTHE PRACTICAL RESPONSE FOR EVERY ENGINEERING TEAM
\nThe October 2025 AWS outage has a direct implication for every team running production workloads on cloud infrastructure. Map your US-EAST-1 dependencies before the next outage, not during it. Specifically: identify every service your application calls that is hosted in US-EAST-1, even if your application code is deployed elsewhere. This includes authentication providers, CDN origins, third-party APIs, and internal microservices. For each dependency, ask: 'If this endpoint returned no DNS records for three hours, what would our users experience?' The answer to that question is your actual blast radius for a US-EAST-1 control-plane failure — and it is almost certainly larger than your architecture diagram suggests.
\n
\n\nAmazon Web Services runs infrastructure at a scale where the cost of a single race condition is measured in hundreds of millions of dollars and 375 million users unable to send a Snapchat. The race condition itself — two processes trying to update the same state concurrently, with a stale check allowing a stale write — is the kind of bug that appears in computer science textbooks under 'concurrent programming gotchas.' The lesson isn't that AWS made an obvious mistake. The lesson is that obvious mistakes at sufficient scale have non-obvious consequences, and the gap between 'finding the bug' and 'recovering from the bug' was twelve hours wide.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-31T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.292804+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Distributed Systems", "AWS"]}, {"id": "https://techlogstack.com/explore/google-cloud-service-control-outage-2025/", "url": "https://techlogstack.com/explore/google-cloud-service-control-outage-2025/", "title": "Google's Own Cleanup Job Crashed Cloud Services Across 4 Continents — and Then Made Recovery Worse", "summary": "On June 12, 2025, a null pointer exception in Google Cloud's Service Control binary — deployed May 29 without error handling or feature flag protection — crashed API", "content_html": "Google · Distributed Systems · 31 May 2026
\nOn May 29, 2025, a Google engineer deployed new quota-checking code to Service Control — the system that authorizes every single API request across Google Cloud. The code had a bug: it couldn't handle a null value. But the bug was invisible during deployment because it could only be triggered by a specific type of policy data that hadn't appeared yet. Two weeks later, on June 12, an automated system pushed a routine policy update containing blank fields. The policy data replicated globally within seconds. Every Service Control binary in every region hit the null pointer, crashed, and refused to restart without eating itself. Spotify went down. Discord went down. Snapchat went down. Google's own status page went down. And when engineers deployed the fix, the restart surge overwhelmed the infrastructure — making the recovery worse than the crash.
\n\nOn May 29, 2025, a new feature was added to Service Control for additional quota policy checks. The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash.
— — Google Cloud — Official Incident Report, June 14, 2025
Service Control is not a product you've heard of. It doesn't have a marketing page or a conference talk. It exists in the infrastructure layer beneath everything else — the system that authorizes every API request across Google Cloud and Google Workspace before that request is allowed to proceed. If you call the Cloud Storage API, Service Control checks your quota. If you authenticate with Google IAM, Service Control validates your policy. If your app on Google Cloud makes any call to any Google service, Service Control is in the critical path. It is, in the most literal sense, the gatekeeper of the entire platform.
\nWhen Service Control crashed on June 12, 2025, it didn't just take down one service. It took down the authorization layer for every service. API calls returned 503 errors not because the underlying services had failed, but because the gatekeeper wasn't there to let them through. Compute Engine instances were running. Cloud Storage buckets were intact. BigQuery jobs were ready to execute. None of it mattered — because without Service Control, nothing could be authorized, and nothing unauthorized can proceed in a correctly secured cloud platform.
\n\n\n\nWHAT SERVICE CONTROL ACTUALLY DOES
\nGoogle Cloud's Service Control system performs three functions on every API request: authentication (is this requester who they claim to be?), authorization (are they allowed to perform this operation?), and quota enforcement (have they exceeded their usage limits?). It processes these checks at massive scale across every region — billions of API calls per day — using policy metadata stored in and synchronized across Spanner, Google's globally distributed database. The May 29 code change was adding more sophisticated quota checking logic to this pipeline. The change worked correctly in every scenario that was tested. The scenario that wasn't tested was the one that appeared on June 12.
\n
Google engineers deployed new quota policy checking code to Service Control. The deployment went through the standard region-by-region rollout and passed all checks. But the new code path had two critical gaps: no error handling for null values, and no feature flag to disable it if something went wrong. The bug was invisible during rollout because the problematic code path could only be triggered by a specific type of policy input — blank fields in the policy metadata. That input hadn't appeared during rollout. The binary was now running in every region with a loaded trap, waiting for the right trigger.
\nAn automated system inserted a routine policy change into the regional Spanner tables that Service Control uses for policy metadata. The policy update contained unintended blank fields — values that should have been populated but weren't. Because quota management is global, the Spanner replication engine distributed this metadata worldwide within seconds. Every Service Control binary in every region hit the new code path, encountered the null values, and threw a null pointer exception. Without error handling, the exception crashed the binary. Service Control was dead globally.
\nGoogle's Site Reliability Engineering team began triaging within two minutes of the first alert. They identified the root cause — the null pointer exception in the new quota checking code path — within 10 minutes. Engineers deployed a 'red button' kill switch within 40 minutes to disable the problematic serving path. Most regions began recovering within two hours. But us-central1, Google's largest region, hit a second problem: the recovery itself.
\nAs Service Control instances restarted in us-central1 after the red button was deployed, they all simultaneously reached for the regional Spanner database to load their policy metadata. Hundreds of instances, all restarting at the same moment, all hitting Spanner at the same time, with no randomization in their startup sequence. The Spanner database — which had been handling steady-state read traffic fine — was overwhelmed by the simultaneous burst. Service Control couldn't load its policies, which meant it couldn't restart properly, which meant it kept trying, which kept hitting Spanner. The recovery created a herd effect that prolonged the outage in us-central1 by more than two hours beyond when other regions had stabilized. Full resolution across all services wasn't complete until 18:18 PDT — more than seven hours after the incident began.
\n\n\n\n🔴
\nGoogle's own Cloud Service Health dashboard went down during the incident — the monitoring system that customers rely on to track outage status was itself affected by the outage it was supposed to report. Engineers and customers trying to understand what was happening couldn't access the standard communication channel. A status page that goes down during the incident it's supposed to report is a monitoring anti-pattern at its most consequential.
\n
The blast radius of the June 12 outage had three concentric rings. The innermost ring was Google's own services: Google Cloud Platform APIs, Google Workspace (Gmail, Calendar, Drive, Docs, Meet), IAM, Cloud Storage, Compute Engine, BigQuery, Cloud SQL, Cloud Spanner, Vertex AI, Cloud Monitoring. The second ring was companies building directly on GCP — Spotify, Snapchat, Fitbit, Replit, GitLab, Shopify, Character.AI, Cursor — whose applications were unable to authorize any backend calls. The third ring was the one that made this incident uniquely instructive: companies that depend on Cloudflare, which depends on Google Cloud. Cloudflare — itself one of the internet's core infrastructure providers — uses Google Cloud for some of its backend operations. When Google Cloud's Service Control failed, Cloudflare experienced partial degradation, which in turn degraded Discord, Twitch, and other services that had built on top of Cloudflare. This is third-order cascading failure: Google fails → Cloudflare degrades → Discord goes down. Discord's users had no idea their outage had anything to do with a null pointer exception in a Google quota management system.
\nThe three-ring cascade of the June 12, 2025 Google Cloud outage — and the dependency chain that connected them
| Failure Ring | What Failed | Why |
|---|---|---|
| First: Google's own infrastructure | Cloud IAM, Compute Engine, Cloud Storage, BigQuery, Cloud SQL, Cloud Spanner, Vertex AI, Cloud Monitoring, Google Workspace (Gmail, Calendar, Drive, Docs, Meet) | Service Control — the authorization gateway — crashed globally, blocking all API requests across GCP and Workspace |
| Second: Direct GCP customers | Spotify (~46K outage reports), Snapchat, Fitbit, Replit, GitLab, Shopify, Character.AI, Cursor, Perplexity AI | Applications built on GCP couldn't authorize any backend calls — services appeared down to users even though underlying compute was running |
| Third: Cloudflare and its customers | Cloudflare (partial degradation), Discord, Twitch | Cloudflare uses Google Cloud for certain backend operations; when those degraded, Cloudflare's services partially degraded, cascading to Cloudflare's own customers |
\n\n\n⚠️
\nThe Dormant Trap: Why Staged Rollouts Didn't Catch This
Google's staged, region-by-region rollout is exactly the right practice for catching bugs introduced by new deployments. It worked correctly for 14 days — no failures appeared during the May 29 rollout because the failure condition required specific policy data (blank fields) that hadn't yet been inserted. The bug was a dormant trap: present in production, but invisible until the exact trigger arrived. This is a class of bug that staged rollouts are structurally unable to catch — because the rollout environment and the trigger environment are separated by two weeks and an automated policy change that nobody controlled. The only defenses against dormant traps are error handling (so the crash doesn't happen when the trigger arrives) and feature flags (so the code path can be disabled immediately when the trigger produces unexpected behavior). The May 29 change had neither.
\n
Google's incident report, published June 14, 2025, outlined specific remediation steps across five categories. Each addresses a distinct failure mode that either caused the outage or made it worse than it needed to be.
\nGoogle's five-category post-incident remediation plan, derived from the official June 14, 2025 incident report
| Failure Mode | What Happened | Google's Fix |
|---|---|---|
| Missing error handling | The new quota checking code had no null-safety — when blank fields appeared, a null pointer exception crashed the binary | Mandatory null-safe code patterns for all Service Control code paths, with additional static analysis to catch null pointer vulnerabilities before deployment |
| No feature flag | Without a feature flag, the new code path could not be disabled without a full binary redeployment — adding 30+ minutes to initial response time | Feature flag protection required for all new code paths in Service Control — a flag would have allowed the problematic path to be disabled within seconds, before most regions crashed |
| Herd effect during recovery | Hundreds of Service Control instances restarting simultaneously all hit Spanner at the same time, overwhelming it and prolonging the us-central1 outage by 2+ hours | Randomized exponential backoff on Service Control startup — instances restart with jittered delays so Spanner load is distributed over time rather than concentrated in a burst |
| Status page availability | Google's Cloud Service Health dashboard went down during the outage, removing the primary customer communication channel | Decoupled status infrastructure — the status page must be architecturally independent of the services it monitors, with its own Service Control dependency removed |
| Service Control architecture | Service Control is a monolithic regional binary — a crash in Service Control takes down all API authorization for the entire region simultaneously | Modularize Service Control's architecture — isolate the quota checking component so a crash in quota logic does not crash the authentication and authorization components |
\n\n\n⚠️
\nThe Feature Flag That Would Have Saved Seven Hours
The most consequential missing safeguard in the June 12 outage was the absence of a feature flag on the new quota checking code path. A feature flag — a configuration switch that enables or disables a code path without a redeployment — would have changed the incident timeline dramatically. When the null pointer exceptions began firing at 10:49 AM PDT, engineers with a feature flag could have disabled the new code path across all regions within seconds or minutes, before the crash had spread globally. Without a feature flag, the only option was a red-button kill switch that required a new binary deployment — a process that took 40 minutes and still left the herd effect problem during restart. 40 minutes of global Service Control outage versus seconds of a feature flag toggle. Google's own incident report acknowledges this directly: 'If this had been flag protected, the issue would have been caught in staging.' The cost of not having a feature flag was measured in hundreds of millions of users unable to access their services for seven hours.
\n
The herd effect that prolonged the us-central1 outage is not a new problem. It has been documented since the earliest days of distributed systems: when many clients restart simultaneously after a shared dependency recovers, they all connect simultaneously and overwhelm the dependency, preventing it from returning to steady state. The canonical solution — randomized exponential backoff — is equally well-documented and simple: when restarting, add a random delay so clients stagger their reconnection attempts over a time window rather than clustering them at a single instant. Every Service Control instance waiting exactly zero milliseconds before hitting Spanner on restart is the problem. Service Control instances waiting a random delay between 0 and 30 seconds before hitting Spanner on restart is the solution. Google committed to implementing this. The fact that it required an outage to prompt the implementation is a reminder that known fixes for known problems often go unimplemented until the cost of not implementing them is paid in production.
\n\n\n\nMTTD VS DURATION: WHAT THE NUMBERS ACTUALLY TELL YOU
\nGoogle's SRE team began triaging within two minutes of the first alert. This is elite incident response. The MTTD (Mean Time to Detect) was near-instantaneous, and the root cause diagnosis took 10 minutes. These are remarkable numbers for a global infrastructure failure of this complexity. The lesson is not that Google's response was slow — it was fast. The lesson is that even elite incident response cannot compensate for missing preventative safeguards. Feature flags, error handling, and randomized backoff would have prevented or dramatically shortened the outage before any human had time to respond. The SRE team's quality is demonstrated by the 10-minute diagnosis. The systems quality gap is demonstrated by the 7-hour duration.
\n
The June 12 outage shares an important structural pattern with the October 2025 AWS DynamoDB incident: the trigger was not a complex new feature or an ambitious architectural change. It was a routine operation — in AWS's case, a cleanup job that deleted stale DNS plans; in Google's case, an automated policy update that inserted routine quota metadata. Routine operations are the hardest to protect against because they're not treated with the same scrutiny as new features. A new feature gets code review, testing, staged rollout, and monitoring. A routine automated policy update is assumed to be safe because it's been running correctly for months. But when the underlying system has changed — when new code is now in the critical path that couldn't handle what the routine operation produces — the routine operation becomes the trigger for a catastrophic failure. The implication is uncomfortable: every automated operation that modifies system-critical metadata must be treated as a potential trigger for any latent bugs in the code that consumes that metadata.
\n\nService Control sits at the intersection of every API request Google Cloud processes. Understanding how it failed — and why the failure spread so quickly and recovered so slowly — requires understanding three things: the role of Spanner as the global policy data store, the absence of safe failure handling in the new code path, and the herd effect as a predictable consequence of synchronized restart under load.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE GLOBAL SPANNER REPLICATION TRAP
\nThe reason the June 12 failure was global rather than regional was Spanner's design strength working against Google in this case. Spanner is Google's globally distributed database, engineered to replicate data to all regions in real time — typically within seconds. This replication is what makes Spanner so powerful for global consistency. On June 12, it was what made the failure instantaneous and universal. When the automated system inserted the policy update with blank fields into the regional Spanner tables, Spanner replicated that policy data to every region within seconds. Every Service Control instance in every region hit the null pointer at essentially the same moment. There was no regional staging, no propagation delay, no opportunity for an alert to fire in one region before the failure had spread to all others. The same architecture that gives Spanner its global consistency guarantee gave this bug its global blast radius.
\n
\n\n\n⚠️
\nThe Status Page That Went Dark
Google's Cloud Service Health dashboard — the system that customers rely on to understand what Google services are experiencing — went offline during the June 12 outage. This happened because the status infrastructure shared a dependency on the same Google Cloud services that were failing. A status page that fails during a widespread outage is not just unhelpful — it is actively harmful. Customers experiencing failures couldn't access the standard channel to confirm they weren't the source of the problem, couldn't track recovery progress, and couldn't communicate accurate information to their own stakeholders. The status page being down created a second outage: an outage of information. Google's commitment to decoupling status infrastructure from the services it monitors is not a nice-to-have. It is the baseline requirement for maintaining communication with customers during incidents.
\n
The June 12, 2025 Google Cloud outage carries lessons that apply to every engineering team — from startups deploying to a single cloud region to hyperscalers managing global infrastructure. The failure modes are not exotic. They are the canonical patterns of distributed systems engineering: missing error handling, absent feature flags, synchronized recovery storms. The scale is Google's. The lessons belong to everyone.
\n\n\n\n✅
\nThe Modularization Commitment
Google's most architecturally significant post-incident commitment was to modularize Service Control — isolating the quota checking component so that a failure in quota logic cannot crash the authentication and authorization components. Currently, Service Control is a single binary: a null pointer in any component crashes everything. In a modularized architecture, the quota checking subsystem can crash or be restarted without affecting authentication. This is the same architectural move GitHub is making with service isolation — and it is the right call for the same reason: blast radius containment. The cost of modularizing a critical monolithic binary is high. The cost of not doing it is measured in seven-hour global outages.
\n
\n\n\nTHE IRONY OF THE RECOVERY MAKING THINGS WORSE
\nThe most memorable aspect of the June 12 outage is the herd effect: the act of fixing the crash created a new problem that extended the outage by hours. This pattern — where the recovery mechanism amplifies the damage — appears across some of the most instructive engineering post-mortems in the industry. Netflix built Chaos Monkey partly because they discovered that their graceful degradation paths, when triggered simultaneously by a real failure, could overload the systems they were supposed to protect. AWS's October 2025 DynamoDB outage extended 12 extra hours because the automatic failover mechanisms were making the DNS inconsistency worse. And Google's June 2025 Service Control outage extended beyond the crash itself because hundreds of instances restarting simultaneously overwhelmed the Spanner database they all depended on. The lesson that runs through all three: design recovery paths with the same care you design primary paths — because recovery paths are the code that runs when everything is already wrong.
\n
\n\nGoogle deployed a null pointer exception on May 29, and it sat patiently in production for two weeks waiting for exactly the wrong policy update to arrive — like a trapdoor that looks like a floor until someone with the right keycard walks over it. Then it took down Spotify, Discord, Snapchat, Cloudflare, and Google's own status page simultaneously, and when engineers hit the kill switch to fix it, the restart surge broke the database they were restarting into. There is a version of this story where the lesson is 'write a null check.' There is a more useful version where the lesson is: the most dangerous code in your system is the code that runs perfectly for two weeks before it fails catastrophically, because you will have completely forgotten it's there by the time it does.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-31T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.378232+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Distributed Systems", "Google"]}, {"id": "https://techlogstack.com/explore/spotify-envoy-proxy-outage-2025/", "url": "https://techlogstack.com/explore/spotify-envoy-proxy-outage-2025/", "title": "Spotify Changed a Filter Order in Their Proxy — Then Every Server in the World Crashed at Once", "summary": "How a low-risk Envoy filter reorder triggered a latent bug, crashed every proxy instance simultaneously, and started a Kubernetes memory-limit death loop that kept S", "content_html": "Spotify · Reliability · 24 May 2026
\nOn April 16, 2025, Spotify's engineering team made a change they deemed low risk: reordering the custom filters inside their Envoy Proxy perimeter. They applied it to all regions simultaneously. Within two minutes, every Envoy instance worldwide had crashed. And then the restart loop began — a loop Kubernetes itself was powering, killing each new server as fast as it came back up. 675 million users couldn't load the app. Asia Pacific stayed up, and the reason why told the engineers exactly what was broken.
\n\nThis crash happened simultaneously on all Envoy instances.
— — Spotify Engineering — Incident Report: Spotify Outage on April 16, 2025, engineering.atspotify.com
There is a specific kind of engineering failure that hurts more than the others: the change that was reviewed, discussed, and approved — the change the team looked at together and agreed was fine. On April 16, 2025, Spotify's team reordered the custom filters within their Envoy Proxy (an open-source, high-performance edge proxy originally built at Lyft and now widely used as the networking perimeter layer in distributed systems — it receives all incoming user traffic before distributing it to backend services) perimeter. This was not a new feature, not a database migration, not a major infrastructure overhaul. It was a filter reorder. The team assessed it as low risk. They applied it to all cloud regions simultaneously. Two minutes later, every single Envoy instance running Spotify's networking perimeter had crashed.
\nSpotify's perimeter is the first layer of software that receives traffic from every user worldwide — every stream request, every search, every login. It sits in front of all backend services and distributes traffic across cloud regions. To extend Envoy's capabilities, Spotify develops and maintains its own custom filters — plugins that run within Envoy to handle rate limiting, authentication, and other cross-cutting concerns. These filters execute in a defined order. The April 16 change altered that order. The new sequence triggered a latent bug in one of the custom filters: a code path that had existed undetected, harmless as long as the filter never received control at that position, suddenly activated. Envoy crashed. Not one instance, not one region. All of them.
\n\n\n\nTHE DEATH LOOP: WHY THE RESTART MADE THINGS WORSE
\nAn Envoy crash is normally survivable — Kubernetes detects the failed pod and starts a replacement. But what happened next on April 16 was not normal. The immediate restart of all Envoy instances, combined with client-side retry logic (every user's app and browser retrying its failed request), created an unprecedented traffic spike onto the new instances. Each new Envoy started, received the full flood of retry traffic, consumed more memory than the Kubernetes memory limit (the maximum memory a Kubernetes pod is allowed to use, defined in its resource spec — when a pod exceeds this limit, Kubernetes automatically terminates it regardless of what the pod is doing), and was automatically killed by Kubernetes. A new instance started. The same thing happened. The loop repeated — powered by Kubernetes itself — for hours.
\n
The change to Envoy filter execution order was applied simultaneously to all cloud regions worldwide. The new order activated a latent bug in a custom Spotify filter. Every Envoy instance on Spotify's networking perimeter crashed at the same moment. Alarms fired two minutes later as the traffic drop became measurable.
\nThe traffic flood from client retries exposed a misconfiguration that had existed undetected: Envoy's max heap size was configured higher than the Kubernetes memory limit for the pod. Under normal traffic, Envoy never approached its heap limit and the misconfiguration was invisible. Under the retry flood, each new instance immediately exceeded the K8s limit and was killed. This turned a crash into an infinite restart loop.
\nAsia Pacific was the only region unaffected. Engineers noticed and investigated why. The answer: lower traffic volume at that time of day (timezone difference) meant APAC Envoy instances never received enough retry traffic to exceed the K8s memory limit. The asymmetry proved the hypothesis: the death loop was memory-limit driven, not bug-driven. Fix the memory headroom, break the loop.
\nIncreasing total perimeter server capacity gave each new Envoy instance enough headroom to stay under the K8s memory limit even while absorbing the retry traffic flood. The death loop broke. Instances stabilized. EU recovered at 14:20 UTC, US at 15:10 UTC, full normalization at 15:40 UTC. Total duration: 3 hours 27 minutes.
\n\n\n\n🌏
\nAsia Pacific's survival was not a result of better engineering in that region. It was a result of time zones. The outage struck at 12:18 UTC — early morning in Europe and the US, but late evening in Asia Pacific where traffic was naturally lower. Fewer users → fewer retries → less memory pressure on new Envoy instances → stayed under the K8s limit. The region that wasn't affected was the one that happened to have the least traffic at the exact wrong moment.
\n
\n\n\n⚠️
\nWhy 'Low Risk' Was Wrong
The filter reorder was assessed as low risk for a defensible reason: reordering filters does not add new code or change individual filter logic. It changes the sequence in which existing, tested filters run. The team's mental model was correct for most cases — but it was missing one scenario: a latent bug in a filter that only activates when that filter receives control at a specific position in the execution chain. Latent bugs that depend on execution context are invisible to tests that don't vary that context. A filter integration test suite that exercises filters in isolation or in their original order will never catch a bug that only manifests in a new order.
\n
The mechanism of the death loop is worth understanding in precise detail because it recurs across infrastructure outages in different forms. The pattern is: a failure event triggers a restart, the restart environment differs from steady state (here: retry flood instead of normal traffic), the restarted instance fails faster than under steady state, the restart mechanism itself (Kubernetes) becomes the engine of the failure. In Kubernetes deployments, the most common version of this pattern involves the OOMKill (Out-Of-Memory Kill — the Kubernetes mechanism that terminates a pod when it exceeds its configured memory limit, to protect other workloads on the node from memory starvation) cycle: a pod exceeds its memory limit under unexpected load, K8s kills it, the replacement starts with no warm state and faces the same load, K8s kills it again. The Spotify outage was this pattern at global perimeter scale.
\n\n\n\nℹ️
\nEnvoyCon 2025: The Custom Filter Spotify Presented
Spotify's engineering team had discussed their custom Envoy filter work publicly — including their rate limiting filter — at EnvoyCon 2025, just before the April 16 incident. The presentation described the same filter system that the April 16 change modified. The public talk was about the filter's capabilities; the April 16 postmortem was about what happened when its execution order changed. The two documents together give a rare complete picture: how the system was designed to work, and exactly how it broke.
\n
The 263 million Spotify Premium subscribers who pay for the service experienced the same outage as the 412 million free-tier users. Spotify's architecture does not provide differentiated reliability between paid and unpaid users at the perimeter layer — the same Envoy proxy handles all traffic. This is consistent with how most streaming platforms operate: the perimeter is a shared resource, and a perimeter failure is total. The 48,000+ Downdetector reports at peak represented the fraction of users actively reporting issues; the actual count of affected users was in the hundreds of millions.
\n\n\n\n🔄
\nThe Retry Amplification Problem
Client-side retry logic is a standard reliability feature: when a request fails, the app retries it, giving transient failures a chance to self-heal. During normal partial failures, retry logic helps. During a simultaneous total failure, retry logic becomes a load amplifier. Every user whose request failed immediately retried — some apps retry multiple times with exponential backoff. The simultaneous crash of all Envoy instances converted the normal traffic level into a retry-amplified spike: each failed request generated one or more retry requests, all arriving at the same moment the replacement instances were starting. The retry logic designed to improve reliability became a key component of the death loop.
\n
\n\n\n❌
\n263 Million Premium Subscribers: No Differentiation
Spotify's perimeter architecture treats all traffic identically — there is no fast lane for paid subscribers at the proxy layer. The 263 million Premium users who pay for an ad-free, uninterrupted experience were indistinguishable from the 412 million free-tier users when the perimeter crashed. This is not a design flaw; building separate perimeter infrastructure for each tier would add enormous complexity. But it means that the reliability guarantee implicit in a Premium subscription depends entirely on the reliability of shared perimeter infrastructure. When the perimeter fails totally, the premium experience fails identically to the free experience.
\n
The fix for the death loop was increasing perimeter server capacity — but this addressed the symptom, not the underlying misconfiguration. The root problem was that Envoy's max heap size was set higher than the Kubernetes memory limit for the pod. In normal operation, Envoy memory usage never approached its heap maximum. The misconfiguration was invisible: Envoy wasn't crashing, K8s wasn't killing pods, monitoring wasn't alerting. The gap between heap size and K8s limit existed in the configuration for an unknown period before April 16 — it simply never mattered because Envoy memory never climbed high enough to expose it. The retry flood was the first event extreme enough to push instances over the K8s limit and trigger the kill cycle.
\n# THE MISCONFIGURATION: Envoy heap limit higher than K8s memory limit\n# This created a hidden gap that was invisible until the retry flood\n\n# Kubernetes pod resource specification (simplified)\napiVersion: v1\nkind: Pod\nspec:\n containers:\n - name: envoy\n resources:\n requests:\n memory: \"2Gi\"\n limits:\n memory: \"3Gi\" # K8s will OOMKill the pod above this\n\n# Envoy overload manager configuration (simplified)\noverload_manager:\n actions:\n - name: envoy.overload_actions.stop_accepting_requests\n triggers:\n - name: envoy.resource_monitors.fixed_heap\n threshold:\n value: 0.95 # Envoy's own heap limit: 95% of max heap\n resource_monitors:\n - name: envoy.resource_monitors.fixed_heap\n typed_config:\n max_heap_size_bytes: 4294967296 # 4GB: HIGHER than K8s 3GB limit!\n\n# Result:\n# - K8s kills at 3GB of memory usage\n# - Envoy's own safety valve triggers at 95% of 4GB = 3.84GB\n# - K8s limit is hit BEFORE Envoy's own graceful degradation kicks in\n# - Under normal load: Envoy peaks at ~1.5GB — misconfiguration invisible\n# - Under retry flood: Envoy climbs past 3GB → K8s OOMKills → restart\n# → same flood → same OOMKill → infinite loop\n\n# THE FIX (immediate): Increase perimeter server count\n# More servers = same retry traffic spread across more instances\n# = each instance stays under 3GB = K8s doesn't kill = loop breaks\n\n# THE FIX (permanent): Align heap config with K8s memory limit\n# max_heap_size_bytes: 2684354560 # 2.5GB: safely below K8s 3GB limit\n\n\nWHY INCREASING CAPACITY FIXED THE LOOP
\nThe death loop's engine was the K8s OOMKill cycle. The K8s memory limit was fixed. The retry traffic load was fixed (determined by user behavior). The only variable Spotify could change quickly was the number of Envoy instances sharing that retry load. More instances → each instance receives a smaller share of the retry flood → each instance's memory usage stays lower → stays under the K8s limit → K8s doesn't kill it → stable. This is why increasing capacity broke the loop: it reduced per-instance memory pressure below the kill threshold. The underlying misconfiguration (heap > K8s limit) was fixed separately afterward as a permanent remediation.
\n
\n\n\nℹ️
\nSpotify's Four Post-Incident Commitments
Spotify's postmortem committed to four specific engineering changes: (1) Fix the Envoy filter bug that caused the initial crash on filter reorder. (2) Fix the configuration mismatch between Envoy heap size and Kubernetes memory limit. (3) Improve the rollout process for configuration changes to the perimeter — staged rather than global simultaneous. (4) Improve monitoring capabilities to detect these issues earlier in the failure chain. Notably, the postmortem linked directly to their EnvoyCon 2025 talk, showing transparency about which system was involved rather than obscuring the component.
\n
\n\n\n✅
\nWhat a Staged Rollout Would Have Caught
If the filter reorder had been applied to a single region first with a monitoring window before global rollout, the failure would have been a regional incident recoverable in minutes, not a global outage lasting 3.5 hours. The Envoy crash would have appeared in one region. Engineers would have investigated, found the latent bug, rolled back the filter order. Total blast radius: one region for ~10 minutes. The simultaneous global rollout removed this safety net entirely. The misconfiguration would still have existed — but would have been exposed only in one region rather than all at once.
\n
Spotify Envoy Outage: Timeline of Events and Recovery Progression
| Time (UTC) | Event | Status |
|---|---|---|
| 12:18 | Envoy filter order changed; all instances crash simultaneously | 🔴 Global failure begins |
| 12:20 | Alarms triggered on traffic drop; death loop already running | 🔴 Engineers paged |
| 12:28 | Situation escalated; only APAC serving traffic | 🔴 Incident declared |
| ~13:xx | Root cause identified via APAC asymmetry; capacity increase planned | 🟡 Diagnosis complete |
| 14:20 | EU regions fully recovered | 🟡 Partial recovery |
| 15:10 | US regions fully recovered | 🟡 Partial recovery |
| 15:40 | All traffic patterns normal globally | 🟢 Full recovery |
\n\n\nℹ️
\nThe Perimeter as a Shared Fate System
Spotify's Envoy perimeter is a shared fate system — all backend services rise and fall with it. Even if every backend service (streaming, search, auth, recommendations) remained perfectly healthy during the April 16 incident, users experienced total unavailability because the perimeter through which all requests flow was in a crash loop. This architectural property is why perimeter changes deserve the highest rollout rigor: a perimeter failure has a blast radius of every service, every user, every region simultaneously. The perimeter is where shared fate is most acute.
\n
Spotify's networking perimeter architecture places Envoy Proxy as the outermost layer — the first software that receives every user request, regardless of what feature or backend it is ultimately destined for. Envoy runs in every cloud region and distributes incoming traffic to the appropriate backend microservices. Custom filters extend Envoy's capabilities beyond the open-source defaults: rate limiting, authentication, request routing customization. Understanding the outage requires understanding that when every Envoy instance worldwide crashes simultaneously, no user request can reach any backend service — the entire platform goes dark regardless of whether individual backend services remain healthy.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE ASIA PACIFIC DIAGNOSTIC: HOW ONE REGION PROVED THE ROOT CAUSE
\nThe most important engineering insight in the April 16 incident was not the initial cause — it was using Asia Pacific's survival to prove the diagnosis. When engineers observed APAC was unaffected, they had two candidate hypotheses: (A) the filter bug is region-specific, or (B) the death loop is traffic-intensity dependent. If (A), APAC has different filter config. If (B), APAC has lower traffic at this hour. Investigation confirmed (B): APAC runs identical filter configuration. Lower traffic meant less retry amplification, meaning per-instance memory pressure never reached the K8s limit. This asymmetry transformed a hard debugging problem (why is the loop happening?) into a tractable one (what's different about APAC?) and pointed directly at the memory-limit misconfiguration.
\n
\n\n\n⚠️
\nConfiguration Drift in Long-Running Systems
The Envoy heap/K8s limit misconfiguration almost certainly existed long before April 16, 2025. It was never caught because Envoy memory usage never reached the dangerous threshold under normal traffic. This is a common pattern: configuration mismatches that are only dangerous under abnormal load go undetected indefinitely in systems where abnormal load doesn't occur. The misconfiguration didn't cause the outage — the filter bug did. But the misconfiguration was what turned a recoverable crash into a multi-hour global outage. Auditing resource limit configurations against actual peak usage — including synthetic stress tests — is the practice that catches these time bombs before they detonate.
\n
\n\n\n🛡️
\nEnvoy's Custom Filter Architecture
Spotify's custom Envoy filters are the engineering investment that makes this outage particularly instructive. Envoy provides a well-defined filter chain (the ordered sequence of processing modules (filters) that each request passes through in an Envoy proxy instance — each filter can inspect, modify, or reject the request before passing it to the next filter in the chain) mechanism for extensibility: developers write C++ or Lua filter plugins that plug into Envoy's request processing pipeline. The order of filters in the chain determines the sequence of execution. Spotify's filters included rate limiting (discussed at EnvoyCon 2025) and other custom logic. Changing this order is semantically meaningful: a filter that assumes it runs after authentication may behave incorrectly if it suddenly runs before it. The latent bug on April 16 was exactly this class of assumption.
\n
The Spotify April 2025 outage is one of the cleanest documented examples of how a reasonable assessment ('low risk') combined with an undetected misconfiguration produces a disproportionate outcome. The lessons here are deeply practical.
\nWhat to remember
\n\n\n✅
\nSpotify's Transparency Standard
The April 16 postmortem was published on May 9 — 23 days after the incident. It named the specific system involved (Envoy Proxy, their custom filters), linked directly to their EnvoyCon 2025 talk about the same system, included exact timestamps for every recovery milestone, and explained the death loop mechanism in precise technical terms. It also enumerated four specific engineering commitments — not aspirational language but concrete actions. This level of technical transparency in a public postmortem is rare and sets a standard. Spotify's engineering culture treats accountability as a tool for improvement, not a liability to be managed.
\n
\n\n\nTHE ENVOYCON IRONY
\nSpotify's engineering team had publicly presented their custom Envoy filter work — including the rate limiting filter — at EnvoyCon 2025, just weeks before the April 16 incident. The presentation described the filter system as a capability Spotify had built to enhance Envoy's performance. The April 16 postmortem described what happened when the execution order of that same filter system was changed without a staged rollout. The two documents together are an accidental case study in the gap between how a system is designed to work and how it fails under an unexpected configuration change. Publishing both — the capability presentation and the failure postmortem — is a model of engineering transparency.
\n
\n\nSpotify changed the order of some filters in their proxy, which seemed fine until every server on Earth crashed simultaneously, and then Kubernetes helpfully restarted them all into the same crash in a loop — which is either a distributed systems problem or a distributed systems feature depending on how you look at it.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-24T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.979968+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "Spotify"]}, {"id": "https://techlogstack.com/explore/airbnb-identity-graph-janusgraph-2026/", "url": "https://techlogstack.com/explore/airbnb-identity-graph-janusgraph-2026/", "title": "Airbnb's Fraud Detection Runs on a Graph of 7 Billion Nodes — Here's Why They Rebuilt It From Scratch", "summary": "How Airbnb rebuilt their 7-billion-node fraud detection identity graph from a third-party SaaS on JanusGraph + DynamoDB, cutting P99 read latency by 49% and enabling", "content_html": "Airbnb · Databases · 24 May 2026
\nAirbnb's identity graph connects 7 billion nodes and 11 billion edges — every user, every device, every listing, every relationship that might reveal a fraudster trying to create a duplicate account or collude on a fake transaction. The third-party vendor powering it required periodic manual reboots to stay stable. Queries that needed 8 hops of graph traversal were hitting 5-second P99 latencies. In 2024, a small team rebuilt the entire thing internally. The results were not incremental.
\nThe stakes of Airbnb's identity graph are not abstract. When a fraudster creates a second account after being banned, tries to rent a listing to damage it, or coordinates with other accounts to inflate reviews, the first system that needs to detect the connection is the identity graph. It holds the relationships between every user, every device, every verified identity, every behavioral signal that Airbnb's Trust and Safety team uses to determine whether a new account is truly new or a known bad actor resurfacing. In 2024, this graph contained 7 billion nodes and 11 billion edges — and was growing by 5 million new edges every day. The vendor powering it was requiring periodic manual reboots to stay stable. That was the state of Airbnb's most critical fraud detection infrastructure when the decision was made to build internally.
\n\n\n\n🕸️
\nAirbnb's identity graph is not the only graph at the company — it was simply the first to migrate to the new internal graph infrastructure. The platform also runs inventory knowledge graphs (relationships between listings, amenities, neighborhoods, and availability) and data lineage graphs (tracking how data flows through pipelines for compliance and debugging). All are now converging onto the same JanusGraph-based infrastructure.
\n
The identity graph's architecture progressed through three distinct generations, each solving the previous generation's limit while introducing new constraints. The first generation used a relational database for user and entity data paired with a key-value store holding JSON-encoded edge lists. This worked at low graph density — when users had few connections, the JSON edge lists were manageable. As graph density increased and individual users accumulated hundreds or thousands of edges, the JSON edge lists became expensive to read and update. A query that needed to traverse relationships between users required deserializing large JSON blobs and joining across tables — operations that relational databases (database systems built around tables, rows, and SQL joins — optimal for normalized structured data but increasingly expensive as relationship traversal depth grows, because each hop requires an additional join) are not optimized for at graph scale.
\n\n\n\nTHE FOUR ANTI-PATTERNS THAT PLAGUED AIRBNB'S GRAPH TEAMS
\nBefore the centralized graph infrastructure, Airbnb teams building graph-based products fell into four documented anti-patterns. Relational graphs: modeling nodes and edges in SQL tables, producing expensive joins during traversal. Offline graphs: building the graph in the data warehouse, limiting data freshness to daily batch snapshots — useless for real-time fraud detection. DIY open source: self-managing community versions of graph databases, creating high operational toil and expertise silos. Managed PaaS: using third-party vendors — better operationally but introducing vendor lock-in, limited tuning access, and performance bottlenecks the team couldn't debug. The identity graph's 2021 migration to a third-party SaaS solved the operational overhead of DIY open source but introduced the last anti-pattern.
\n
The first-generation architecture used a relational database for entity data and a KV store holding JSON-encoded edge lists. As the identity graph grew in density — individual users accumulating hundreds of edges — querying became expensive. JSON deserialization and cross-table joins are not optimized for the multi-hop traversal patterns that fraud detection requires. The architecture became difficult and expensive to scale.
\nThe 2021 migration to a third-party SaaS graph database improved horizontal scalability. But it introduced new problems: long-tail latency (P99 read latency reaching 5 seconds on 8-hop queries), operational instability requiring periodic manual reboots, limited ability to tune performance for Airbnb's specific query patterns, and no fine-grained access controls. The vendor was a black box the team couldn't debug.
\nIn 2024, Airbnb built an internal graph infrastructure on JanusGraph (open-source, Apache TinkerPop stack, Gremlin query language) with DynamoDB as the storage backend and OpenSearch for indexing. The pluggable storage architecture meant Airbnb could leverage DynamoDB's operational reliability without reinventing distributed storage — while maintaining full control over the graph logic layer. They forked JanusGraph internally to add custom optimizations.
\nP99 read end-to-end latency dropped from 5.0s to 2.5s (-49%). P95 from 2.1s to 1.0s (-51%). Write P95 from 353ms to 156ms (-56%). Write QPS during load testing: 10× the previous vendor's maximum. Manual reboots eliminated entirely. Incident investigation time shortened through transparent internal observability. Auto-scaling enabled for the first time.
\n\n\n\n⚠️
\nThe Long-Tail Latency Problem at High Fanout
The most technically interesting challenge in the identity graph is long-tail latency (the phenomenon where the slowest requests in a system (P95, P99) are dramatically slower than the median — particularly damaging for real-time applications where even a small fraction of slow responses degrades user experience) on high-fanout queries. The graph is not uniformly dense — some nodes (users who have been on Airbnb for years and made hundreds of bookings) have hundreds or thousands of edges. When a fraud detection query traverses relationships at depth and hits one of these high-fanout nodes, the amount of data retrieved explodes. A query with 4–8 hops that hits a high-fanout node at hop 2 can return orders of magnitude more data than the same query on a sparse node. The P50 latency looks fine; the P99 reveals the reality.
\n
The choice of Gremlin as the query language was not coincidental — it was a migration enabler. Both the outgoing vendor system and the incoming JanusGraph implementation support Gremlin (a graph traversal language developed as part of the Apache TinkerPop framework — reads like a path through the graph, e.g. g.V(userId).out('booked').in('listed') means 'find all users who listed properties that this user has booked'), which meant Airbnb could run the same Gremlin queries against both systems simultaneously during the migration. This shadow traffic approach allowed direct performance benchmarking under real production load before any cutover — a stark contrast to migrations that require rewriting queries for the new system before they can be tested. The query language compatibility was a deliberate evaluation criterion, not an accident.
\n\n\n\nℹ️
\nTrust Graph: The Fraud Detection Use Case
Airbnb's internal name for the identity graph's fraud detection application is the Trust Graph. It models connections between Airbnb users and detects two primary fraud patterns: account duplication (a banned user creating a new account and re-joining the platform) and collusion (groups of accounts coordinating to execute fraudulent transactions, inflate review scores, or circumvent platform policies). The Trust Graph feeds ML models that learn patterns of fraudulent connectivity — the specific graph structures that appear before fraud events — and score new accounts and transactions in real time. For this use case, query latency directly impacts both fraud detection speed and host/guest experience during booking.
\n
\n\n\n📦
\nStorage Separation: Why DynamoDB as the Backend
JanusGraph's pluggable storage architecture was the property that made it the right choice for Airbnb. By using DynamoDB as the storage backend, Airbnb decoupled graph logic (JanusGraph) from distributed storage operations (DynamoDB). DynamoDB brings auto-scaling, multi-region replication, and operational reliability that Airbnb's infrastructure team already understood and trusted. JanusGraph handles the graph data model, schema management, and Gremlin query execution. The combination gave Airbnb full control over the graph layer while standing on a storage foundation that didn't need to be invented from scratch.
\n
\n\n\n⚠️
\n5 Million New Edges Per Day: The Write Problem
The identity graph's write challenge is as significant as its read challenge. Every day, Airbnb adds approximately 5 million new edges — new bookings creating host-guest relationships, new device associations, new identity verification links. Each edge must be ingested in near real-time through asynchronous events and stored durably before downstream fraud models can use them. The vendor system's write P95 of 353ms was tolerable when the graph was smaller. At 11 billion edges growing at 5 million per day, write throughput headroom becomes critical — a system at its write QPS ceiling cannot absorb traffic spikes without dropping ingestion events. The internal solution's 10× write QPS ceiling during load testing created the headroom the vendor never provided.
\n
The decision framework Airbnb used to evaluate JanusGraph against alternatives reflects a principled approach to graph database selection that is worth examining. Four requirements shaped the evaluation: scalability for online queries (the system had to handle real-time graph traversal at P95 latencies under 500ms), expressive schema and query capabilities (the Gremlin traversal language and labeled property graph model needed to support the identity graph's data model without structural compromises), fit with Airbnb's infrastructure and operational model (DynamoDB as the storage backend meant the team was standing on infrastructure they already operated), and a visible, extensible codebase — the specific requirement that ruled out the previous vendor and every other black-box alternative. Access to the source code was not a preference; it was a prerequisite.
\n\n\n\nCONNECTED ACCOUNTS: HOW THE GRAPH DETECTS FRAUD
\nThe specific fraud detection application that depends on the identity graph is called Connected Accounts (also referred to as the Trust Graph). It works by finding structural patterns in the graph that correlate with fraud. A legitimate user typically has one main account, one primary device, and verified identity credentials. A fraudster attempting to re-enter the platform after a ban might create a new account — but often reuses the same phone number, the same payment method, the same device, or overlaps with the banned account's booking history. The Connected Accounts system traverses the graph to find these connections: \"this new account shares a device with a banned account, which shared a payment method with another banned account, which has reviewed listings that the new account also reviewed.\" That traversal pattern — spanning 4–8 hops — is exactly why graph depth performance matters for fraud detection.
\n
Deploying stock JanusGraph with DynamoDB would not have been sufficient — Airbnb's query patterns, particularly the high-fanout traversals that caused the worst P99 spikes, required modifications to the JanusGraph engine itself. The team forked JanusGraph internally and made three targeted optimizations: replacing the default locking mechanism with a DynamoDB-native approach, adding parallel execution to the high-fanout data fetching interface, and instrumenting the internal fork with distributed tracing that the open-source version lacked. Together, these changes addressed the specific failure modes that had made the vendor solution unacceptable.
\n# Conceptual illustration of the three JanusGraph engine optimizations\n# that reduced long-tail latency on Airbnb's identity graph\n\n# OPTIMIZATION 1: Custom transaction strategy using DynamoDB conditional writes\n# JanusGraph's default locking: acquire explicit distributed lock before write\n# Problem: lock acquisition is a round-trip to the storage backend = overhead\n\n# Old approach (simplified — uses JanusGraph's default locking):\ndef write_edge_default(tx, src_vertex, dst_vertex, edge_label):\n lock = acquire_distributed_lock(src_vertex, edge_label) # expensive\n try:\n tx.add_edge(src_vertex, dst_vertex, edge_label)\n tx.commit()\n finally:\n release_lock(lock)\n\n# New approach: DynamoDB conditional writes (atomic compare-and-swap)\n# DynamoDB's native conditional writes ensure integrity without separate lock\ndef write_edge_optimized(tx, src_vertex, dst_vertex, edge_label):\n # Condition: only write if edge doesn't already exist\n # DynamoDB evaluates the condition atomically server-side — no round-trip lock\n tx.add_edge_with_condition(\n src_vertex, dst_vertex, edge_label,\n condition=\"attribute_not_exists(edge_key)\" # DynamoDB conditional expression\n ) # Lower overhead, same integrity guarantee\n\n# OPTIMIZATION 2: Parallel query execution via improved getMultiSlices\n# Problem: high-fanout queries (user with 1000+ edges) fetch data serially\n# Each 'slice' of edge data retrieved one at a time from DynamoDB\n\n# Before: serial fetching of edge slices\ndef get_edges_serial(vertex_id, num_slices=50):\n results = []\n for slice_key in compute_slice_keys(vertex_id, num_slices):\n results.append(dynamo.get_item(slice_key)) # Sequential round-trips\n return merge(results) # N sequential DynamoDB calls\n\n# After: parallel fetching via improved getMultiSlices interface\ndef get_edges_parallel(vertex_id, num_slices=50):\n slice_keys = compute_slice_keys(vertex_id, num_slices)\n # DynamoDB BatchGetItem fetches all slices concurrently\n results = dynamo.batch_get_items(slice_keys) # Single batched DynamoDB call\n return merge(results) # 1 call instead of N — critical for high-fanout nodes\n\n# OPTIMIZATION 3: Distributed tracing integrated into JanusGraph fork\n# OSS JanusGraph: no tracing instrumentation — impossible to profile slow queries\n# Internal fork: Airbnb's distributed trace context propagated through graph ops\ndef execute_gremlin_traversal(query, trace_context):\n with airbnb_tracer.start_span('janusgraph.traversal', parent=trace_context) as span:\n span.set_tag('query.hops', count_hops(query))\n span.set_tag('query.fanout', estimated_fanout(query))\n result = janusgraph.execute(query) # Each DynamoDB call creates child spans\n span.set_tag('result.edges_traversed', result.edge_count)\n return result # Full trace: query → graph ops → DynamoDB calls → result\n\n\nGREMLIN QUERY REWRITING: CLIENT-SIDE OPTIMIZATION
\nEven with a faster JanusGraph engine, Airbnb discovered that identical Gremlin queries produced significantly different performance on JanusGraph compared to the vendor system — because each implements different query planning optimizations over TinkerPop steps. Two specific patterns required client-side rewrites. Path steps (Gremlin's
\npath()andsimplePath()operators) are not optimized as batched queries in JanusGraph, causing non-batched storage backend queries that saturate the DynamoDB connection pool. These were replaced with conditional queries ensuring acyclic results. Side-effect aggregation steps produced non-batched substeps in JanusGraph's query planner and were restructured to minimize unoptimized computation. Both changes required deep knowledge of JanusGraph's query planning internals — knowledge only available because Airbnb owns the code.
\n\n\nℹ️
\nSchema Enforcement via the Management Service
JanusGraph's open-source version ships with minimal schema management tooling. Airbnb built a Graph Management Service on top of JanusGraph to handle schema enforcement, index management, and schematized Thrift APIs for client access. The management service acts as the control plane for the graph infrastructure: it ensures that vertex and edge schemas are validated before data is written, manages secondary indexes in OpenSearch, and provides the typed API surface that downstream services (fraud detection models, Trust & Safety pipelines) call. This separates schema governance from query execution and prevents different teams' graph data from colliding in a multi-tenant infrastructure.
\n
\n\n\n✅
\nThe Shadow Traffic Migration Strategy
Migrating 7 billion nodes and 11 billion edges without downtime required a migration strategy that validated the new system under real production load before any cutover. Airbnb's approach: run both the vendor system and the internal JanusGraph system in parallel, routing the same production queries to both and comparing results. Because both systems use Gremlin, the same queries ran unchanged on both systems simultaneously. This shadow traffic phase provided two things: a performance benchmark under real load (not synthetic tests), and correctness validation (ensuring internal JanusGraph returned the same results as the vendor for the same queries). Only after shadow traffic validated both correctness and performance was production traffic cut over and the vendor deprecated.
\n
Third-Party Vendor vs Internal JanusGraph: Performance Comparison Across Query Types
| Query Type | Vendor P95 | Internal P95 | Improvement | Vendor P99 | Internal P99 |
|---|---|---|---|---|---|
| 1-hop query | ~180ms | ~65ms | -64% | ~420ms | ~150ms |
| 2-hop query | ~350ms | ~130ms | -63% | ~900ms | ~280ms |
| 2-hop (high fanout) | ~620ms | ~200ms | -68% | ~1,800ms | ~450ms |
| 4-hop query | ~900ms | ~380ms | -58% | ~2,500ms | ~850ms |
| 8-hop query (max depth) | ~2,100ms | ~1,000ms | -52% | ~5,000ms | ~2,500ms |
| Write (edge creation) | ~353ms | ~156ms | -56% | ~800ms | ~360ms |
\n\n\nℹ️
\nTwo Ingestion Paths: Event-Driven and Bulk Load
The identity graph's data ingestion architecture has two distinct paths. Event-driven ingestion handles the 5 million daily new edges in near real-time — asynchronous events from Airbnb's platform (new booking, new device association, identity verification) trigger graph mutations through the JanusGraph write path within seconds of occurring. Bulk loading handles backfills, historical data migrations, and large-scale data corrections — optimized for high throughput rather than low latency, running as offline jobs that write directly to DynamoDB's storage layer. The two paths are served by separate applications in the identity graph service, ensuring that bulk load operations during data migrations don't contend with real-time fraud detection writes.
\n
The architecture of Airbnb's new graph infrastructure has three conceptual layers. The storage layer is DynamoDB for graph data persistence and OpenSearch for secondary indexes — both managed AWS services that auto-scale without Airbnb managing the distributed storage operations. The graph engine layer is Airbnb's internal JanusGraph fork — the Gremlin server that executes traversal queries against the storage layer, with custom optimizations for Airbnb's specific access patterns. The management layer is the Graph Management Service — schema enforcement, index management, multi-tenant namespace isolation, and the Thrift API surface that client services call. Each tenant (the identity graph, inventory knowledge graph, data lineage graph) operates in an isolated namespace within the same infrastructure.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nJANUSGRAPH'S PLUGGABLE STORAGE: THE ARCHITECTURAL DECISION THAT MADE THIS POSSIBLE
\nMost graph databases tightly couple the graph logic layer with the storage layer — the query engine and the data store are one system. JanusGraph is architecturally different: it uses a pluggable storage backend, meaning the graph logic layer (Gremlin server, transaction management, schema enforcement) can be decoupled from the distributed storage layer. Airbnb chose DynamoDB as the backend — a service their infrastructure team already operated at scale and trusted. This separation gave Airbnb the ability to: iterate on graph engine features without touching storage operations, leverage DynamoDB's auto-scaling for write throughput bursts (like the 5M daily edges), and evolve the storage backend in the future without rewriting the graph layer.
\n
\n\n\n⚠️
\nThe Open Source Observability Gap
One of the specific problems that drove Airbnb to fork JanusGraph internally was the absence of distributed tracing in the open-source version. Without tracing, there was no way to profile which graph queries were slow, which DynamoDB operations within a query were taking the most time, or which high-fanout nodes were causing P99 spikes. Debugging latency issues required guesswork or custom logging that was expensive to build and maintain. The internal fork integrated Airbnb's distributed tracing infrastructure into every graph operation — Gremlin steps, DynamoDB calls, OpenSearch index queries — giving the team the observability needed to find and fix the exact operations driving long-tail latency. You cannot optimize what you cannot measure; you cannot measure what you cannot instrument.
\n
Airbnb's identity graph migration is a case study in the specific moment when the accumulation of vendor limitations justifies the cost of building internally — and in the engineering decisions that made the build worth the investment. The lessons are as much about when to leave a vendor as about how to build a graph database.
\nWhat to remember
\n\n\n✅
\n7 Billion Nodes, One Platform, Multiple Use Cases
The identity graph was the first use case to migrate to Airbnb's internal graph infrastructure — but the infrastructure was built as a multi-tenant platform from the start. Inventory knowledge graphs (relationships between listings, neighborhoods, experiences, amenities) and data lineage graphs (how data flows through Airbnb's pipelines) are among the next use cases converging onto the same JanusGraph infrastructure. Building the identity graph migration as a paved-path platform rather than a one-off solution means every subsequent team that needs a graph database inherits the operational work Airbnb already did on the first migration — schema tooling, observability, auto-scaling, migration playbook.
\n
\n\n\nTHE MULTI-TENANT PLATFORM PAYOFF
\nAirbnb built the internal graph infrastructure as a paved-path multi-tenant platform from the start — a conscious architectural decision to build once and serve many graph use cases, rather than building a one-off solution for the identity graph. The identity graph was tenant 0: the first adopter that validated the platform under real production load. Inventory knowledge graphs and data lineage graphs are following. Each new tenant inherits the schema tooling, observability, auto-scaling, query optimization work, and migration playbook that the identity graph team built. The marginal cost of the second graph use case is dramatically lower than the first, because the platform absorbs the infrastructure complexity.
\n
\n\nAirbnb's fraud detection system runs on a graph of 7 billion nodes and 11 billion edges that required periodic manual reboots to stay stable — until a team rebuilt it internally, cut P99 latency in half, and eliminated the reboots, which raises the question of why they waited until 2024 to do it, but the answer is probably 'because that's how long it takes to get frustrated enough with a vendor.'
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-24T00:00:00+00:00", "date_modified": "2026-06-13T19:05:03.956856+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Databases", "Airbnb"]}, {"id": "https://techlogstack.com/explore/google-stitch-ai-design-tool-2026/", "url": "https://techlogstack.com/explore/google-stitch-ai-design-tool-2026/", "title": "Google Built a Free Design Tool That Generates Production Code From a Sentence — Then Added Multiplayer", "summary": "How Google acquired Galileo AI, rebranded it as Stitch, powered it with Gemini 2.5 Pro, and shipped multiplayer and a streaming design agent at I/O 2026 — for free.", "content_html": "Google · Performance · 21 May 2026
\nAt Google I/O 2025, Sundar Pichai demoed a tool that turned a plain English description into a complete mobile UI in under 30 seconds. Figma charges $15 per editor per month for collaborative design. Google Stitch does it free. A year later, Google added real-time multiplayer, a streaming design agent, and voice input. The design industry noticed.
\nOn May 20, 2025, Sundar Pichai stood on stage at Google I/O and ran a live demo that made designers and developers simultaneously uncomfortable. He typed a one-sentence description of a mobile app. In under 30 seconds, Google Stitch rendered a complete, multi-component mobile UI with matching color palette, navigation structure, and typography. One button click exported it as React code ready to paste into a development environment. Another exported it as a Figma file with editable layers and auto-layout. The audience applauded. Figma's product team took notes. The tool was free.
\nGoogle Stitch did not emerge from Google's internal R&D labs. It began with the early-2025 acquisition of Galileo AI — a startup that had built one of the first credible text-to-UI generators, capable of interpreting product descriptions and producing coherent interface layouts. Google acquired Galileo, rebranded the technology as Stitch, integrated it with the Gemini 2.5 Pro (Google's most capable multimodal model at the time of Stitch's launch — able to process text, images, audio, and video simultaneously and generate structured outputs across all of them) model family, and launched it as a Google Labs experiment at I/O 2025. The deliberate Labs framing was a signal: Google was testing the market before committing to a full product. The response was immediate. Over 1 million waitlist signups appeared overnight after the live demo.
\n\n\n\nWHAT 'VIBE DESIGN' ACTUALLY MEANS
\nStitch entered the vocabulary alongside 'vibe coding' — the practice of describing software intent to an AI and refining the output iteratively rather than building from first principles. Vibe design applies the same model to interface creation: describe a screen, watch it appear, ask for changes conversationally. The skill shifts from pixel manipulation to intent specification. A founder who cannot use Figma can now produce a working prototype in minutes. A product manager can test five layout variations in the time it would previously have taken to brief a designer on one.
\n
The evolution from launch to I/O 2026 followed a clear product trajectory, compressed by user feedback and Google's resources. The May 2025 version was single-screen only — one prompt, one screen, export. By July 2025, theme customization and Figma export were added based on beta user feedback showing that designers needed design-system integration, not just raw screens. December 2025 brought the Prototypes feature alongside Gemini 3 integration — for the first time, multiple related screens could be linked into interactive flows. March 19, 2026 was Stitch 2.0: infinite canvas, multi-screen generation, voice input, app-flow generation, and 5-screen simultaneous canvas rendering. Ten months of user feedback had transformed a demo into a workspace.
\n\nThe traditional design-to-development pipeline required designers to build components in Figma, annotate specifications manually, and hand off to developers who re-implemented everything in code. Even with design tokens and component libraries, the gap between 'designed' and 'built' consumed weeks of coordination. For small teams and solo founders, this gap was existential: they lacked either the design skill or the engineering skill to complete the loop alone.
\nBy early 2025, Gemini's multimodal capabilities had reached a threshold where they could reliably interpret both text descriptions of interfaces and uploaded images of existing UIs, and generate coherent layouts with appropriate component choices, spacing, and visual hierarchy. The Galileo AI acquisition gave Google a product layer that had already worked out the prompt engineering, training data, and output format questions on top of this capability.
\nStitch accepted three input types simultaneously: natural language descriptions, uploaded reference images or screenshots, and annotated screenshots with modification notes. Gemini 2.5 Pro processed all three modalities together to produce screen designs. Export paths were designed for real developer workflows: Figma files with editable layers and auto-layout, production-ready HTML/CSS, React components, and Vue code.
\nAt I/O 2026 on May 20, 2026, Google launched a streaming design agent that renders UI components onto the canvas in real time as a designer types or speaks — before generation completes, mid-generation course correction is possible. Simultaneous multi-user editing was also added, directly matching Figma's flagship collaboration feature. Both are free. Figma's professional plan charges $15 per editor per month.
\n\n\n\n🎨
\nGoogle Stitch accepts three types of input simultaneously: plain language prompts ('a healthcare app onboarding screen for elderly users'), uploaded images or screenshots ('match this visual style'), and annotated screenshots with modification notes ('make this header bigger and change the blue to green'). Gemini 2.5 Pro processes all three modalities in a single context window, producing a UI design that reflects all constraints at once.
\n
\n\n\nℹ️
\nThe Figma Bridge: Complement, Not Replace
When the original Stitch tool was unveiled at I/O 2025, Sarah Drasner, Director of Engineering at Google, was explicit: the Figma export function was designed to complement rather than replace existing design workflows. Stitch generates a starting point; Figma is where professional designers refine, apply design systems, and collaborate with stakeholders. The paste-to-Figma function exports fully editable layers with auto-layout intact, giving designers a high-fidelity starting point that respects their existing tooling. This positioning is why Stitch attracted both vibe designers (who need no Figma) and professional designers (who use both).
\n
\n\n\n⚠️
\nThe Free Tier's Real Constraints
Google Stitch is free, but not unlimited. The standard free tier provides 350 standard generations + 50 experimental generations per month. For solo founders and students, this is ample. For teams using Stitch for daily rapid prototyping, 350 generations can deplete mid-month. The $20/month Pro tier provides unlimited access. Critically, the March 2026 Stitch 2.0 update — which introduced multi-screen generation and infinite canvas — still required separate generation credit per screen, meaning a 5-screen app prototype consumed 5 credits rather than 1. Users building complex flows discovered this quickly.
\n
The I/O 2026 streaming agent represents the most technically ambitious evolution of Stitch's architecture. Previous versions followed a turn-based model: submit a prompt, wait for generation to complete, review the finished result, submit a revision. The streaming model replaces this with a continuous render: as a designer types or speaks, the agent renders UI components directly onto the canvas in real time, reflowing layouts before generation finishes. The practical difference is the ability to steer mid-generation — if a layout is heading in the wrong direction, a designer can interrupt before it finishes and redirect. Voice input, integrated since March 2026, works within this streaming loop: speech is parsed in real time, and the canvas responds while the designer is still talking.
\n\n\n\n🔌
\nThree Export Paths for Three Audiences
Stitch's export architecture targets three distinct audiences. Figma export (editable layers + auto-layout) for designers who refine in their primary tool. Production code export (HTML/CSS, React, Vue, Tailwind) for developers who need deployable components. AI Studio integration for developers who want to wire backend logic to the UI without leaving the Stitch workflow. Power users chain all three: Figma for design review, code export for development handoff, AI Studio for full-stack experimentation.
\n
\n\n\nℹ️
\nThe Internal Google Labs Strategy
Launching Stitch as a Google Labs experiment rather than a full Google product was a deliberate risk-management decision. Labs experiments carry lower accountability expectations — they can be deprecated without the product embarrassment of killing a flagship tool. They also attract the early-adopter, feedback-rich user base that a new AI product needs: developers, designers, and founders who are comfortable with rough edges in exchange for early access. The Labs label signaled 'come help us build this' rather than 'this is production-ready,' which attracted exactly the right population. By March 2026, 10 months of Labs usage had turned enough of those experiments into validated product decisions to justify the Stitch 2.0 announcement.
\n
\n\n\nTHE VIT CONNECTION IN GOOGLE TRENDS
\nThe same Google Trends report that showed Stitch trending also showed VIT Counselling at +120% — referring to Vellore Institute of Technology admissions season in India. This demographic overlap is instructive: the Indian engineering student population (VIT, IIT, NIT applicants) represents a massive early-adopter cohort for tools like Stitch. Students who cannot afford Figma's professional pricing ($15/seat) but need to prototype apps for hackathons, capstone projects, and startup pitches are an ideal-fit audience for a free AI-native design tool. Stitch's zero-cost model and browser-based access (no installation, no hardware requirements) make it accessible to exactly this demographic.
\n
Google Stitch's core is not a purpose-built design model — it is Gemini 2.5 Pro (Google's multimodal frontier model capable of processing text, images, audio, and code simultaneously — trained on a corpus that includes UI design patterns, code, natural language, and visual references) with a specialized prompt engineering and output parsing layer on top. This architectural choice explains both Stitch's strengths and its limitations. Stitch understands design concepts like 'glassmorphism', 'neumorphic', 'material design', and 'iOS Human Interface Guidelines' because Gemini was trained on documentation and examples of all of them. It can interpret a hand-drawn sketch and understand design intent because Gemini's vision capability has seen thousands of wireframes. And it generates production-quality React code because Gemini understands React at a level that exceeds most specialized code generation models.
\n// Conceptual: How Stitch's streaming agent differs from the original turn-based model\n// The I/O 2026 streaming upgrade is an architectural change, not just a speed improvement\n\n// BEFORE (turn-based): submit → wait → review → resubmit\nasync function generateUI_old(prompt) {\n const result = await stitch.generate(prompt); // blocking — wait for complete output\n // Designer sees nothing until fully done\n // If it's wrong: submit an entirely new prompt from scratch\n return result.screens; // [{ html, css, figmaLayers }]\n}\n\n// AFTER (streaming agent): real-time render + mid-generation steering\nasync function generateUI_streaming(prompt) {\n const stream = stitch.generateStream(prompt);\n \n // Components render onto canvas as they are generated\n stream.on('component', (component) => {\n canvas.renderPartial(component); // visible immediately\n });\n \n // Designer can interrupt and redirect before generation finishes\n stream.on('layoutDecision', (decision) => {\n const userFeedback = canvas.checkInterrupt();\n if (userFeedback) {\n // Mid-generation course correction — no waiting, no re-prompting from scratch\n stream.steer(userFeedback);\n }\n });\n \n // Voice input works inline with the stream:\n // \"make the header larger\" spoken mid-generation → reflected in remaining components\n voiceInput.on('command', (cmd) => stream.steer(cmd));\n \n await stream.complete();\n return canvas.getCurrentState(); // final assembled result\n}\n\n\nTHE GALILEO ACQUISITION RATIONALE
\nGoogle could have built Stitch from scratch using its existing Gemini capabilities. It chose instead to acquire Galileo AI and use their product as the foundation. The rationale is clear in retrospect: Galileo had already solved the hardest non-model problems — the prompt engineering approach that reliably produces coherent UIs, the output parser that converts model outputs into valid design tokens and component trees, the training data pipeline for UI-specific examples, and the user experience model for iterative refinement. Rebuilding these would have taken months. The acquisition compresses that to days. Galileo's technology became the product layer; Gemini became the intelligence underneath it.
\n
\n\n\nℹ️
\nRLHF for UI Quality: Training With Design Feedback
Stitch's code export quality reached 95% accuracy (as measured by component rendering fidelity) in the March 2025 closed beta, up from earlier estimates of around 70%. The improvement was driven by RLHF — Reinforcement Learning from Human Feedback — applied specifically to UI generation quality. The beta involved 500+ partner users including Vercel developers, who provided direct feedback on generated code quality and design accuracy. This domain-specific RLHF signal tuned Gemini's output for the specific quality criteria that professional designers and developers cared about: component naming, layout accuracy, code cleanliness, and design system compatibility.
\n
Google Stitch Evolution: Feature Timeline from Launch to I/O 2026
| Date | Major Update | Key Feature Added |
|---|---|---|
| May 20, 2025 | Google I/O Launch | Single-screen generation, Figma export, code export (HTML/CSS/React) |
| Jul–Aug 2025 | Public beta | Theme customization, RTL language support, bug fixes |
| Dec 2025 | Stitch 2.0 preview | Prototypes (multi-screen flows), Gemini 3 integration |
| Mar 19, 2026 | Stitch 2.0 GA | Infinite canvas, 5-screen canvas, voice input, app-flow generation |
| May 20, 2026 | I/O 2026 update | Streaming agent (real-time canvas render), multiplayer editing — both free |
\n\n\n✅
\nThe Complete Design-to-Production Pipeline
Power users combine all three export paths sequentially to achieve a full design-to-production workflow without opening additional tools: generate in Stitch (AI-driven rapid exploration), paste to Figma for team review and design system application, export code (React/HTML) for developer handoff, push to AI Studio for backend integration. A workflow that previously took weeks of designer-developer coordination can now be completed by a solo founder in hours. The step-change in productivity has been most pronounced for indie developers and small startup teams.
\n
\n\n\n🎙️
\nVoice-First Design: The March 2026 Input Upgrade
The voice input feature added in March 2026 was more than a convenience — it signaled a shift in the interaction model for AI design tools. Text prompts require switching mental context from 'I'm designing' to 'I'm writing instructions.' Voice input keeps the designer in continuous creative flow: say 'give me three different menu styles' while looking at the canvas, watch three variants appear, point and say 'more of that one, but with rounded corners.' The integration of voice into the streaming agent at I/O 2026 completed this loop: voice commands steer the generation in real time, not as post-hoc revision instructions.
\n
Stitch's internal architecture has three distinct layers. The input layer processes multimodal inputs through Gemini 2.5 Pro — text prompts, reference images, and annotated screenshots are all converted into a unified context window that Gemini reasons across simultaneously. The generation layer produces structured design outputs — not raw HTML, but an intermediate representation of design tokens, component hierarchies, and layout constraints that the export layer then converts into format-specific outputs. The export layer translates the intermediate representation into Figma-compatible JSON (with proper component structure and auto-layout), production-grade React/HTML/CSS, and AI Studio integration configs.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE INTERMEDIATE REPRESENTATION: WHY NOT JUST GENERATE CODE DIRECTLY
\nStitch generates an intermediate design representation rather than generating Figma JSON or React code directly. This is a key architectural decision. Direct code generation from a natural language prompt produces valid code but loses design semantics — a developer can see the code but cannot easily edit the visual design. Intermediate representation preserves design intent — component names, spacing relationships, and design tokens — enabling export to multiple formats from one generation while also enabling the Figma integration to produce files with proper component structure, not just static vector exports. The IR is what makes Stitch genuinely useful for professional design workflows rather than just for throwaway prototypes.
\n
\n\n\nℹ️
\nThe Multiplayer Technical Challenge
Adding simultaneous multi-user editing to an AI-native canvas is harder than adding it to a traditional design tool like Figma. In Figma, multiplayer synchronizes object positions, properties, and selection states — deterministic operations with well-understood CRDT (Conflict-free Replicated Data Type — a data structure designed for distributed systems that allows multiple users to edit the same data concurrently without conflicts, automatically merging changes) semantics. In Stitch, two users can simultaneously be prompting the AI to modify the same canvas, producing non-deterministic outputs that may conflict visually. Google's implementation queues concurrent AI generation requests per canvas object and applies a last-write-wins merge for AI-generated changes, while standard CRDT semantics apply for manual edits.
\n
\n\n\n⚠️
\nThe Design Quality Ceiling
Stitch's core limitation remains consistent across all reviews: generated designs are starting points, not finished products. The AI produces layouts with appropriate components and reasonable visual hierarchy, but professional polish — precise spacing, custom illustration integration, brand-specific typography choices, edge-case state design (empty states, error states, loading states) — still requires human design expertise. Stitch is strongest for exploration and prototyping; weakest for production-ready UI that needs to meet professional brand standards. This ceiling is exactly where Figma integration matters: Stitch generates, Figma polishes.
\n
Google Stitch is the most significant challenge to Figma's category dominance since Figma itself displaced Sketch in 2019. The lessons here are as much about product strategy as about engineering.
\nWhat to remember
\n\n\n✅
\nThe Demographic It Actually Disrupts
Stitch's most significant disruption is not to professional designers using Figma full-time — it's to the 90% of product people who needed design work done but couldn't justify hiring a designer for it: founders, product managers, startup engineers, and indie developers. For this population, Stitch doesn't replace Figma — it replaces the decision not to have a designed product at all. The expansion of who can produce designed interfaces is the market story; Figma competition is the noise around it.
\n
\n\n\n✅
\nThe Multiplayer Moment
When Google announced simultaneous multi-user editing at I/O 2026, it was specifically compared to Figma's real-time collaboration — the feature that made Figma the dominant design platform. The comparison was deliberate. Figma's multiplayer is the product feature most closely associated with its enterprise value. Google offering equivalent functionality at zero cost changes the calculus for every design team evaluating their tool budget. Whether Stitch can match Figma's workflow depth is a separate question. Whether it will pressure Figma's pricing is not.
\n
\n\nGoogle built a tool that turns a sentence into a React component in 30 seconds, then made it free, then added multiplayer, which is either a product strategy or a way of saying 'we're very sorry about Google Workspace.'
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-21T00:00:00+00:00", "date_modified": "2026-06-13T19:32:59.524747+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Performance", "Google"]}, {"id": "https://techlogstack.com/explore/google-gemini-omni-2026/", "url": "https://techlogstack.com/explore/google-gemini-omni-2026/", "title": "Google's Gemini Omni Is the First AI That Creates From Anything — Here Is What That Actually Means", "summary": "How Google built Gemini Omni — the first natively multimodal any-to-any model — replacing chained Veo+Imagen+Lyria pipelines with one transformer trained on all moda", "content_html": "Google · Distributed Systems · 21 May 2026
\nFor three years, Google built Gemini to be 'natively multimodal.' At I/O 2026, they finally demonstrated what that phrase means in practice. Gemini Omni takes a photo, an audio clip, a video, and a text description — all at once — and produces a new video that reflects all of them simultaneously. This is not four models chained together. It is one.
\n\nWhen we first announced Gemini, it was our first AI model to be natively multimodal. We knew that training it on a combination of text, code, audio, images, and video would give it a deeper understanding of the world. With world models, AI is moving from predicting text to simulating reality. Gemini Omni is the next step in that direction.
— — Sundar Pichai, CEO of Google — Google I/O 2026, May 19, 2026
The phrase 'natively multimodal' has been in Google's AI vocabulary since Gemini's December 2023 announcement. For two and a half years, it described an aspiration — a model that could process multiple modalities together rather than routing between specialized models. At Google I/O 2026 on May 19, Google delivered the concrete realization of that aspiration: Gemini Omni, a model that accepts text, image, audio, and video as inputs simultaneously and generates video as output — not by chaining Veo (video generation), Imagen (image generation), and Lyria (audio generation) together, but by processing all of them within a single transformer's forward pass. The distinction is architectural, not cosmetic. A chain of models cannot reason about relationships between its inputs. A unified model can.
\nThe path from Gemini's original announcement to Omni runs through three engineering milestones. Gemini 2.0 Flash (late 2024) introduced native audio output and real-time multimodal interaction through the Live API — the first demonstration that Gemini could not just understand audio and video but generate them natively. Project Astra (ongoing research) explored what it means for an AI to have a continuous, persistent understanding of a physical environment through video and audio streams — seeing the world in real time rather than processing discrete inputs. Nano Banana (2025) brought Gemini's intelligence to image generation and editing — restoring old photos, designing from sketches, visualizing ideas from natural language. Omni synthesizes all three threads into a production model: real-time multimodal understanding (Astra) + generative output across modalities (Nano Banana) + unified architecture (Gemini 2.0 Flash's Live API direction).
\n\n\n\nCHAINED MODELS VS NATIVE OMNI: THE FUNDAMENTAL DIFFERENCE
\nOpenAI's Sora (text-to-video) and Google's Veo were both excellent at their specific task but could not natively reason across modalities. A user who wanted to generate a video matching a specific audio track and a reference image had to: (1) generate a video with Veo using the text description, (2) separately process the audio with a music AI, (3) manually synchronize the two. Gemini Omni collapses these three steps into one prompt: upload the image, the audio, and write a description — the model reasons about all three simultaneously and produces a video where the visuals respond to the audio tempo, match the visual reference, and reflect the text description. The unified context window is what enables this.
\n
The previous state-of-the-art for multimodal content creation required chaining specialized models: text-to-video (Veo, Sora), text-to-image (Imagen, DALL-E), text-to-audio (Lyria, ElevenLabs), and manual integration. Each handoff between models lost context — the relationship between audio tempo and visual rhythm, the visual style of a reference image, the emotional tone of a text prompt. Creators managed these integrations manually, which required technical skill that limited access to specialists.
\nThe limitation was architectural. A video model that receives a reference image as a text description (\"generate a video that looks like this photo\") has lost the actual pixel relationships. A video model that receives an audio file as a description (\"generate a video that matches this music\") has lost the actual waveform data. Genuine multimodal reasoning requires all modalities in the same context window — not converted to text summaries of each other.
\nGemini Omni was trained on text, image, audio, and video simultaneously within a single transformer architecture. The model develops internal representations that encode cross-modal relationships — understanding that a warm color palette relates to a particular musical key, that a specific visual style is associated with a cultural context, that physical object behavior in video follows the laws of physics that Gemini has observed across its training data.
\nGemini Omni Flash launched May 19, 2026 in the Gemini app and YouTube Shorts — 10-second clips, with API access planned within weeks. The model accepted any combination of text, image, audio, and video inputs and produced video output with character consistency, physics grounding, and SynthID watermarking. Conversational editing retained full context across turns — a generated scene could be revised through natural language without re-prompting from scratch.
\n\n\n\n🎬
\nGemini Omni Flash's launch was simultaneous in the Gemini app and YouTube Shorts — the latter integration meaning that YouTube's 2+ billion monthly active users could generate AI videos directly within the YouTube Shorts creation flow. The distribution reach of this integration dwarfs any standalone AI video tool's install base on launch day one.
\n
\n\n\nℹ️
\nCharacter Consistency: The Long-Context Advantage
One of the most practically important features of Gemini Omni is character consistency across shots — a character introduced in scene 1 retains their face, clothing, and voice across all subsequent scenes in the same conversation, without the creator re-uploading the reference image for each shot. This is enabled by Gemini's long context window: the model carries the character's visual description as an implicit context throughout the conversation. Competing video models, which have shorter effective contexts, required reference images at every generation turn and still produced inconsistent results. For content creators building multi-scene narratives, this is the feature that makes Omni viable for professional work rather than just single-shot experiments.
\n
The conversational editing model is Omni's most transformative product experience. Previous video generation tools operated like vending machines: insert prompt, receive video, discard and re-insert if wrong. Gemini Omni operates like a continuous creative collaboration: generate a scene, ask for the camera angle to change, ask for the background color to shift, ask for a second character to enter — and the model keeps the context of every previous instruction and modification. The resulting video reflects all decisions made across the conversation, not just the most recent prompt. Creators describe this as the difference between 'generating video' and 'directing a scene.'
\n\n\n\n⚠️
\nWhat Omni Cannot Do Yet
The initial Gemini Omni Flash release has explicit limitations that Google acknowledges. Output is capped at 10 seconds per clip. Image and audio output (generating images or audio files as outputs, not just accepting them as inputs) are on the roadmap but not in the initial release. Complex motion, exact text rendering, and cross-scene consistency for background elements remain challenging. Google's own documentation notes that consistency across edits, complex motion, and precise text in video are 'still challenging.' These are the frontiers where the next model generation will push.
\n
\n\n\nℹ️
\nThe Veo to Omni Transition
Gemini Omni Flash does not deprecate Veo — Google's prior video generation model — immediately. Veo 3 and Veo 3.1 Light remain available for use cases that need pure text-to-video without the multimodal complexity. But Omni is positioned as the future of video generation within the Gemini ecosystem: as Omni's capabilities expand (longer clips, image output, audio output), Veo's separate product line will converge into the Omni family. The Flash suffix — the same naming convention used for Gemini 2.0 Flash — signals that a fuller, more capable Omni Pro version is on the roadmap. Flash is the fast, accessible entry point; Pro will be the quality-ceiling version for professional creators.
\n
\n\n\nTHE NANO BANANA PRECURSOR
\nBefore Omni, Google shipped Nano Banana in 2025 — a product that brought Gemini's intelligence to image generation and editing. Nano Banana could restore old photos, generate images from sketches, and edit photos with natural language commands ('remove the background', 'change the season to winter'). It reached millions of users and established the UX patterns — natural language editing, reference image input, conversational refinement — that Omni extends to video. Omni is, in the product lineage, Nano Banana for video. The audience, the interaction model, and the safety infrastructure were all validated by Nano Banana's image generation rollout before being applied to the harder problem of video.
\n
\n\n\n🌊
\nOmni's videos are described as being grounded in Gemini's real-world knowledge — meaning that objects in generated videos behave according to physical laws the model has internalized from training data, not just based on visual pattern matching to existing videos. A wave breaks on a beach with correct fluid dynamics. A ball falls with correct gravitational arc. A flag moves with correct cloth simulation under wind. This physics grounding is the property that makes Omni's outputs feel more real than the uncanny outputs of earlier text-to-video models where motion was statistically plausible but physically wrong.
\n
Gemini Omni's architecture is a transformer trained across all modalities simultaneously — not a mixture of experts (a neural network architecture where different 'expert' subnetworks specialize in different input types, with a routing mechanism that directs each input to the appropriate expert) architecture with separate video, image, and audio experts, but a single dense model where all modalities interact in every layer. This is the design choice that enables cross-modal reasoning: a visual token and an audio token from the same moment in a video can attend to each other directly within the same attention layer, rather than being processed by separate specialized networks and their outputs later merged. The training corpus includes synchronized video+audio, image+text pairs, audio+text transcriptions, and video+text descriptions at scale — forcing the model to learn the statistical relationships between modalities, not just how to process each in isolation.
\n# Conceptual: Gemini Omni API vs the chained model approach it replaces\n# This illustrates the architectural difference — API details TBC when GA\n\n# OLD APPROACH: Chaining specialized models\n# Each model gets a text description of the other modalities — context is lost\nfrom veo import VeoClient\nfrom lyria import LyriaClient\n\nveo = VeoClient()\naudio_gen = LyriaClient()\n\n# Step 1: Generate audio from description\naudio_clip = audio_gen.generate(\n prompt=\"upbeat electronic music, 10 seconds\"\n) # has no knowledge of the visual reference\n\n# Step 2: Generate video from text — no actual audio waveform input\nvideo = veo.generate(\n prompt=\"city timelapse, upbeat electronic vibe, matches photo style\",\n reference_image=None # can't actually process image input\n) # can't see the audio; can't see the image style\n\n# Manual synchronization: the user's problem now\n\n# GEMINI OMNI: One model, all modalities in one prompt\nimport google.generativeai as genai\n\nmodel = genai.GenerativeModel('gemini-omni-flash')\n\n# All four modalities provided simultaneously — model reasons across all of them\nresponse = model.generate_content([\n \"Create a 10-second timelapse of a city transforming from day to night.\",\n genai.upload_file('reference_photo.jpg'), # actual pixel data — style extracted\n genai.upload_file('audio_track.mp3'), # actual waveform — beat sync possible\n genai.upload_file('reference_clip.mp4') # actual video — motion style extracted\n])\n# Output: video clip that reflects the photo's visual style,\n# syncs transitions to the audio's beat, and uses the reference clip's camera movement\n\n# Conversational editing — context is preserved\nresponse2 = model.generate_content(\n \"Same scene, but make it rain and show the character from my last prompt\"\n # Model remembers: the character, the city style, the audio — all retained\n)\n\n\nSYNTHID: WATERMARKING THAT CANNOT BE REMOVED
\nEvery video generated by Gemini Omni carries Google's SynthID digital watermark — an imperceptible signal embedded in the pixel data itself. Unlike visible watermarks (which are trivially removed by cropping) or metadata watermarks (which disappear on re-encoding), SynthID is embedded into the statistical patterns of the pixels in a way that survives common processing: re-encoding to different codecs, resizing, color grading, and speed adjustments. The watermark allows any tool with the SynthID detector to verify that a video was AI-generated by a Gemini product. As part of the C2PA Content Credentials standard, SynthID watermarked videos can be verified by any C2PA-compatible platform. Digital avatars additionally require mandatory onboarding (recording yourself, speaking verification numbers) before use — a guardrail against deepfakes.
\n
\n\n\nℹ️
\nWorld Models: The Theoretical Foundation
Sundar Pichai described Omni as a step toward world models — AI systems that simulate physical and social reality rather than just predict token sequences. The distinction matters for video generation: a language model predicting video token sequences will produce realistic-looking but physically incorrect motion (objects falling upward, light sources moving inconsistently, human bodies with impossible joint angles). A world model that has internalized physics, causality, and spatial relationships from its training data produces videos where motion is physically coherent because the model understands why objects move the way they do, not just what they look like when they move. Gemini's training corpus — which includes vast quantities of video annotated with physical and contextual descriptions — is what gives Omni's outputs their reported grounding in real-world knowledge.
\n
\n\n\n✅
\nDigital Avatars: The Use Case That Needed Safety First
Gemini Omni includes a digital avatar feature — users record themselves, Google stores a personal avatar, and the avatar can appear in any future Omni generation. The feature is explicitly framed as a response to the deepfake problem: your own likeness, under your own control, with verifiable AI provenance via SynthID. OpenAI had popularized digital avatars in Sora ('Cameos') before Sora's app was deprecated. Google's implementation adds the safety layer — mandatory verification onboarding, SynthID watermarking, and C2PA content credentials — that transforms a deepfake risk into a controlled creative feature.
\n
\n\n\nℹ️
\nThe API Rollout: Weeks After Launch
Gemini Omni Flash launched in the Gemini app and YouTube Shorts on May 19, 2026, with API access described as arriving 'within weeks.' This staggered rollout is standard for Google's AI product launches: consumer surface first (to validate quality and gather real-world usage signal), developer API second (once the team has confidence the model performs as expected across the diversity of real-world use cases). The API will be available through Google AI Studio and Vertex AI, following the same access model as Gemini 2.0 Flash and other Gemini family models.
\n
\n\n\n✅
\nThe YouTube Shorts Pipeline
YouTube Shorts creation flow, as of May 20 2026, includes Omni as a native video generation option accessible directly from the Shorts composer. A creator can generate a base scene, refine it conversationally, and publish directly to Shorts without leaving YouTube's mobile app. The Shorts algorithm already understands Omni-generated content through SynthID — these videos are labeled as AI-generated in discovery surfaces, giving creators transparency credit while maintaining their reach. This is the first time a frontier AI video model has had a direct distribution path to a 2-billion-user platform on launch day.
\n
The architecture of Gemini Omni represents the culmination of the Gemini model family's design philosophy from its first announcement: train a single model on all modalities simultaneously so that cross-modal understanding is emergent from the training process rather than engineered through explicit routing. The practical consequence is that Omni's internal representation of a video frame encodes relationships to audio, text context, and physical physics simultaneously — enabling generation that reflects all input modalities without explicit instructions about how to combine them.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE YOUTUBE SHORTS INTEGRATION: DISTRIBUTION AT SCALE
\nGemini Omni's Day 1 integration into YouTube Shorts is a distribution strategy that no standalone AI video tool can match. YouTube Shorts has 2+ billion monthly logged-in users. The Shorts creation flow integrates Omni as a native generation option — creators can generate a 10-second AI video clip directly within YouTube's creation tools without downloading a separate app or managing an API key. The integration also means that every Omni-generated Short carries YouTube's standard content policy enforcement on top of SynthID watermarking — a two-layer safety system for the most viral content format in the world.
\n
\n\n\n⚠️
\nThe Context Window Limit for Long-Form Video
Gemini's long context window enables character consistency within a conversation, but 10-second clip limits reflect real constraints in generating long-form coherent video. Video generation at 10+ seconds requires planning scene transitions, maintaining narrative coherence, and generating consistent motion physics across hundreds of frames — a computational and quality challenge that current transformer architectures address better over short sequences. The 10-second cap at launch is an engineering constraint, not a product decision, and it will extend as the model and infrastructure mature. The conversational multi-shot workflow is Google's practical solution for longer-form content in the interim: generate shots individually, retain character context across turns, assemble the narrative manually.
\n
\n\n\n🔬
\nC2PA Content Credentials: The Open Standard for AI Provenance
C2PA (Coalition for Content Provenance and Authenticity — an open technical standard co-developed by Adobe, Microsoft, BBC, Intel, Sony, and others that cryptographically signs digital content at the point of creation with metadata about its origin and modification history, enabling any C2PA-compatible tool to verify whether a piece of content is AI-generated, human-made, or modified from either source) integrates with SynthID to give Gemini Omni-generated videos a verifiable provenance chain. Any C2PA-compatible media player or content verification tool can confirm that a video was generated by Gemini Omni, when it was generated, and (if the user consented) by whom. This is the standard that resolves the 'is this real?' question for media at scale — not by restricting AI generation, but by making AI generation verifiable.
\n
Gemini Omni is the first product-grade demonstration of what 'natively multimodal' means in practice. The lessons here are as much about AI architecture philosophy as about the specific product.
\nWhat to remember
\n\n\n✅
\nThe Project Astra Thread
Gemini Omni's launch completes an arc that began with Project Astra — Google's research into a universal AI assistant that processes real-time audio and video streams continuously. Astra demonstrated that Gemini could understand the physical world in real time. Omni demonstrates that it can generate representations of the physical world from any input. The research-to-product pipeline that runs from Astra's prototype glasses through Omni's Flash model is one of the cleaner demonstrations of Google DeepMind's model: do the research under a project name, productize when the model quality meets the distribution threshold.
\n
\n\n\nTHE COMPETITIVE CONTEXT: SORA'S RETREAT
\nOpenAI's Sora launched with enormous fanfare, then had its public-facing app deprecated after relatively brief availability. The gap between Sora's demo-quality output and production-ready video generation proved larger than expected. Google's Omni launch comes with a different posture: explicit acknowledgment of limitations (10-second caps, challenging complex motion), safety infrastructure (SynthID, C2PA, avatar verification) built in from day one, and distribution through existing products rather than a standalone app. The contrast is instructive: building safety and distribution infrastructure before launch is slower but more durable than building capabilities first and retrofitting safety later.
\n
\n\nGoogle spent three years explaining that Gemini was 'natively multimodal' and then at I/O 2026 showed what that actually means by letting you upload a photo, an audio clip, and a text prompt and getting back a video where the sun moves in time with the music — which is either an impressive technical achievement or proof that 'natively multimodal' needed a better marketing team from the start.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-21T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.276622+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Distributed Systems", "Google"]}, {"id": "https://techlogstack.com/explore/openai-chatgpt-kubernetes-outage-2024/", "url": "https://techlogstack.com/explore/openai-chatgpt-kubernetes-outage-2024/", "title": "OpenAI Deployed a Tool to Monitor Kubernetes — and It Took Down All of Kubernetes", "summary": "How OpenAI's December 11, 2024 deployment of a telemetry service overwhelmed the Kubernetes control plane, caused cascading failures across all services, and locked", "content_html": "OpenAI · Reliability · 21 May 2026
\nOn December 11, 2024, OpenAI deployed a new telemetry service designed to improve Kubernetes observability. Within 29 minutes, it had crashed the Kubernetes control plane across every cluster. ChatGPT, the API, and Sora were all unavailable for over four hours. The engineers trying to fix it couldn't run kubectl. The control plane that manages clusters was down — and it was the only way back in.
\nThere is a particular category of failure that strikes harder than a random bug or infrastructure crash: the failure of a system deployed specifically to prevent other failures. On December 11, 2024, OpenAI deployed a new telemetry service designed to improve observability of their Kubernetes control planes — to give engineers better visibility into how their clusters were behaving, to catch problems earlier. Within minutes of deployment, the telemetry service was itself the problem. By 3:16 PM PST, every OpenAI service was degraded or completely unavailable. ChatGPT. The API. Sora. All down. And the engineers responsible for fixing it had a problem that made everything worse: the Kubernetes control plane — the system through which you manage Kubernetes — was itself down, which meant engineers couldn't run kubectl. The tools for recovery depended on the infrastructure that had failed.
\n\nOur tests didn't catch the impact the change was having on the Kubernetes control plane. DNS caching added a delay between making the change and when services started failing. Remediation was very slow because of the locked out effect.
— — OpenAI — December 11, 2024 Incident Postmortem, status.openai.com
The events unfolded with the particular cruelty of incidents where staging does not predict production. On December 10, the telemetry service was deployed to a staging cluster and verified as working correctly. On December 11 at 2:23 PM, the code was merged and the deployment pipeline triggered. From 2:51 PM to 3:20 PM, the change was applied to all production clusters. At 3:16 PM — five minutes before the rollout was even complete — all OpenAI products began degrading. The root cause was a configuration in the new service that caused every node in every cluster to execute resource-intensive Kubernetes API operations simultaneously. The cost of these operations scaled with the size of the cluster — which meant the largest clusters, which also happened to be the most critical, were hit hardest and fastest.
\n\n\n\nDNS CACHING: THE HIDDEN TIME BOMB
\nThe staging environment passed because of a combination of two factors. First, the staging cluster was small — the telemetry service's API load scaled with cluster size, so a small staging cluster generated manageable load. Second, and more insidiously: DNS caching masked the failure. When the telemetry service started overwhelming the Kubernetes API servers, services that had already cached DNS responses continued functioning temporarily — they could still reach their dependencies through stale cache entries. This created a delay between the moment the change was applied and the moment services began failing. Engineers saw a clean deployment, saw services continuing to function, and assumed success — until the DNS cache expired and services that hadn't failed yet began failing all at once.
\n
On December 11, 2024 at 2:51 PM PST, the new telemetry service configuration began rolling out to all Kubernetes clusters. The rollout completed at approximately 3:20 PM. The service's configuration caused every node in every cluster to issue simultaneous resource-intensive Kubernetes API calls — a load that scaled with cluster size, hitting the largest, most critical clusters hardest.
\nWith thousands of nodes simultaneously hammering the Kubernetes API servers, the control planes of most large clusters crashed. Kubernetes's control plane manages service discovery and DNS resolution — when it failed, services could no longer find each other. DNS cache expiry then propagated the failure to services that had been temporarily protected by stale cache entries, turning a partial degradation into a complete cascading failure.
\nRecovery required rolling back the telemetry configuration — but rolling back Kubernetes configurations requires kubectl, which requires a functioning Kubernetes control plane. The control plane was down. Engineers were effectively locked out of the clusters they needed to fix. Recovery required out-of-band mechanisms: directly accessing nodes through cloud provider management consoles, bypassing the Kubernetes layer entirely to remove the telemetry service's configuration.
\nChatGPT reached substantial recovery at 5:45 PM PST. Full recovery across all services was achieved at 7:38 PM PST — 4 hours and 22 minutes after the incident began. OpenAI published a detailed postmortem within days, identifying four root causes and committing to specific architectural changes including break-glass emergency access mechanisms and staged rollouts for all infrastructure changes.
\n\n\n\n📱
\nApple released iOS 18.2 on December 11, 2024 — the same day as the ChatGPT outage. iOS 18.2 introduced ChatGPT integration into Apple Intelligence. The timing was spectacularly bad: millions of iPhone users who had just updated their OS to get ChatGPT access discovered ChatGPT was down. Many initially assumed the iOS update had caused the outage. OpenAI's postmortem explicitly confirmed it had not: the iOS 18.2 launch was coincidental. The real cause had nothing to do with the traffic spike from Apple users.
\n
\n\n\n⚠️
\nThe Circular Dependency That Made Recovery Hard
The deepest structural problem revealed by the December 11 outage was a circular dependency between the Kubernetes control plane and the services that depend on it. When the control plane failed, it took down: (1) DNS resolution for all services, (2) service discovery across the cluster, (3) the ability to schedule new pods or reschedule crashed ones, and (4) the primary mechanism engineers use to manage all of the above. Recovery from a Kubernetes control plane failure required access to a system that the control plane failure had disabled. This is the engineering equivalent of locking your keys inside your car — and the standard response (calling a locksmith) had not been pre-arranged.
\n
The 'locked out effect' that OpenAI's postmortem names is a well-known failure mode in Kubernetes operations, though it often appears in less severe forms. Kubernetes is a complex distributed system where the control plane manages the state of the cluster, and the data plane (the nodes) depends on that state to function. But the management tools (kubectl, Helm, the Kubernetes API) also depend on the control plane. When the control plane goes down, the cluster enters a state where it continues running existing workloads on warm nodes (the data plane doesn't immediately die) but nothing can be changed, fixed, scaled, or recovered through standard channels. The cluster is frozen — and thawing it requires direct node access bypassing Kubernetes's own abstractions.
\n\n\n\n🔬
\nThe Telemetry-Ironically-Causes-Outage Pattern
The December 11 outage belongs to a specific failure pattern category that has appeared at multiple major companies: the observability tool that causes the outage it's meant to detect. A new metrics agent deployed across a large fleet issues unexpected API calls. A distributed tracing system generates load spikes while capturing other services' load spikes. A log aggregation service fills up disk on the servers it monitors. The pattern is instructive: observability infrastructure touches every service in the fleet and therefore has a potential blast radius of everything. It requires the same staged rollout rigor as any production service deployment.
\n
\n\n\n❌
\nThe Scale of Impact
OpenAI's services reach hundreds of millions of users. On December 11, 2024, every one of them hit the same wall: ChatGPT is unavailable. Beyond consumer users, the ChatGPT API powers thousands of production applications — startups that had built their products on top of OpenAI's API, enterprises that had integrated ChatGPT into customer support flows, developers whose production services were serving errors to their own users. A single infrastructure configuration error at OpenAI propagated into cascading failures across the entire ecosystem of businesses built on its infrastructure. This is the multiplier effect of platform outages.
\n
\n\n\nTHE SAME DAY AS SORA'S DEBUT
\nThe December 11 outage came on the same day that OpenAI was also managing the pressure of Sora's recent launch — its video generation platform, which had seen immediate scaling challenges upon release. The Sora platform was itself affected by the December 11 outage (listed in the postmortem alongside ChatGPT and the API as impacted products). This confluence made December 11 OpenAI's most visible reliability day: its most-watched new product and its most-used existing product, both down simultaneously. The postmortem was unusually forthcoming about the operational context — acknowledging explicitly that the organization was managing multiple scaling challenges at once.
\n
\n\n\n⏰
\nThe most counterintuitive fact in the December 11 timeline: services began degrading at 3:16 PM, but the rollout had only started at 2:51 PM and wasn't complete until 3:20 PM. Services started failing while the rollout was still in progress. This is the DNS cache masking effect in action: the earliest-affected clusters (the large ones, which received the change first) started degrading immediately; clusters that received the change later showed the failure slightly later. From the engineers' monitoring dashboards, it looked like a gradual degradation — masking the true cause until DNS caches expired everywhere and the pattern became undeniable.
\n
Understanding the December 11 recovery timeline requires understanding the specific Kubernetes failure mode. The telemetry service's configuration caused each node to watch Kubernetes API resources continuously — a watch operation that made API calls proportional to the number of resources in the cluster. Across thousands of nodes in large clusters, these API calls compounded into an overwhelming flood. The Kubernetes API servers — the stateful components of the control plane that maintain cluster state — became saturated. With the API servers unresponsive, etcd (the distributed key-value store that backs Kubernetes' state — all cluster state (node metadata, pod specifications, service definitions) lives in etcd, and the API servers cannot function without access to it) became unreachable. Without etcd, the API servers couldn't recover. Without the API servers, nothing could be changed. The cluster was in a deadlock.
\n# Simplified model of the failure mode: telemetry service overwhelming K8s API\n# Each node watches K8s API objects — cost scales with cluster size\n\n# The telemetry service configuration (simplified)\nTELEMETRY_CONFIG = {\n \"watch_all_pods\": True, # Watch all pod events in the cluster\n \"watch_all_nodes\": True, # Watch all node events\n \"watch_all_services\": True, # Watch all service definitions\n \"poll_interval_ms\": 100, # Check for changes every 100ms (aggressive)\n}\n\n# What happens when this runs on N nodes simultaneously:\ndef nodes_making_api_calls(cluster_size: int) -> int:\n # Each node creates 3 watchers, each calling K8s API every 100ms\n return cluster_size * 3 * (1000 / 100) # calls per second\n\n# Small staging cluster (100 nodes):\nstaging_load = nodes_making_api_calls(100) # 3,000 API calls/sec — manageable\n\n# Large production clusters (thousands of nodes):\nprod_load = nodes_making_api_calls(5000) # 150,000 API calls/sec — CATASTROPHIC\n# K8s API server limit: typically ~1,000-2,000 requests/sec\n# At 150,000: API server becomes unresponsive within seconds\n# DNS resolution breaks: services can't find each other\n# kubectl stops working: engineers can't recover\n\n# The fix: remove the watch configuration from the telemetry service\n# But to apply a config change, you need kubectl\n# kubectl requires a working API server\n# The API server is down because of the config\n# ↑ The locked-out effect ↑\n\n# Recovery path: bypass Kubernetes entirely\n# SSH directly to nodes via cloud provider console\n# Manually stop the telemetry service process on each node\n# API server load drops\n# Control plane recovers\n# kubectl works again\n# Verify and clean up\n\n\nTHE FOUR ROOT CAUSES FROM OPENAI'S POSTMORTEM
\nOpenAI's postmortem identified four specific contributing factors: (1) The staging cluster was too small to reproduce the load scaling behavior — the failure only manifested at production cluster sizes. (2) DNS caching masked the initial failure — services continued functioning on stale cache entries, giving engineers a false signal that the deployment was clean before cache expiry revealed the truth. (3) No canary deployment — the configuration was applied to all clusters simultaneously rather than validated incrementally on one cluster first. (4) No break-glass mechanism — there was no pre-arranged out-of-band access path for exactly this class of failure where the standard Kubernetes management plane was unavailable.
\n
\n\n\nℹ️
\nThe Recovery Steps
OpenAI's engineers recovered the cluster through a sequence that bypassed Kubernetes abstractions entirely: Step 1 — access individual nodes directly through the cloud provider's management console (not through Kubernetes), bypassing the downed control plane. Step 2 — manually stop the telemetry service process on each node to eliminate the API call flood. Step 3 — with load removed, Kubernetes API servers began recovering. Step 4 — once kubectl was functional, roll back the telemetry service configuration through standard channels. Step 5 — monitor service recovery and DNS propagation across the fleet. Each step added latency because it required manual execution across thousands of nodes.
\n
\n\n\n✅
\nPost-Incident Actions: Four Engineering Commitments
OpenAI's postmortem committed to four concrete engineering changes: (1) Immediate: locked the telemetry configuration to prevent re-deployment. (2) Short-term: implement break-glass emergency access mechanisms that function even when the Kubernetes control plane is unavailable. (3) Medium-term: decouple observability infrastructure from the components it monitors, so a failing telemetry system cannot cascade into the monitored services. (4) Long-term: all infrastructure-related configuration changes will use staged deployment with continuous monitoring and the ability to halt at any percentage. The staged rollout commitment was the same lesson Cloudflare had learned twice.
\n
\n\n\n⚠️
\nThe Kubernetes Watch API Amplification
The specific mechanism was a Kubernetes Watch API (a Kubernetes API feature that allows clients to receive a stream of events as resources change — an efficient alternative to polling, but one that creates a persistent connection from the watching client to the API server, consuming API server resources proportional to the number of watchers) misuse. Rather than polling for cluster state on a schedule, watch operations create a persistent long-lived connection from each watcher to the API server. The telemetry service created three watch connections per node — at 5,000 nodes in a large cluster, that's 15,000 persistent watch connections. Each watch connection requires API server resources to maintain. The API server, designed for a few hundred concurrent operations, was maintaining thousands — and also handling the event stream updates that each watch triggered as cluster state changed.
\n
\n\n\n✅
\nThe Immediate Locking Action
Within hours of the outage's resolution, OpenAI took one immediate action while longer-term architectural work was planned: locked the telemetry configuration so it could not be re-deployed in its original form without an explicit manual override. This lock-before-investigation pattern is a standard SRE practice: after a configuration causes a production incident, prevent it from being accidentally reapplied during the postmortem or by a team member who doesn't yet know about the incident. The lock is a cheap, immediate mitigation that buys time for proper architectural fixes. It is the equivalent of removing a circuit breaker from service rather than leaving it in a state where it could trip again.
\n
OpenAI's Kubernetes architecture runs the inference clusters that power ChatGPT's model serving, the API gateway that handles developer requests, and the Sora video generation pipeline. All of these depend on Kubernetes's control plane for service discovery, DNS resolution, pod scheduling, and configuration management. Understanding how a single telemetry service configuration could take all of them down requires understanding both the structure of Kubernetes and the specific amplification mechanism the December 11 configuration triggered.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nWHY KUBERNETES CONTROL PLANE FAILURE IS CATASTROPHIC
\nThe Kubernetes control plane manages three things that are catastrophic to lose simultaneously: DNS resolution (services find each other by name, not IP — without DNS, microservices go blind), service discovery (load balancers can't route to healthy pods without the API server updating their configuration), and pod scheduling (crashed pods can't be restarted, replicas can't be scaled). In most partial failures, you lose one of these. A control plane failure loses all three. And because the standard recovery path requires the control plane to function, recovery from total control plane failure requires out-of-band mechanisms that most teams haven't pre-arranged.
\n
\n\n\n⚠️
\nThe Staging Trap: Size-Dependent Bugs
The December 11 outage is a textbook example of a size-dependent bug — a failure that only manifests at production scale. The telemetry service worked correctly in staging because staging clusters were small enough that the aggregate API call load from all nodes was within the API server's capacity. Every small-scale test passed cleanly. At production scale, with thousands of nodes instead of dozens, the same configuration produced 100× the load — enough to overwhelm even a properly sized API server. Size-dependent bugs require load testing at production scale, not just functional testing at representative scale. The standard 'test in staging' process is insufficient for infrastructure changes whose failure modes are non-linear functions of cluster size.
\n
\n\n\nℹ️
\nThe Kubernetes Control Plane Architecture
Kubernetes control plane (the set of components that manage the overall state of a Kubernetes cluster — including the API server (handles all REST operations), etcd (distributed key-value store backing all cluster state), the scheduler (assigns pods to nodes), and the controller manager (runs reconciliation loops)) is itself a distributed system running on dedicated master nodes. In OpenAI's architecture, the control plane runs on separate infrastructure from the data plane nodes that run model inference. When the control plane fails, data plane nodes continue running their existing workloads (model inference pods don't immediately die) but cannot be managed — pods can't be restarted, scaled, or reconfigured. Services that depended on service discovery (which uses control-plane-managed DNS) began failing immediately. Services with static configuration or warm DNS caches survived longer before failing.
\n
The December 11 ChatGPT outage is among the most instructive Kubernetes incidents ever publicly documented — partly because OpenAI published a detailed postmortem, and partly because the failure pattern recurs across the industry whenever teams deploy infrastructure changes without accounting for scale-dependent behavior.
\nWhat to remember
\n\n\nℹ️
\nThe iOS 18.2 Coincidence
Apple shipped iOS 18.2 — which introduced ChatGPT integration into Apple Intelligence — on the same day as the outage. Millions of users who updated and then tried ChatGPT saw it was unavailable. Social media immediately speculated that the iOS update had caused the outage. OpenAI's postmortem was explicit: iOS 18.2 had nothing to do with the outage. The telemetry service failure had already begun degrading the infrastructure before the iOS update's traffic could have any effect. The coincidence is a useful reminder that correlation — especially coincidence of timing — is not causation, and that attributing outage causes to the most visible concurrent event is a common and often wrong instinct.
\n
\n\n\nTHE STAGED ROLLOUT THAT WOULD HAVE CAUGHT IT
\nA staged rollout with the pattern: 1 cluster → verify 30 minutes → 10% of clusters → verify 30 minutes → 50% → verify → 100%, would have caught this failure at the 1-cluster stage. One large cluster showing API server saturation is a signal. One large cluster crashing before engineers even understood why is an outage. The difference between the two outcomes is the presence of a verification window between deployment stages — time where the system's behavior can be observed before the next deployment stage commits. OpenAI's December 11 deployment had no such window: the configuration was applied to all clusters in 29 minutes without a verification pause.
\n
\n\nOpenAI deployed a service to watch their Kubernetes clusters more carefully, and the service watched so carefully it broke every cluster simultaneously — which is either ironic or a very expensive way to learn that observability infrastructure has a blast radius.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-21T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.286784+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "OpenAI"]}, {"id": "https://techlogstack.com/explore/linkedin-kafka-origin-2011/", "url": "https://techlogstack.com/explore/linkedin-kafka-origin-2011/", "title": "LinkedIn Needed a Message Queue. They Built the One the Entire Internet Runs On.", "summary": "How LinkedIn engineers Jay Kreps, Jun Rao, and Neha Narkhede built Apache Kafka in 2010 — the append-only log that went from 1 billion to 7 trillion messages per day", "content_html": "LinkedIn · Messaging · 19 May 2026
\nIn 2010, LinkedIn was drowning in data it couldn't move. Every ML model, every recommendation engine, every real-time feature was starving because there was no reliable way to get activity data from the website into the systems that needed it. Jay Kreps, Jun Rao, and Neha Narkhede spent a year building a fix. They named it after Franz Kafka. The rest of the internet adopted it.
\nIn 2010, LinkedIn was a growing professional network with a problem that every ambitious data-driven company eventually hits: a massive accumulation of valuable activity data that was effectively locked inside the systems that generated it. Every page view, every job click, every connection request, every search query was data. LinkedIn's ML engineers wanted that data to train recommendation models. LinkedIn's analytics engineers wanted it to understand user behavior. LinkedIn's search engineers needed it to keep the index fresh within seconds of updates. But the pipelines connecting these systems to their data sources were a fragile, inconsistent web of point-to-point integrations — each one custom-built, each one brittle, none of them sharing any infrastructure. Jay Kreps, who was leading data infrastructure engineering at LinkedIn, later described the root cause with characteristic directness: \"Everyone wanted to build fancy machine-learning algorithms, but without the data, the algorithms were useless. Getting the data from source systems and reliably moving it around was very difficult.\"
\n\n\n\n📊
\nBefore Kafka, LinkedIn's data architecture had an N×M integration problem: every data source needed a custom pipeline to every data destination. With dozens of source systems and dozens of consumer systems, engineers were maintaining hundreds of individual pipelines — each with its own error handling, schema management, and operational burden. Adding one new data source meant writing N new pipelines. Adding one new consumer meant updating M existing sources.
\n
Kreps, alongside Jun Rao (who had joined from IBM's database group) and Neha Narkhede (who had come from Oracle), evaluated every existing solution. Traditional message queues — ActiveMQ (an open-source message broker implementing the JMS specification, designed for reliable, ordered message delivery between enterprise applications), RabbitMQ (a message broker built around the AMQP protocol, designed for flexible routing, delivery guarantees, and complex messaging patterns) — were designed for a different problem. They offered rich delivery guarantees, complex routing, and transaction semantics, but they were built for the reliable delivery of individual task messages, not for the high-throughput streaming of millions of activity events. The broker in these systems tracked the delivery state of every message — consuming memory and CPU proportional to the number of outstanding messages. They were designed for near-immediate consumption. They could not handle the situation where a Hadoop job needed to replay yesterday's activity data. They could not scale to the throughput LinkedIn needed. Most critically: their per-message overhead was enormous. ActiveMQ's message format had 144 bytes of overhead per message. LinkedIn needed to process millions of messages per second.
\n\n\n\nTHE INSIGHT: TREAT DATA MOVEMENT LIKE A LOG
\nThe founding insight of Kafka was recognizing that LinkedIn's data movement problem was not a messaging problem — it was a log problem. Databases have used append-only logs for decades: the write-ahead log (a sequential record of all changes made to a database, written before the changes are applied — used for crash recovery, replication, and point-in-time restoration) is how MySQL, Postgres, and every serious database achieves durability and replication. Jay Kreps asked: what if the data pipeline itself was an append-only log? Producers append events. Consumers read them at their own pace. The log retains messages for a configured period. Any consumer can replay from any point. The broker doesn't track state. The simplicity unlocked everything.
\n
By 2010, LinkedIn had dozens of data source systems and dozens of consumer systems — ML models, analytics pipelines, search indexers, real-time features — all of which needed the same activity stream data. Point-to-point custom pipelines were the solution, but maintaining hundreds of them was unsustainable. The existing messaging systems (ActiveMQ, RabbitMQ) couldn't handle LinkedIn's throughput requirements and were designed for task queues, not event streams.
\nThe problem in 2010 had two halves: batch systems (Hadoop) could handle large volumes but only hours later; traditional message queues could deliver in real-time but couldn't scale to LinkedIn's volume or support replay. There was no system that provided high throughput, low latency, durability, replayability, and horizontal scalability simultaneously. The three engineers concluded that the tool they needed did not exist.
\nKreps, Rao, and Narkhede spent approximately one year building the first version of Kafka. The core architectural decision was treating the message store as an append-only log rather than a queue. This single choice enabled sequential disk I/O (orders of magnitude faster than random I/O), stateless brokers (consumers track their own position), arbitrary replay (consumers read from any offset), and horizontal partitioning (each partition is an independent log that scales independently).
\nKafka went into production at LinkedIn in 2011 and immediately became the backbone of the company's real-time infrastructure. At launch it was ingesting over 1 billion events per day. LinkedIn open-sourced it in early 2011. It became an Apache Top-Level Project in October 2012. By 2015 it was processing 1 trillion messages per day. By 2019, 7 trillion. Kreps, Narkhede, and Rao left LinkedIn in November 2014 to found Confluent, building the commercial ecosystem around Kafka.
\n\nI thought that since Kafka was a system optimized for writing, using a writer's name would make sense. I had taken a lot of lit classes in college and liked Franz Kafka. Plus the name sounded cool for an open source project.
— — Jay Kreps — on naming Kafka after Franz Kafka, via Quora
The name was chosen when the project was being prepared for open-sourcing. Jay Kreps was inspired by Franz Kafka (the German-language novelist (1883–1924) known for works including The Metamorphosis, The Trial, and The Castle — exploring themes of alienation, bureaucracy, and transformation) — a writer whose work Kreps admired and whose name, he felt, suited a system built for writing. The practical truth is that the naming was an afterthought. The engineering came first. In the original academic paper published at the NetDB workshop in June 2011, the system is described without literary flourish: a distributed messaging system for log processing, designed for high throughput, low latency, and horizontal scalability. The paper's benchmarks were direct: Kafka produced messages at a rate that was orders of magnitude faster than ActiveMQ or RabbitMQ. The numbers were not close.
\n\n\n\nℹ️
\nWhy LinkedIn's Data Architecture Needed Real-Time
LinkedIn's core value proposition — showing you the right jobs, the right connections, the right content — required real-time signals. If you search for \"software engineers in San Francisco\" and connect with three of them, LinkedIn's recommendations should update within seconds to reflect what your new connections know and who they know. With Hadoop batch jobs, this update happened hours later. With Kafka feeding real-time stream processing, updates became searchable within seconds of being posted. The latency reduction from hours to seconds was not a technical nicety — it was the product feature that made LinkedIn's social graph feel alive.
\n
\n\n\n⚠️
\nWhat Existing Systems Got Wrong at Scale
The original Kafka paper published the benchmark numbers without ceremony. LinkedIn configured a single producer to publish 10 million messages of 200 bytes each. Kafka with batch size 50: ~50MB/sec. ActiveMQ: ~2MB/sec. RabbitMQ: slightly better than ActiveMQ but far below Kafka. The gap was not 10% or even 2x — it was an order of magnitude. The performance difference traced directly to Kafka's design: sequential disk writes, zero per-message broker state, batched I/O, and a message format with 9 bytes of overhead versus ActiveMQ's 144 bytes.
\n
Jay Kreps wrote one of the most-cited engineering essays of the last decade: \"The Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction\" (LinkedIn Engineering, 2013). The essay argued that the append-only log was not just a Kafka implementation detail — it was a universal primitive for distributed systems. Databases use it for replication. Kafka uses it for messaging. Stream processors use it for state. The essay made the case that any system that needs to integrate data across multiple consumers should be built on a log, not on point-to-point integrations. At the time Kreps wrote the essay, LinkedIn was running over 60 billion unique message writes through Kafka per day — several hundred billion counting cross-datacenter replication. The argument was not theoretical.
\n\n\n\nℹ️
\nLinkedIn's Real-Time Feature Hunger
LinkedIn's product ambitions in 2010 were fundamentally real-time: who viewed your profile in the last 24 hours? Which of your connections just updated their job title? When a recruiter posts a job matching your skills, how fast does it appear in your feed? These features required that activity data flowing from the website into the recommendation and notification systems be fresh — not hours old. The batch pipeline to Hadoop was adequate for weekly model training but useless for features that needed sub-minute freshness. Kafka was not just a performance improvement over existing infrastructure; it was the prerequisite for an entire class of real-time product features that didn't yet exist.
\n
\n\n\n1 BILLION EVENTS PER DAY — IMMEDIATELY
\nWhen Kafka went into production at LinkedIn in 2011, it was immediately processing more than 1 billion events per day. This was not a gradual ramp — the scale was there from day one because LinkedIn's existing activity volume was already at that level; Kafka simply replaced the fragile point-to-point pipelines that had been handling it. The immediate billion-event scale validated the architecture under real production conditions within weeks of launch. It also meant the open-source release in mid-2011 came with a credibility that mattered: this was not a research prototype. It was a system already running at significant scale.
\n
Kafka's performance advantage over existing systems was not the result of clever optimization of a standard architecture. It was the result of choosing a fundamentally different architecture, where every key design decision reinforced the same goal: maximize throughput for streaming event data. Five decisions stand out as architecturally defining — and each was a deliberate rejection of how existing messaging systems had been built.
\n// The five key Kafka design decisions in code form\n\n// DECISION 1: Append-only log storage (not a queue)\n// Each partition is a directory of segment files, appended to sequentially\n// /kafka-logs/my-topic-0/00000000000000000000.log\n// /kafka-logs/my-topic-0/00000000000000100000.log (new segment at 100k messages)\n// → Sequential writes: disk seeks are expensive; sequential I/O is 100x faster\n\n// DECISION 2: Consumer tracks its own offset\n// The broker doesn't care what consumers have read — it just serves bytes\nlong consumerOffset = consumer.position(topicPartition); // consumer owns this\n// → Brokers are stateless: no per-consumer memory, no ack tracking overhead\n// → Consumers can replay any time: just reset the offset\nconsumer.seek(topicPartition, 0); // replay from the beginning\n\n// DECISION 3: Topics are partitioned for horizontal scale\n// Each partition is an independent log — producers and consumers parallelise\nProducerRecord record =\n new ProducerRecord<>(\"user-activity\",\n userId, // partition key: same user → same partition = ordered\n eventJson // the message\n );\n// → Topic with N partitions can be consumed by N consumers in parallel\n// → Add brokers, add partitions: linear throughput scaling\n\n// DECISION 4: Batch I/O from client to broker\nprops.put(\"batch.size\", 16384); // batch up to 16KB before sending\nprops.put(\"linger.ms\", 5); // or wait up to 5ms for the batch\n// The original paper: batch size 50 improved throughput by ~10x vs batch size 1\n\n// DECISION 5: Zero-copy transfer using sendfile()\n// When a consumer fetches data, Kafka uses OS sendfile() syscall:\n// data goes directly disk → network socket, bypassing userspace entirely\n// → No data copy into JVM heap → no GC pressure → consistent low latency\n// This is why Kafka can deliver data nearly as fast as the network allows \n\n\nTHE STATELESS BROKER: THE COUNTERINTUITIVE MASTERSTROKE
\nThe most counterintuitive decision in Kafka's design is making the broker stateless — the broker doesn't track which consumers have read which messages. In ActiveMQ and RabbitMQ, the broker maintains delivery state for every message: who acknowledged it, who hasn't, what needs to be retried. At scale, this per-message state tracking consumes enormous memory and creates a bottleneck. Kafka's solution was radical: let consumers track their own position (their offset in each partition). The broker just stores bytes in a log. Consumers read at their own pace, commit their offset to Zookeeper (later to a Kafka topic itself), and can reset to any offset to replay. The broker's memory footprint is constant regardless of consumer count or message backlog.
\n
Kafka vs Traditional Message Queues: Architectural Comparison (2011 original benchmarks and design properties)
| Property | ActiveMQ / RabbitMQ | Kafka |
|---|---|---|
| Storage model | Queue (messages deleted after ack) | Append-only log (messages retained by time/size) |
| Broker state | Tracks ack state per message per consumer | Stateless — consumers track own offset |
| Producer throughput (bench) | ~2 MB/sec (ActiveMQ) | ~50 MB/sec (batch size 50) |
| Message overhead | 144 bytes (ActiveMQ, JMS header) | 9 bytes |
| Consumer replay | Not supported (message gone after ack) | Supported — seek to any offset |
| Horizontal scale | Limited (complex cluster configs) | Native — add partitions, add consumers |
| Use case fit | Task queues, guaranteed delivery, routing | Event streaming, log aggregation, activity tracking |
\n\n\n✅
\nWhat LinkedIn Actually Used Kafka For
By 2019, Kafka was the circulatory system of LinkedIn's infrastructure. Activity tracking (the original use case): every page view, search, ad impression fed to both Hadoop and real-time processors. Real-time search indexing: profile updates searchable within seconds. Database replication: Espresso CDC via Kafka replaced MySQL replication. Inter-service messaging: microservices decoupled through Kafka topics. Stream processing: Apache Samza (LinkedIn's open-source stream processor) used Kafka as both input and durable state store. Every part of LinkedIn's data plane ran on Kafka.
\n
\n\n\nℹ️
\nZero-Copy: The OS Kernel Trick That Doubled Throughput
One of Kafka's most impactful performance optimizations is invisible to application code: zero-copy data transfer using the OS-level
\nsendfile()syscall. In a traditional data transfer, data moves from disk to kernel buffer, kernel buffer to userspace, userspace to socket buffer, socket buffer to network. In Kafka's consumer path, the OSsendfile()call transfers data directly from the page cache (disk buffer) to the network socket, bypassing userspace entirely. This means no data is copied into the JVM heap — no GC pressure, no object allocation overhead. At LinkedIn's throughput rates, this optimization alone is responsible for significant throughput gains and, more importantly, for Kafka's consistent low latency even under high load.
\n\n\n✅
\nThe Open-Source Flywheel
LinkedIn open-sourced Kafka in early 2011 — before it was even an Apache project. The decision to share the core infrastructure was not philanthropic; it was strategic. LinkedIn's engineers knew that the data pipeline problem they had solved was universal. By open-sourcing Kafka, they attracted contributions from engineers at Netflix, Uber, Twitter, and hundreds of other companies — all of whom had the same problem. The community built tooling LinkedIn would never have had resources to build alone: Kafka Streams, Kafka Connect, ksqlDB, MirrorMaker, Schema Registry. The open-source flywheel turned a LinkedIn internal tool into the internet's standard real-time data infrastructure.
\n
Kafka's architecture has three layers. The storage layer is a set of partitioned, replicated append-only log files on disk — each partition is an independent, totally ordered sequence of records. The broker layer is a cluster of server processes that manage partition assignment, replication, and client connections — but hold no consumer state. The client layer is producers writing to partitions and consumer groups reading from them, each consumer group maintaining its own independent offset position per partition. Understanding why this architecture outperforms traditional queues requires visualizing the data flow.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE LOG/TABLE DUALITY: JAY KREPS' DEEPER INSIGHT
\nIn his 2013 essay \"The Log,\" Kreps articulated a concept that went beyond Kafka's implementation: the log/table duality (a mathematical relationship where any database table can be derived by replaying a log of changes from the beginning, and any log can be materialized into a table by applying each event as a state update — they are two views of the same underlying truth). Every database table can be derived by replaying the change log from the beginning. Every stream of events can be materialized into a table by accumulating state. This duality means a Kafka topic is simultaneously a stream and a database — you can query it as a stream in motion (stream processing) or materialize it as a snapshot (a table). This insight later became the foundation for Kafka Streams, ksqlDB, and the entire stream-processing ecosystem.
\n
\n\n\nℹ️
\nLinkedIn's Kafka by 2019: The Numbers
By 2019, LinkedIn's Kafka deployment had become one of the largest publicly documented distributed systems in existence: 7 trillion messages per day, spread across 100+ clusters, 4,000+ brokers, 100,000+ topics, and 7 million partitions. Each message was consumed by approximately four consumer groups on average. The cross-datacenter replication system (Brooklin) was itself mirroring over 7 trillion messages per day. From 1 billion events per day at launch in 2011 to 7 trillion by 2019: a 7,000x growth in eight years on the same fundamental architecture.
\n
Kafka is fifteen years old and it powers a majority of the world's real-time data infrastructure. Its success is not luck — it emerged directly from the architectural decisions Jay Kreps, Jun Rao, and Neha Narkhede made in 2010. The lessons here are about identifying the right abstraction, challenging assumptions baked into existing tools, and the compounding returns of open-sourcing infrastructure.
\nWhat to remember
\n\n\n✅
\nFrom 1 Billion to 7 Trillion: The Same Architecture
The most remarkable fact about Kafka's growth is that the core architecture described in the 2011 paper — append-only partitioned log, stateless brokers, consumer-side offsets — is still the architecture running at 7 trillion messages per day in 2019. The system scaled 7,000x on the same fundamental design. Operational complexity grew (Cruise Control for rebalancing, Burrow for consumer lag monitoring, Brooklin for cross-datacenter replication), but the append-only log at the center of it all never needed to be replaced. Good architecture ages well.
\n
\n\n\nTHE PLATFORM THAT MADE KAFKA A COMPANY
\nIn November 2014, three years after Kafka's public launch, Jay Kreps, Neha Narkhede, and Jun Rao left LinkedIn to found Confluent — a company built to provide enterprise Kafka services, managed Kafka infrastructure, and the commercial ecosystem around the open-source project. Confluent went public in 2021 at a $4.5 billion valuation. The path from LinkedIn internal tool to billion-dollar company in thirteen years is one of the most compelling data points for the value of open-sourcing well-designed infrastructure. The tool built to solve LinkedIn's data pipeline problem had become the data pipeline solution for most of the internet.
\n
\n\nJay Kreps named Kafka after Franz Kafka because it was 'a system optimized for writing' — and then built something that the entire internet writes 7 trillion messages through per day, which is exactly the kind of outcome Franz Kafka would have found deeply, cosmically absurd.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-19T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.188763+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Messaging", "LinkedIn"]}, {"id": "https://techlogstack.com/explore/shopify-llm-evaluation-production-2025/", "url": "https://techlogstack.com/explore/shopify-llm-evaluation-production-2025/", "title": "The 80% Problem: Why Getting an LLM System to 'Works in Demo' Is 20% of the Work", "summary": "How Shopify's Sidekick and Flow agent teams built LLM evaluation infrastructure — LLM judges, merchant simulators, and production mirroring — to close the demo-to-pr", "content_html": "Shopify · Reliability · 19 May 2026
\nEvery team building with LLMs discovers the same brutal truth: 80% quality arrives in a few weeks. The final 15% — the gap between 'impressive demo' and 'product I'd trust with my customers' — takes the rest of the time. Shopify's Flow agent and Sidekick teams lived this curve and came back with a systematic playbook. It is mostly about measurement.
\nZenML analyzed 1,200 production LLM deployments across companies ranging from startups to large enterprises and found a pattern so consistent it has become a rule: reaching 80% quality happens quickly, but pushing past 95% requires the majority of total development time. The teams that hit 80% in four weeks and spend the next six months trying to reach 95% are not failing — they are experiencing the standard engineering curve for AI systems. The teams that mistake 80% for done are the ones shipping products that quietly erode user trust. Shopify's engineering teams, building both Sidekick (the merchant AI assistant) and the Flow agent (automated workflow generation), lived this curve in production. Their solution was not a better model. It was a better measurement system.
\n\n\n\nWHY EVALUATION IS THE HARD PART
\nTraditional software has a truth oracle: does the function return the correct value? LLM systems have no such oracle. A response can be grammatically correct, semantically reasonable, formatted perfectly — and still be wrong in ways that only a domain expert would notice, or only appear wrong on the tenth interaction in a specific workflow. Without a reliable way to measure quality, you cannot improve systematically. You are optimizing blind, hoping that the next prompt change or model upgrade makes things better without making other things worse. Evaluation infrastructure is not overhead — it is the prerequisite for all other AI engineering work.
\n
Shopify's Flow agent generates Shopify Flow automations from natural language — merchants describe what they want to happen ('when an order is over $200, add the customer to my VIP segment'), and the agent produces the workflow. The task requires tool calling (a pattern where an LLM is given a set of available functions (tools) with descriptions, and can request that a specific tool be executed by generating a structured function call — enabling LLMs to take real-world actions beyond text generation) and produces a structured output in a domain-specific format. It sounds well-bounded. In practice, the diversity of merchant intent is vast, the edge cases accumulate rapidly, and subtle errors in the generated workflow — a wrong condition operator, a missing trigger — produce silently incorrect automations that only fail when a merchant's order actually arrives.
\n\n\n\n📏
\nShopify calibrated their LLM judge from a Cohen's Kappa of 0.02 (essentially random — the judge agreed with human evaluators no more than chance would predict) to 0.61, close to the human evaluator baseline of 0.69. The human baseline itself was 0.69 rather than 1.0 — a reminder that human evaluators don't perfectly agree with each other either. The goal is not a perfect judge; it's a judge trustworthy enough that its signals drive reliable engineering decisions.
\n
Shopify's fine-tuned Flow agent passed a hand-crafted 300-example benchmark at high accuracy. When deployed to production shadow traffic, performance on real merchant workflows diverged from the benchmark. The benchmark had been crafted by engineers who knew the system well and implicitly sampled from the distribution they understood. Real merchant intent had a long tail the benchmark didn't capture.
\nThe early LLM judge had a Cohen's Kappa of 0.02 — barely better than random agreement with human evaluators. This meant the judge's verdicts could not reliably distinguish good responses from bad ones. Engineering decisions based on judge verdicts were effectively noise. Human evaluation at scale was impractical. Without a trustworthy quality signal, iteration was slow and direction was unclear.
\nThe team iteratively improved the LLM judge through systematic calibration against human labels (Kappa 0.02 → 0.61), then used it to score production traffic at scale. Production mirroring — routing real traffic through both current and candidate models — generated the failure cases that didn't appear in benchmarks. Those failures were fed back into the training dataset, closing the benchmark-to-production gap.
\nThe gap from 'benchmark-ready' to 'production-ready' closed in two weeks using the production mirroring flywheel. The fine-tuned Flow agent now serves the majority of production traffic. Weekly retraining cycles on H200 GPUs mean the model continuously improves from new production signal rather than drifting as merchant behavior evolves.
\n\n\n\n⚠️
\nThe Human Agreement Ceiling
One of the most grounding facts in Shopify's evaluation system is that human evaluators agreed with each other at Cohen's Kappa of 0.69 — not 1.0. Humans disagree about quality. This is not a failure of the evaluation process; it reflects genuine ambiguity in what 'correct' means for natural language tasks. The practical implication: don't try to build a perfect judge. Build a judge that matches or approaches human agreement levels, and treat that as the meaningful ceiling. Optimizing a judge past the human agreement level is overfitting to individual human annotators, not finding truth.
\n
The merchant simulator deserves particular attention as an engineering pattern. Before any system change ships to production, it is tested against simulated merchant interactions derived from real production conversations. The simulator captures the 'essence' — the underlying merchant goal — from real conversations and replays that goal against the new system. This is fundamentally different from benchmark evaluation: it tests the new system against realistic merchant intent distributions, including the long tail that engineering-crafted benchmarks consistently miss. It is also fundamentally different from A/B testing: it catches regressions before any real merchant sees them, without requiring a traffic split.
\n\n\n\nℹ️
\nSynthetic Data: Closing the Training Data Gap
The Flow agent's fine-tuning training data was almost entirely synthetic — generated by an LLM, not labeled by humans. The process: sample a real production workflow, use a stronger model to generate a plausible natural-language request that would produce it, construct the ideal multi-turn tool trajectory. The synthetic data generation was the majority of the engineering effort. The resulting dataset covered the breadth of Flow's usage in a way that manual annotation never could — because the diversity of real workflows provided the supervision signal, and the LLM provided the scale. This is the emerging pattern for fine-tuning specialists: synthetic data from real production outputs, not expensive human annotation from scratch.
\n
\n\n\n🔬
\nThe Industry Pattern: ZenML's 1,200 Case Studies
ZenML's LLMOps database of 1,200+ production deployments confirms that Shopify's experience is universal, not exceptional. The summary from their analysis: 'Perhaps this is a truism by now, but you'll spend more time building evaluation infrastructure than you will on the actual application logic. And if you're not, you're probably shipping broken features.' LLM-as-judge has emerged as the dominant pattern for scalable quality measurement. But every successful deployment maintains human-in-the-loop golden datasets for critical domains. The dual-layer approach — LLM judges for velocity, human ground truth for calibration — is the de facto standard.
\n
\n\n\nℹ️
\nThe A/B Test That Isn't
Teams new to LLM evaluation often reach for A/B testing as the measurement tool: split traffic, measure conversion, pick the winner. A/B testing has a fatal problem for LLM evaluation: a 5% improvement in a downstream metric like merchant click-through might take weeks of traffic to reach statistical significance — and you may have introduced a subtle quality regression in a different dimension that the metric doesn't capture. Production mirroring with direct output comparison is faster and richer: you see whether response quality improved for the same inputs, without waiting for downstream business metric movement. Business metrics confirm value; output comparison guides engineering.
\n
\n\n\nTHE COST CURVE IS ASYMMETRIC
\nThe 80% → 95% quality journey is asymmetric in effort. The first 80% comes from model capability: the LLM already knows how to generate text, use tools, and follow instructions. The final 15% comes from understanding the specific failure modes of your specific application on your specific user distribution — and that knowledge cannot be bought or downloaded. It is earned through measurement, systematic failure analysis, and targeted training data creation. The companies that invest in this domain-specific evaluation work build durable advantages over those that simply upgrade to the next model version and hope.
\n
\n\n\n🏭
\nNotion AI (referenced in ZenML's analysis) built a multi-layer evaluation stack that balances speed and cost: cheap heuristic checks on every commit, LLM judge scoring on every merge, and expensive human evaluation on every release candidate. Teams that adopted this tiered approach reported 10x faster development velocity compared to running full human evaluation on every change. The insight: match eval depth to the stakes of the change, not to a uniform 'always run everything' policy.
\n
Shopify's evaluation architecture is best understood as a flywheel: production traffic generates failures, failures feed the training pipeline, retraining improves the model, the improved model generates fewer failures, and the cycle continues. Each turn of the flywheel reduces the gap between benchmark performance and production performance. The flywheel only works if each component — quality measurement (LLM judge), failure collection (production mirroring), training (fine-tuning pipeline), deployment (shadow traffic + promotion) — is production-grade itself. A miscalibrated judge produces misleading signal. A flaky training pipeline slows iteration. A low-coverage benchmark misses the failures that actually matter.
\n# LLM Judge calibration: the process that takes you from Kappa 0.02 to 0.61\n# A judge is only useful if it agrees with humans. Measure this first.\n\nfrom sklearn.metrics import cohen_kappa_score\n\ndef calibrate_llm_judge(judge_prompt: str, calibration_set: list[dict]) -> float:\n \"\"\"\n calibration_set: list of {conversation, human_label} pairs\n human_label: 'good' | 'bad' | 'needs_improvement'\n Returns Cohen's Kappa between judge and human labels.\n \"\"\"\n judge_labels = []\n for sample in calibration_set:\n # Ask the judge to evaluate this conversation\n judge_verdict = call_llm(judge_prompt, sample['conversation'])\n judge_labels.append(judge_verdict)\n \n human_labels = [s['human_label'] for s in calibration_set]\n kappa = cohen_kappa_score(human_labels, judge_labels)\n return kappa # target: >0.60 before trusting judge at scale\n\n# The calibration loop:\nkappa = 0.02 # initial judge is barely better than random\nwhile kappa < 0.60:\n # Analyze where judge and humans disagree\n disagreements = find_disagreements(calibration_set, current_judge_labels)\n \n # Improve judge prompt based on disagreement patterns:\n # - Add clarifying criteria for ambiguous cases\n # - Add few-shot examples from disagreements (human = ground truth)\n # - Adjust rubric language to match human intuitions\n new_judge_prompt = improve_prompt(current_judge_prompt, disagreements)\n \n kappa = calibrate_llm_judge(new_judge_prompt, calibration_set)\n print(f\"Kappa after iteration: {kappa:.2f}\") # logs: 0.02 → 0.15 → 0.31 → 0.48 → 0.61\n\n\nPRODUCTION MIRRORING: THE GROUND TRUTH TEST
\nBenchmarks are necessary but not sufficient. A benchmark is a fixed dataset that reflects the understanding of the engineers who created it. Production traffic reflects the actual diversity of user intent — including all the edge cases, unusual phrasings, and unexpected use patterns that no engineer anticipated. Production mirroring routes a percentage of real traffic through both the current model and the candidate model simultaneously, comparing outputs. Differences trigger human review of high-value or uncertain cases. This is the only way to discover whether a model improvement that looks good on a benchmark actually performs better for real users — or merely performs better on what engineers think real users want.
\n
\n\n\nℹ️
\nThe Synthetic Data Generation Pipeline
Shopify's Flow agent training data was generated through a three-step pipeline: Step 1 — sample a diverse set of validated production workflows (at least one workflow per unique workflow descriptor, from merchants with two or more qualifying workflows). Step 2 — use a stronger LLM to generate a plausible natural-language merchant request that would lead to that workflow. Step 3 — construct the ideal multi-turn tool call trajectory from request to completed workflow. The resulting dataset had two properties manual annotation lacks: scale (the production workflow corpus is large) and grounding (every training example was derived from a real workflow that actually ran).
\n
\n\n\n✅
\nTangle: The ML Pipeline That Enables Weekly Retraining
The full training pipeline — data collection, synthetic data generation, fine-tuning, evaluation, deployment — runs on Tangle, Shopify's open-source ML experimentation platform. Tangle composes each pipeline step as a reproducible workflow with intelligent caching: only the steps affected by a change re-run. This means a change to the synthetic data generator doesn't trigger a full pipeline rerun from scratch — only the data generation step and its downstream steps re-execute. The caching infrastructure is what makes weekly retraining economically and operationally viable. Without it, the iteration cycle would be measured in months, not weeks.
\n
\n\n\n⚠️
\nGolden Datasets Are Non-Negotiable
ZenML's analysis is unambiguous: every successful production LLM deployment they analyzed maintains human-in-the-loop golden datasets for critical domains. LLM judges are used for velocity — scoring production traffic at scale. But they drift. A judge trained on last month's quality standards may give wrong verdicts on today's outputs. Golden datasets — small, carefully curated, human-labeled examples that represent ground truth — anchor the judge calibration and detect judge drift. Without a golden dataset, you have no way to know when your quality measurement system itself has stopped working.
\n
\n\n\n✅
\nThe Two-Week Rule
Shopify's experience with the production mirroring flywheel produced a rule of thumb that has since appeared in multiple other teams' postmortems: if your candidate model passes benchmark evaluation, it takes approximately two weeks of production mirroring to confirm whether it's truly production-ready. Two weeks of real traffic at a shadow percentage generates enough diverse examples to surface the tail failures that the benchmark didn't cover. If the flywheel is working, those failures are incorporated into the training data and the model improves. If the failures are systematic — indicating a training distribution problem rather than isolated edge cases — the two weeks reveals this before the model is promoted to full production.
\n
The evaluation architecture for production LLM systems has four components that form a cycle. Benchmark evaluation provides fast, reproducible quality gates during development. LLM-as-judge scoring provides continuous quality measurement at production traffic scale. Production mirroring provides ground truth about whether a candidate model performs better for real users. The training flywheel converts production failures into training examples, closing the gap each cycle. Each component is necessary; none is sufficient alone.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE MERCHANT SIMULATOR AS PRE-DEPLOYMENT SAFETY NET
\nThe merchant simulator sits between benchmark evaluation and production mirroring — it's a synthetic production environment. It replays real merchant intents (extracted from production conversations) against candidate systems in a controlled environment, before any real merchant sees the new system. This catches the specific failure mode that benchmarks miss: correct behavior on engineer-anticipated test cases, incorrect behavior on the realistic distribution of merchant intent in production. The simulator doesn't replace production mirroring — it prevents the worst regressions from reaching the production mirroring stage at all.
\n
\n\n\n⚠️
\nEval Budget vs Training Budget: The Cost Trap
ZenML's analysis of 1,200 production deployments found that teams frequently discover that running comprehensive evaluations on every commit burns through inference budget faster than production traffic. Running a full eval suite — LLM judge on 1,000 examples × multiple iterations per PR — can cost more per day than serving users. The solution is a tiered eval strategy: fast, cheap unit evals on every commit; comprehensive judge-scored evals on every merge; full production mirroring only for release candidates. Eval should be sized to the stakes of what's being changed, not run at maximum coverage on every code change.
\n
\n\n\n🧪
\nThe Multi-LLM Annotation Pattern
For high-stakes quality assessments (like Shopify's Global Catalogue product taxonomy), a single LLM judge has too much variance. The production pattern is to run multiple LLMs independently on the same evaluation task, then use an arbitration system — a specialized model — to resolve disagreements. This ensemble approach dramatically reduces false positives in quality assessment: a response that confuses one model but is rated correctly by three others is probably correct. The arbitration model applies structured ruling logic for edge cases that simple voting would misclassify. The pattern adds cost but reduces the error rate of the quality signal for critical decisions.
\n
The 80% quality curve is the defining challenge of production AI engineering. The teams that accept it and build systematic measurement infrastructure navigate it successfully. The teams that are surprised by it and try to push past it with more prompting and model upgrades are still on it.
\nWhat to remember
\n\n\n✅
\nThe Universal Pattern Across 1,200 Deployments
ZenML's analysis of 1,200 production LLM deployments confirms Shopify's findings are not unique: the organizations extracting real value from AI are not the ones with the most innovative demos — they are the ones doing the less glamorous engineering work: building evaluation pipelines, implementing guardrails, designing for uncertainty, and treating their LLM systems with the same rigor they'd apply to any critical infrastructure. The pattern is consistent across startups, mid-market, and enterprise. Model quality is table stakes. Evaluation infrastructure is competitive differentiation.
\n
\n\n\nEVALUATION INFRASTRUCTURE IS PRODUCT
\nThe merchant simulator, the calibrated LLM judge, the production mirroring pipeline, the golden dataset maintenance process — these are not internal tooling that engineers built for themselves. They are the product quality infrastructure that Shopify's merchants depend on, even though they will never see it. Every improvement to the evaluation system is an improvement to Sidekick's and Flow's reliability. Building evaluation infrastructure is building the product. Teams that separate 'evaluation tooling' from 'product work' are misclassifying one of their highest-value investments.
\n
\n\nShopify's engineers discovered that getting an AI to produce a correct Shopify Flow automation 80% of the time takes two weeks, and getting it to 95% takes the rest of the year — which is either a profound insight about probabilistic systems or a profound insight about how hard it is to write good evals for commerce automation, and it turns out to be both.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-19T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.889156+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "Shopify"]}, {"id": "https://techlogstack.com/explore/ibm-quantum-advantage-2026/", "url": "https://techlogstack.com/explore/ibm-quantum-advantage-2026/", "title": "Quantum Computing Just Beat the Best Classical Computer — Here Is the Engineering That Made It Happen", "summary": "How Q-CTRL achieved 3,000× quantum speedup over classical computers on May 6, 2026 — and how IBM's engineering stack made it possible: Fire Opal, qLDPC, and quantum-", "content_html": "IBM · Distributed Systems · 19 May 2026
\nOn May 6, 2026, Q-CTRL ran a materials science simulation on an IBM quantum computer in 2 minutes. The best classical supercomputer needed over 100 hours to reach the same accuracy — and then gave up. The day before, IBM's quantum computers simulated a 12,635-atom protein with Cleveland Clinic and RIKEN, 40 times larger than anything attempted six months prior. After 30 years of promises, quantum advantage arrived. Here is what actually changed.
\nOn May 19, 2026, Google Trends for India showed the query \"what is quantum computing in simple terms\" with a BREAKOUT signal — the highest possible designation, meaning search volume had increased so dramatically that the normal percentage scale couldn't contain it. The trigger was two announcements that had landed within 48 hours of each other. On May 5, scientists at Cleveland Clinic, RIKEN, and IBM used quantum computers to simulate trypsin, a protein with 12,635 atoms — the largest biologically meaningful molecule ever simulated with quantum hardware, 40 times larger than what the same method could achieve just six months prior. On May 6, Q-CTRL announced they had run a materials science calculation on an IBM quantum processor in 2 minutes. The best classical supercomputer took over 100 hours to reach equivalent accuracy. That is a 3,000× speedup. The physics community called it practical quantum advantage — the first time a quantum computer had demonstrably outperformed the best classical tool on a problem of real commercial relevance.
\n\nFor years, quantum computing has been a promise. Now, quantum computers are producing results that matter to science. The systems we simulated here are the kind of molecules that biologists and chemists work with in the real world.
— — Jay Gambetta, Director of IBM Research — IBM Think 2026, Boston
Understanding why these results matter requires understanding what stood in the way. NISQ (Noisy Intermediate-Scale Quantum — the current era of quantum computing, characterized by processors with 50–1,000 qubits that are not error-corrected, meaning errors accumulate as circuit depth grows and place hard limits on what computations can run reliably) quantum computers — the hardware that exists today — are fundamentally noisy. Every two-qubit gate (the fundamental entangling operation in quantum computing that creates correlations between qubits — essential for quantum algorithms but a primary source of error in NISQ hardware, with typical error rates of 0.1–1% per gate) introduces a small probability of error. At shallow circuit depths with a handful of gates, this is manageable. At the depth required for commercially meaningful simulations — 10,000+ two-qubit gates across 120 qubits — errors compound exponentially and the computation collapses into noise. For years, this was the wall. The May 2026 results are not the wall coming down. They are the first evidence that engineers have found a way to work within the wall's constraints precisely enough, and extend it enough, that real problems now fall on the quantum side of it.
\n\n\n\nTHE SEARCH BREAKOUT EXPLAINED
\nGoogle Trends in India showed a BREAKOUT on 'what is quantum computing in simple terms' within hours of the May 5-6 announcements reaching mainstream media. India's large engineering student population — studying at IITs, NITs, VITs, and hundreds of other technical universities — represents one of the highest densities of people who both understand enough to be curious and don't yet know enough to explain it themselves. The query 'in simple terms' is the signature of real scientific interest crossing from specialist to general audience. BREAKOUT signals on engineering topics in India reliably indicate a moment when a technical development has become a cultural one.
\n
Q-CTRL's achievement used an IBM 156-qubit Heron processor on the IBM Quantum Platform, enhanced by Q-CTRL's own Fire Opal performance-management software. The target problem was the Fermi-Hubbard model (a foundational physics model that describes how electrons interact in a crystal lattice — capturing phenomena like high-temperature superconductivity and quantum magnetism — problems whose classical simulation cost grows exponentially with system size, making them natural candidates for quantum advantage) — a system of 60 interacting electrons in a 1D chain, using 120 of the chip's 156 qubits and executing over 10,000 two-qubit gate operations. The classical competitor was ITensor's TDVP solver running on a 32-vCPU, 64GB-RAM AWS instance — the acknowledged best-in-class classical tool for this class of problem. The quantum computation completed in ~2 minutes. The classical computation, to reach the same accuracy, required over 100 hours — and at longer evolution times required over 160 hours before the two results diverged irreconcilably, meaning the classical computer ran out of ability to match the quantum result entirely.
\n\n\n\n⚛️
\nQ-CTRL's Fire Opal compiler reduced the number of two-qubit gates required for the Fermi-Hubbard calculation by 60% compared to IBM's native Qiskit implementation of the same algorithm. Fewer gates means less error accumulation. This single optimization was the difference between a circuit that collapsed into noise at this scale and one that produced results accurate enough to match — and then exceed — the classical benchmark.
\n
NISQ quantum processors accumulate errors with every two-qubit gate. For shallow circuits (hundreds of gates), error mitigation techniques can recover useful results. For commercially meaningful simulations (10,000+ gates), errors historically compounded to the point where the quantum output was indistinguishable from random noise. This wall had blocked practical quantum advantage for three decades.
\nEvery additional two-qubit gate multiplies error probability. IBM's native Qiskit compiler produced correct but gate-heavy implementations. Q-CTRL's Fire Opal compiler took the same algorithm and reduced gate count by 60% through advanced circuit optimization and error suppression techniques built on years of quantum control research. The 60% reduction was the difference between circuits that collapsed into noise and circuits that produced valid results.
\nMay 5: IBM, Cleveland Clinic, and RIKEN simulated a 12,635-atom protein using quantum-centric supercomputing — fragmenting the molecule, computing quantum-mechanical behavior on IBM Heron processors, and assembling results on Fugaku and Miyabi-G supercomputers. May 6: Q-CTRL demonstrated 3,000× speedup on the Fermi-Hubbard model, completing in 2 minutes what took classical computers 100+ hours.
\nOn May 6, 2026, Q-CTRL declared practical quantum advantage — the first time a quantum computer had outperformed the best available classical tool on a problem of known commercial relevance, using hardware accessible to any developer via the IBM Quantum Platform. IBM CEO Arvind Krishna had predicted quantum advantage would arrive in 2026. The prediction was correct.
\n\n\n\nℹ️
\nThe Cleveland Clinic Protein Simulation
The May 5 protein simulation used a different approach — quantum-centric supercomputing (QCSC) — pairing IBM Heron quantum processors at both Cleveland Clinic (USA) and RIKEN (Japan) with two classical supercomputers: Fugaku at RIKEN and Miyabi-G at the University of Tokyo. The key algorithm was EWF-TrimSQD — a quantum-classical hybrid that fragmented the 12,635-atom trypsin protein into computable pieces, computed quantum-mechanical behavior on QPUs (up to 94 qubits, ~6,000 quantum operations per fragment), and reconstructed the full protein's behavior on classical supercomputers. The result: a 40-fold increase in system size and 210× improvement in accuracy compared to results from just six months earlier.
\n
\n\n\n⚠️
\nWhat These Results Are Not
Precision is important here. The Fermi-Hubbard result is not proof that quantum computers beat classical computers at everything — or even most things. The advantage holds for this specific class of fermionic simulation problems, which scale poorly for classical computers by a known theoretical argument. Breaking RSA-2048 with Shor's algorithm requires hundreds of thousands to millions of physical qubits under error correction — a challenge orders of magnitude harder. The May 2026 results are the first concrete proof that quantum advantage is achievable on useful, commercially relevant problems with today's hardware, properly engineered.
\n
\n\n\nℹ️
\nThe Fermi-Hubbard Model: Why This Problem Matters
The Fermi-Hubbard model (a foundational model in condensed matter physics that describes interacting electrons on a lattice — used to understand phenomena including high-temperature superconductivity, Mott insulators, and quantum magnetism) is not an academic toy problem. It is the theoretical framework that physicists use to understand high-temperature superconductors — materials that could enable lossless power transmission and dramatically more efficient computing. Classical computers struggle with Fermi-Hubbard at scale because the number of quantum states grows exponentially with system size — a 60-electron system has 2^60 possible states, far beyond any classical memory capacity. Quantum computers naturally represent these states using quantum superposition. This is the textbook case of exponential quantum advantage — and May 2026 is the first real-world confirmation that it holds in practice.
\n
\n\n\nTHE 300MM WAFER SHIFT: SCALING QUANTUM MANUFACTURING
\nAlongside the algorithm and software achievements, IBM made a manufacturing announcement that will define the next decade of quantum hardware: shifting quantum processor wafer fabrication to 300mm wafers at the Albany NanoTech Complex — the same fabrication scale used by the most advanced classical semiconductor fabs. The shift from smaller wafers doubles IBM's development speed while enabling 10× more complex chips for the fault-tolerant error correction roadmap. This is the semiconductor industry's hard-won manufacturing knowledge being applied to quantum hardware — the industrialization of quantum chip production.
\n
The Q-CTRL result did not emerge from better quantum hardware alone — it emerged from a full engineering stack combining IBM's hardware, Q-CTRL's compiler, and years of quantum control research. Three layers mattered: the hardware layer (IBM Heron's 156-qubit chip with improved coherence times and gate fidelity), the compilation layer (Q-CTRL's Fire Opal reducing gate count by 60% through circuit optimization and noise-aware compilation), and the error suppression layer (runtime techniques that actively suppress errors during execution rather than correcting them after). None of these layers alone would have been sufficient. The result is an emergent property of all three operating together.
\n# Conceptual: What Fire Opal does differently from native Qiskit compilation\n# The 60% gate reduction is the engineering story in code form\n\n# Native Qiskit compilation: correct but gate-heavy\nfrom qiskit import QuantumCircuit, transpile\nfrom qiskit_ibm_runtime import QiskitRuntimeService\n\nservice = QiskitRuntimeService()\nbackend = service.backend('ibm_heron_r2') # 156-qubit Heron\n\n# The Fermi-Hubbard Trotter circuit before optimization:\n# Each Trotter step requires multiple CNOT layers\n# At 90 Trotter steps: ~15,000+ two-qubit gates in the naive implementation\ncircuit = build_fermi_hubbard_circuit(n_qubits=120, n_trotter_steps=90)\nnative_transpiled = transpile(circuit, backend=backend)\nprint(f\"Native gate count: {native_transpiled.count_ops()['cx']} CX gates\")\n# Output: ~15,000+ CX gates → error rate too high → output is noise\n\n# Q-CTRL Fire Opal: noise-aware compilation\nimport fire_opal\n\n# Fire Opal applies:\n# 1. Circuit rewriting: finds equivalent circuits with fewer gates\n# 2. Noise-aware mapping: places qubits to minimize cross-talk\n# 3. Dynamical decoupling: inserts refocusing pulses to cancel drift\n# 4. Gate fusion: combines adjacent compatible gates\noptimized_result = fire_opal.run(\n circuits=[circuit],\n backend=backend,\n optimization_level='aggressive',\n error_suppression=['dynamical_decoupling', 'gate_twirling']\n)\nprint(f\"Fire Opal gate count: ~6,000 CX gates\")\n# 60% reduction → circuit runs within error tolerance → useful result\n# 120 qubits × 90 Trotter steps × 2 minutes wall time\n# vs TDVP classical: 100+ hours before classical diverges from quantum\n\n\nERROR SUPPRESSION VS ERROR CORRECTION
\nThe May 2026 results were achieved with error suppression, not error correction. The distinction is fundamental. Error correction (the goal for 2029) uses logical qubits — groups of physical qubits that encode information redundantly and can detect and fix errors in real-time. It requires hundreds of physical qubits per logical qubit. Error suppression (what Q-CTRL and IBM use now) cannot fix errors — it minimizes them through circuit optimization, noise-aware compilation, and runtime control techniques. Error suppression works within the limits of NISQ hardware. Error correction eliminates those limits entirely. The 3,000× result was achieved within the NISQ limits. What becomes possible once error correction arrives is qualitatively different.
\n
\n\n\nℹ️
\nIBM Quantum Loon: All Hardware Elements for Fault Tolerance
IBM's roadmap toward fault-tolerant quantum computing advanced significantly in November 2025 with the IBM Quantum Loon processor — the first to demonstrate all hardware components required for fault-tolerant quantum computing: c-couplers for long-range qubit connectivity, qubit reset between computations, and high-fidelity gates at FTQC-relevant speeds. Simultaneously, IBM achieved real-time qLDPC (quasi-cyclic Low-Density Parity-Check codes — a class of quantum error-correcting code that IBM believes provides the most efficient path to large-scale fault tolerance, requiring fewer physical qubits per logical qubit than the surface code used by most competitors) decoding in under 480 nanoseconds — a full year ahead of schedule. Loon is not a production processor — it is an experimental validation platform. But it proves the components exist.
\n
\n\n\n✅
\nThe EWF-TrimSQD Algorithm: 40× Larger in Six Months
The Cleveland Clinic simulation's 40-fold increase in system size in six months was driven by a new algorithm: EWF-TrimSQD (Embedding Workflow with Tailored Reduced-qubit Molecular Dynamics). It improved the efficiency of how the protein was fragmented into quantum-computable pieces, reducing overhead per fragment and enabling larger total simulations within the same qubit budget. The algorithm was a joint development between IBM, Cleveland Clinic, and RIKEN. The 40× improvement in six months means the scaling is not linear — each algorithmic improvement compounds with the hardware improvements, accelerating both simultaneously.
\n
\n\n\n⚠️
\nWhat Quantum Does Not Yet Beat Classical At
Intellectual honesty requires noting what Q-CTRL's team itself acknowledged: on variational quantum eigensolver problems (a different class of quantum chemistry simulation), they found in their own research that a new classical method they developed outperformed the quantum computer just days before their quantum advantage announcement. IBM's Borja Peropadre was candid at CES 2026 about this: quantum and classical methods advance simultaneously, and each quantum claim must be verified against the best classical methods — not methods from two years ago. The advantage frontier moves, and keeping track of it requires constant benchmarking against current classical state-of-the-art.
\n
\n\n\n✅
\nIBM Quantum Platform: Cloud Access to Advantage
The IBM Quantum Platform has been accessible to developers via cloud since May 2016 — a full decade before the May 2026 advantage demonstrations. The platform provides tiered access: free access to smaller processors for experimentation, and paid premium access to the most capable systems including the Heron processors used in the advantage demonstrations. Q-CTRL's 3,000× speedup was demonstrated on hardware accessible to any registered IBM Quantum Platform user. The advantage is not locked in a research lab. It is available now, on public cloud infrastructure, to any team willing to develop the quantum expertise to use it.
\n
The architecture of both May 2026 achievements reflects a common pattern: quantum processors are not stand-alone computers that replace classical ones. They are specialized accelerators for specific types of computation, tightly integrated with classical CPUs and GPUs that handle the parts of the problem where quantum offers no advantage. IBM calls this Quantum-Centric Supercomputing (QCSC) — a heterogeneous computing architecture where tasks are assigned to the compute layer where they run best. Understanding this architecture is essential to understanding what quantum advantage actually means in practice.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nQUANTUM + AI: CONVERGENCE, NOT COMPETITION
\nIBM CEO Arvind Krishna at Think 2026 was direct: \"Quantum and AI do not compete; they converge and complement each other.\" Quantum can solve optimization and simulation problems that AI cannot reach through gradient descent. AI can learn from quantum-computed results and develop faster classical approximations. The trajectory: quantum computes what AI cannot, AI learns from what quantum computed, the combination advances faster than either could alone. This is why IBM's quantum research program sits alongside its AI research program rather than in competition with it — and why the Cleveland Clinic drug discovery work matters: quantum simulates molecular interactions that ML models can then learn to approximate.
\n
\n\n\nℹ️
\n10 Years of Cloud Quantum: IBM Quantum Platform
IBM put the first quantum computer on the cloud on May 4, 2016. IBM Think 2026 coincided almost exactly with the 10th anniversary of cloud-accessible quantum computing. In that decade, the IBM Quantum Platform grew from a single 5-qubit processor to a fleet of processors up to 156 qubits, serving hundreds of thousands of users globally via tiered access plans. The cloud accessibility is not incidental to the May 2026 results — Q-CTRL's 3,000× speedup was achieved on hardware accessible to any developer via the IBM Quantum Platform's API, not on a private research machine. Practical quantum advantage arrived on public cloud infrastructure.
\n
May 5–6, 2026 is the week where quantum computing stopped being a future technology and became a present one — for specific, bounded, commercially relevant problems. The lessons here are about what actually changed, what the engineering looks like, and what it means for the decade ahead.
\nWhat to remember
\n\n\n✅
\nThe Community Advantage Tracker
IBM, Algorithmiq, researchers at the Flatiron Institute, and BlueQubit are contributing results to an open, community-led quantum advantage tracker — a systematic framework for verifying quantum advantage claims across three experiment types: observable estimation, variational problems, and classically verifiable problems. This tracker is the scientific community's answer to the reproducibility question: quantum advantage claims require independent verification, and the tracker provides the framework for that verification. It is the peer review system for quantum advantage — and its existence is itself evidence that the field has matured from speculation to engineering.
\n
\n\n\nTHE DRUG DISCOVERY IMPLICATION
\nCleveland Clinic's motivation for the protein simulation work is direct: drug discovery. If quantum computers can accurately simulate how drug molecules bind to protein targets like trypsin, pharmaceutical researchers can screen candidate molecules computationally before any physical experiment. The typical drug development cycle takes over 10 years and costs billions of dollars. Accurate quantum simulation of binding energies could identify non-starters earlier and prioritize promising candidates faster. The current 12,635-atom result is a milestone, not a final destination. But the 40× size increase in six months shows the trajectory is steep.
\n
\n\nFor 30 years, quantum computing was always 10 years away — until May 2026, when Q-CTRL ran a computation in 2 minutes that took the best classical supercomputer 100 hours, and the only thing that changed was that engineers got a 60% better compiler.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-19T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.975775+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Distributed Systems", "IBM"]}, {"id": "https://techlogstack.com/explore/netflix-chaos-monkey-2011/", "url": "https://techlogstack.com/explore/netflix-chaos-monkey-2011/", "title": "Netflix Unleashed a Monkey With a Weapon in Its Own Data Center — On Purpose", "summary": "How Netflix's 2008 database outage led to Chaos Monkey — and how deliberately killing production servers during business hours spawned an entire engineering discipli", "content_html": "Netflix · Chaos Engineering · 18 May 2026
\nIt was 2011 and Netflix had just migrated hundreds of microservices to AWS. Their architecture was distributed, horizontally scaled, and theoretically fault-tolerant. But theory and production are different things. The only way to know if a system could survive failures was to cause failures — constantly, deliberately, during business hours, and in production. So they built a monkey.
\n\nThe name comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables — all the while we continue serving our customers without interruption.
— — Yury Izrailevsky & Ariel Tseitlin — via The Netflix Simian Army, Netflix Tech Blog, July 19, 2011
The origin of Chaos Monkey is not a clever engineering insight — it is a three-day disaster. In August 2008, Netflix was still primarily a DVD-by-mail business, running its technology on vertically scaled servers in its own data centers. A major database corruption took down the entire system. For three days, Netflix could not ship DVDs to its customers. It wasn't a complicated failure. It was a single point of failure (a component whose failure brings down the entire system — the exact opposite of a fault-tolerant distributed architecture) at the most basic level: one database, one failure mode, total outage. The company's engineering leadership concluded that the only path forward was to move away from centralized relational databases in their own datacenter toward highly reliable, horizontally scalable, distributed systems in the cloud. They chose Amazon Web Services. The seven-year cloud migration that followed would produce one of the most influential engineering philosophies in the history of distributed systems.
\nThe migration to AWS presented a new problem in place of the old one. Netflix was moving from a single monolith with a small number of failure points — each catastrophic — to a microservices architecture (a system design where an application is broken into many small, independently deployable services that communicate over a network, improving scalability and team autonomy at the cost of increased distributed systems complexity) with hundreds of services, each potentially failing in its own unique way. The distributed system was theoretically more resilient. But theory is not production. Netflix's engineers designed systems with graceful degradation in mind — if the recommendations service failed, show popular titles instead of personalized ones; if the search service was slow, streaming should still work. They wrote the code. They reviewed it. They tested it in staging. And then they realized: there was no way to know if the fault tolerance actually worked without experiencing actual failures. The staging environment couldn't reproduce the chaos of production. Controlled tests couldn't capture the emergent failure modes of hundreds of interdependent services under real load.
\n\n\n\nTHE CORE INSIGHT: FAIL CONSTANTLY
\nNetflix's founding philosophy for Chaos Engineering was radical in its simplicity: the best way to avoid failure is to fail constantly. If you only experience failures accidentally, in production, at 3am, your engineers have no muscle memory for responding to them and your systems have never been forced to prove their resilience claims. If you fail constantly, during business hours, with engineers present — your systems either prove they can recover or they expose the gaps so engineers can fix them before those gaps become incidents.
\n
Chaos Monkey is, mechanically, a simple tool. It runs continuously across Netflix's AWS environment and at some point during business hours, picks one EC2 instance at random from each cluster and terminates it. No warning. No coordination. No grace period. The instance just stops. This deceptively simple act forces every service in Netflix's architecture to prove, continuously, that it can tolerate the loss of an individual instance. Services that depend on a single backend instance fail immediately and obviously. Services built with proper fallbacks — load balancers, retries, graceful degradation paths — continue working. The business hours constraint is deliberate: when Chaos Monkey strikes at 2pm on a Tuesday, engineers are at their desks and can respond to any cascading failure. Striking at 2am would produce the exact scenario Netflix was trying to avoid — unplanned, unattended failures with no one ready to respond.
\n\nNetflix's vertically scaled infrastructure suffered a major database corruption that halted DVD shipping for three days. The root cause was architectural: a single relational database instance, a single point of failure. No redundancy, no graceful degradation, no recovery path faster than manual intervention. The outage made the problem concrete: this architecture couldn't support Netflix's growth.
\nMoving to hundreds of microservices on AWS solved the single-point-of-failure problem at the architecture level — but introduced new questions: did the code actually implement the graceful degradation it was supposed to? Staging environments couldn't tell you. Code review couldn't tell you. The only honest answer required production failures, and those were the thing Netflix was trying to avoid.
\nNetflix built Chaos Monkey — a script that randomly terminates EC2 instances during business hours — and deployed it in all production environments. Engineers came in every day knowing that Chaos Monkey was running, knowing their services might get an instance killed at any moment, and knowing they had to build recovery mechanisms or face a very bad afternoon. The tool made fault tolerance a daily engineering discipline, not a theoretical design principle.
\nOn September 25, 2014, AWS rebooted approximately 10% of its EC2 instances without warning. Netflix's systems handled it without customer impact. Netflix explicitly credited Chaos Monkey: the engineers had already been building and proving recovery mechanisms every day for years. When AWS created an unplanned failure event at scale, Netflix's systems responded exactly as they'd been trained to respond — automatically, gracefully, and without requiring an emergency war room.
\n\n\n\n🐒
\nChaos Monkey was one of the first systems Netflix engineers built in AWS during the cloud migration. Not a caching layer, not a deployment system, not a monitoring platform — a tool to randomly kill their own production servers. This sequencing was intentional: the discipline came first, and the architecture was shaped by it.
\n
\n\n\nℹ️
\nThe Rambo Architecture
Netflix's engineering team coined the term Rambo Architecture for the design philosophy that Chaos Monkey enforced: each system must be able to succeed no matter what, even all on its own. If the recommendations service is down, still respond — show popular titles. If the search service is slow, streaming still works. If a dependent microservice returns an error, handle it gracefully. Every service is both a potential failure source and a potential victim of failures, and must be designed for both roles simultaneously.
\n
The success of Chaos Monkey triggered a proliferation. If randomly killing instances made Netflix more resilient to instance failures, what would it take to become resilient to other failure categories? In July 2011 — the same blog post that named Chaos Monkey publicly — Netflix announced the Simian Army: a growing suite of failure-injection and resilience-verification tools, each targeting a different class of failure. The roster was remarkable in its scope and its naming creativity. Latency Monkey (a tool that injects artificial delays into Netflix's RESTful service communication layer, simulating network degradation to verify that upstream services detect and respond to downstream slowdowns appropriately) introduced artificial delays in service communication to simulate degradation. Conformity Monkey identified and shut down instances not following engineering best practices. Doctor Monkey ran health checks and removed unhealthy instances from service. Janitor Monkey cleaned up unused cloud resources to reduce costs and complexity. Security Monkey hunted for security vulnerabilities. 10-18 Monkey detected multi-region configuration problems. And Chaos Gorilla (a Simian Army tool that simulates the failure of an entire AWS availability zone — one step up from Chaos Monkey's instance-level failures, testing whether Netflix's architecture could survive losing an entire AZ) simulated the complete failure of an AWS availability zone.
\n\n\n\n⚠️
\nChaos Kong: The Region Killer
Above Chaos Gorilla in the hierarchy sat Chaos Kong — the most extreme tool in the Simian Army, designed to simulate the complete failure of an entire AWS region. If Chaos Monkey proved Netflix could survive an instance failure and Chaos Gorilla proved it could survive an AZ failure, Chaos Kong tested the hardest question: could Netflix continue streaming if us-east-1 went dark? The answer, after years of Chaos Engineering practice, was yes — with careful architecture involving active-active multi-region deployment and data replication strategies that Netflix documented in subsequent engineering blog posts.
\n
The most important thing Chaos Monkey fixed was not a technical system — it was an organizational incentive. Before Chaos Monkey, engineers at Netflix could ship code that was theoretically fault-tolerant but practically fragile without facing immediate consequences. The fragility would only become visible during a real, unplanned outage — at which point it was someone else's problem. After Chaos Monkey, the consequences were immediate and personal: if your service didn't handle instance failures gracefully, Chaos Monkey would expose this during your working hours, while you were at your desk, with your team watching. This behavioral economics effect — where the cost of fragility was paid by the person who created it, immediately — transformed how Netflix engineers thought about resilience. It was no longer a design principle to be aspirationally implemented. It was a daily test to be continuously passed.
\n# Simplified version of what Chaos Monkey does\n# Real implementation was originally Java, later Go (v2.0)\n# Runs continuously during configurable business hours\n\nimport random\nimport time\nfrom datetime import datetime\n\nclass ChaosMonkey:\n def __init__(self, aws_client, excluded_clusters=None):\n self.aws = aws_client\n self.excluded = excluded_clusters or []\n \n def is_business_hours(self) -> bool:\n \"\"\"Only run during business hours so engineers are present.\n The key safety constraint of Chaos Monkey's original design.\"\"\"\n now = datetime.now()\n return (\n now.weekday() < 5 and # Monday–Friday\n 9 <= now.hour < 17 # 9am–5pm local time\n )\n \n def run(self):\n while True:\n if self.is_business_hours():\n # Identify all clusters Chaos Monkey is configured to target\n clusters = self.aws.get_all_clusters()\n \n for cluster in clusters:\n if cluster.name in self.excluded:\n continue\n \n # Pick one instance at random from each cluster\n instances = cluster.get_running_instances()\n if not instances:\n continue\n \n victim = random.choice(instances)\n \n # Terminate it. No warning. No coordination.\n # If the system doesn't survive this, the engineers\n # will know about it immediately — and fix it.\n self.aws.terminate_instance(victim.id)\n print(f\"[Chaos Monkey] Terminated {victim.id} \"\n f\"in cluster {cluster.name}\")\n \n # Wait before running again — mean time between terminations\n # configured per cluster, not random probability\n time.sleep(self.config.termination_interval_seconds)\n\n\nFAILURE INJECTION TESTING (FIT): THE EVOLUTION
\nIn 2014, Netflix engineers (including Kolton Andrus, who later co-founded Gremlin) introduced FIT — Failure Injection Testing. Where Chaos Monkey operated at the infrastructure level (kill an EC2 instance), FIT operated at the application level: injecting failure metadata through Zuul (Netflix's edge proxy that handles all requests from devices and applications to Netflix's backend services) to simulate specific service failures with surgical precision. FIT could say 'for this specific user's request, pretend the recommendations service is timing out' without actually degrading the recommendations service for everyone. This precision made chaos experiments far more targeted and safer to run continuously.
\n
\n\n\nℹ️
\nChaos Monkey 2.0: Open-Sourced and Rebuilt in Go
Chaos Monkey was open-sourced in 2012 and rebuilt in 2016 as version 2.0. The new version was written in Go, used Spinnaker as its deployment platform dependency, and introduced mean-time-between-terminations (rather than probabilistic scheduling) for more predictable test coverage. Version 2.0 also added Trackers — Go language objects that report instance terminations to external monitoring systems, enabling downstream correlation of Chaos Monkey events with application metrics and alerts.
\n
\n\n\n✅
\nIndustry Adoption: From Netflix to Everywhere
By 2015, Netflix's Chaos Engineering practices had been codified in the Principles of Chaos Engineering document (published by a team including Casey Rosenthal, who led Netflix's Chaos Engineering team), transforming what had been an internal Netflix tool into a formal engineering discipline. Companies including LinkedIn, Facebook, Google, Amazon, and Twilio adopted chaos engineering practices. Kolton Andrus (from Netflix's FIT team) founded Gremlin in 2016 to commercialize chaos engineering tooling. AWS launched its own Fault Injection Simulator service in 2021.
\n
\n\n\nℹ️
\nThe Open-Source Release and Industry Spread
Netflix open-sourced Chaos Monkey in 2012, making the tool available to any engineering team that wanted to adopt the practice. The release did something more important than provide the code: it legitimized the approach. Engineering teams at other companies who had been quietly running similar experiments could now point to Netflix's published methodology as industry precedent. By 2015, companies including LinkedIn, Facebook, Google, Amazon, and Twilio had publicly acknowledged chaos engineering practices. The 2015 publication of the Principles of Chaos Engineering by Netflix's Casey Rosenthal and colleagues formalized the discipline with scientific language: hypothesis, experiment, steady state, blast radius. What had been a Netflix internal tool became a named engineering discipline.
\n
\n\n\nTHE SPINNAKER DEPENDENCY IN V2.0
\nChaos Monkey 2.0 (2016) introduced a significant constraint: it requires Spinnaker (Netflix's open-source multi-cloud continuous delivery platform that manages application deployments across AWS, Azure, Kubernetes, and other providers) as its deployment platform. This means that teams wanting to use Chaos Monkey 2.0 must also adopt Spinnaker — a substantial investment. Companies unwilling to commit to Spinnaker found Chaos Monkey 2.0 inaccessible, which opened market space for alternatives like Gremlin (founded by Netflix alumni Kolton Andrus and Matt Fornaciari) that offered chaos engineering as a service without infrastructure prerequisites.
\n
Netflix's architecture in 2011 was organized around a principle that Chaos Monkey enforced: every service must be independently deployable, independently scalable, and independently recoverable. The microservices were connected through REST APIs, with each service maintaining its own data store and exposing a versioned interface to its consumers. Chaos Monkey operated at the AWS EC2 instance layer — the individual virtual machines running each service's processes. When an instance was terminated, the load balancer in front of that service's cluster detected the unhealthy instance and stopped routing traffic to it. If the cluster had been sized with enough redundancy, other instances absorbed the traffic without degradation. If not, the service degraded — and the engineers learned something.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE BEHAVIORAL ECONOMICS OF CHAOS ENGINEERING
\nChaos Monkey's deepest contribution to Netflix's culture was aligning incentives. Without it, the cost of fragile code was paid by whoever happened to be on-call when a real failure occurred — often not the engineer who wrote the fragile code. With Chaos Monkey, the cost was paid immediately and visibly by the team whose service broke. Engineers who experienced a Chaos Monkey failure during business hours had a powerful motivator to invest in proper fault tolerance: they didn't want to experience it again. This is DevOps incentive design at its finest — not policy mandates, but a system where the right behavior is the path of least resistance.
\n
\n\n\nℹ️
\nWhy Business Hours Only — The Safety Constraint
The original Chaos Monkey ran only during business hours, and this was not a limitation — it was the essential design principle. An instance killed at 2am when engineers are asleep creates exactly the scenario Netflix wanted to avoid: unplanned, unattended failure with long MTTD (Mean Time To Detect) and long MTTR (Mean Time To Recover). An instance killed at 2pm on a Tuesday is pedagogical, not adversarial: engineers learn from it, fix the gap, and build better systems. As Netflix's confidence in its architecture grew, chaos experiments expanded to cover more scenarios and broader failure scopes — but the principle of human-attended chaos remained core to responsible chaos engineering practice.
\n
\n\n\n⚠️
\nWhat Chaos Monkey Doesn't Test
Chaos Monkey's instance-termination model is powerful but deliberately narrow. It does not test network partitions (instances visible but unreachable), latency degradation (Latency Monkey's job), data corruption, or slow memory leaks that cause gradual performance degradation over hours. Chaos Monkey's successors in the Simian Army and in tools like Gremlin were created precisely to cover these gaps. The original insight — failing constantly builds resilience — generalizes to all failure types, but the specific mechanism must match the specific failure mode being tested. A chaos engineering program that only kills instances is missing most of the failure surface.
\n
Chaos Monkey is fourteen years old and it has influenced every major engineering organization's approach to reliability. Its lessons are not about the specific tool — they are about the philosophy that the tool embodies and the cultural transformation it requires.
\nWhat to remember
\n\n\n✅
\nThe September 2014 Test That Validated Everything
Netflix's most public validation of Chaos Monkey's philosophy came not from their own experiments but from AWS itself. On September 25, 2014, AWS rebooted approximately 10% of its EC2 instances across regions without warning — a real, unplanned failure event at significant scale. Netflix handled it without customer impact. The years of Chaos Monkey practice had built exactly the muscle memory and architectural robustness required. Engineers didn't panic. Systems didn't cascade. Services degraded gracefully and recovered automatically. This was the experiment Netflix couldn't have designed themselves — and they passed it.
\n
\n\n\nFROM TOOL TO DISCIPLINE: THE PRINCIPLES OF CHAOS ENGINEERING
\nIn 2015, Netflix's Casey Rosenthal formalized Chaos Monkey's philosophy into the Principles of Chaos Engineering — a document that defined chaos engineering with scientific rigor: establish a steady-state hypothesis, vary real-world events, run experiments in production, automate continuously, minimize blast radius. These principles transformed chaos engineering from 'Netflix's thing where they kill their own servers' into a reproducible engineering discipline with clear methodologies. The formalization is what allowed chaos engineering to spread beyond Netflix — teams could now implement the practice without having to rediscover the same principles themselves.
\n
\n\nNetflix built a tool that killed their own servers on purpose every business day for years, and the one time AWS killed 10% of their servers by accident, nobody noticed — which is either the best possible outcome of a chaos engineering program or proof that Netflix engineers have very high stress tolerances.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-18T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.084337+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Chaos Engineering", "Netflix"]}, {"id": "https://techlogstack.com/explore/figma-postgres-horizontal-sharding-2024/", "url": "https://techlogstack.com/explore/figma-postgres-horizontal-sharding-2024/", "title": "Figma's Database Grew 100x in Four Years — Here's How a Small Team Kept It From Toppling", "summary": "How Figma's small databases team horizontally sharded a 100x-grown Postgres stack in 9 months using colos, logical sharding via Postgres views, and a custom Go query", "content_html": "Figma · Databases · 18 May 2026
\nIn 2020, Figma ran on a single Postgres instance on AWS's largest available machine. Four years later, that database had grown nearly 100x. Some tables had swelled to several terabytes and billions of rows. The Postgres vacuum process — the background job that keeps Postgres alive — was causing reliability incidents. They had months of runway left before hitting the IOPS ceiling. A small databases team had nine months to fix it.
\n\nWe needed a bigger lever.
— — Sammy Steele, Tech Lead — Figma Databases Team, via Figma Engineering Blog
Figma's database story follows a pattern familiar to every fast-growing product company, but with stakes that were unusually high and a timeline that was unusually compressed. In 2020, Figma ran on a single Postgres database on AWS's largest available RDS instance. By the end of 2022, the team had done what most scaling playbooks suggest first: add read replicas, add a connection pooler (PgBouncer (a lightweight PostgreSQL connection pooler that sits between application code and the database, multiplexing many application connections down to a smaller pool of real database connections — reducing connection overhead significantly)), and vertically partition (splitting a single database into multiple smaller databases, each containing a logical group of related tables — for example, one database for Figma files data, another for organization data) the database into a dozen domain-specific shards. These steps bought them runway. They did not buy them enough runway.
\nThe data was unambiguous. Certain tables — the ones tracking Figma files, user activity, and collaboration state — were growing at rates that would soon exceed what Amazon RDS could support in IOPS. Some of these tables already contained several terabytes and billions of rows. At that size, Postgres's vacuum process (a critical background maintenance operation in Postgres that reclaims storage from deleted rows and prevents the database from running out of 32-bit transaction IDs — if vacuuming falls behind, it can cause severe performance degradation and, in extreme cases, force the database offline) was beginning to cause reliability incidents — it was falling behind on the largest tables, unable to reclaim space fast enough to keep up with write volume. Vertical partitioning couldn't fix this: the smallest unit of vertical partitioning is a single table, and these individual tables were the problem.
\n\n\n\nTHE VACUUM PROBLEM AT SCALE
\nPostgres must periodically vacuum tables to reclaim space from deleted and updated rows. This is not optional — if a table accumulates too many dead tuples, query performance degrades severely. On tables with billions of rows and high write rates, the vacuum process can fall behind the rate of new writes. When this happens, the database starts showing reliability symptoms: bloated tables, degraded query plans, and in extreme cases the risk of transaction ID wraparound — a catastrophic condition that forces Postgres into read-only emergency mode. Figma was seeing the early signs of this at scale.
\n
Figma's databases team evaluated every obvious alternative before committing to building their own horizontal sharding layer. CockroachDB, TiDB, Google Spanner, and Vitess were all on the list. All were rejected for the same core reason: switching to any of them would have required a complex data migration across two different database stores simultaneously. With only months of runway remaining before hitting critical IOPS limits, a migration to an unfamiliar storage layer under deadline pressure was a risk the team couldn't accept. They had also accumulated significant operational expertise running RDS Postgres. That expertise would have to be rebuilt from scratch for any new system. The team instead chose to build horizontal sharding on top of their existing RDS Postgres infrastructure — not a generic solution, but one scoped precisely to Figma's data model and access patterns.
\n\nBy late 2022, Figma's largest tables had grown to several terabytes with billions of rows, and the Postgres vacuum process was causing reliability incidents on the highest-write tables. Projections showed the team would exceed RDS maximum IOPS within months. Vertical partitioning — splitting databases by domain — could not help because individual tables were the bottleneck, not cross-domain coupling.
\nHorizontal sharding (splitting a single large table's rows across multiple physical database instances based on a shard key — allowing any individual table to grow beyond the limits of a single machine) was the only viable path. But implementing it on a complex relational data model, with hundreds of engineers writing queries, required solving three hard problems simultaneously: routing queries correctly, maintaining developer productivity, and enabling rollback if something went wrong.
\nThe team invented three interlocking abstractions: 'colos' (colocation groups of related tables sharing a shard key), logical sharding via Postgres views (which allowed safe percentage-based rollout without moving any data), and DBProxy (a custom Go query proxy with an AST parser that routed queries to the correct physical shard). Together these allowed incremental, reversible rollout of horizontal sharding without disrupting product development.
\nThe migration completed in nine months with zero downtime and the ability to roll back at any step. Future shard splits at the physical level are now transparent to application developers — after the initial upfront work to make a table compatible with horizontal sharding, all subsequent scale-outs happen in the infrastructure layer without any product team involvement.
\n\n\n\n🪄
\nThe most elegant part of Figma's sharding approach was using standard Postgres views to implement logical sharding. A view like
\nCREATE VIEW table_shard1 AS SELECT * FROM table WHERE hash(shard_key) BETWEEN min AND maxlets Postgres behave as if data is already sharded — without any data moving. This made the logical sharding phase essentially free to roll back: change the view definition, flip the config, done.
\n\n\nℹ️
\nThe Shadow Planning Framework
Before building DBProxy's query engine, the team needed to know which queries to support. They built a shadow planning framework that let engineers define potential sharding schemes for their tables, then ran those plans against live production traffic — logging the queries and plans to Snowflake for offline analysis. This gave them empirical data to design a query language covering the most common 90% of queries while deliberately excluding the rare worst-case patterns that would have made DBProxy impossibly complex.
\n
The constraints the team placed on their query language were deliberate and principled. All range scan and point queries were supported. Cross-table joins were only allowed when both tables belonged to the same colo and the join was on the sharding key. Scatter-gather queries — those that must fan out to all shards because they lack a shard key — were supported but their use was actively discouraged because each scatter-gather effectively multiplies database load by the shard count. Application developers were encouraged to refactor scatter-gather access patterns before sharding their tables, using the shadow planning data to understand which of their queries fell into this category.
\n\n\n\n⚠️
\nThe Five Goals They Refused to Compromise
Figma's team defined five non-negotiables before writing a line of sharding code: minimize developer impact (product engineers shouldn't need to rewrite queries), scale-out transparency (future shard splits invisible to application layer), no expensive backfills (no solution requiring moving terabytes before going live), incremental progress (percentage-based rollout at every step), and rollback at any stage — even after physical sharding. Every architectural decision was measured against these five goals.
\n
\n\n\nℹ️
\nThe Postgres Vacuum Threat Nobody Talks About
Postgres uses a 32-bit transaction counter. Every write increments it. If the database ever gets close to the maximum 2^31 transactions without vacuuming reclaimed space, Postgres enters a read-only emergency mode called transaction ID wraparound — a database-wide shutdown to prevent data corruption. On tables with billions of rows and heavy write rates, falling behind on vacuuming is not a theoretical risk. Figma was experiencing real reliability incidents from vacuum lag on their largest tables. This was the alarm that confirmed horizontal sharding was urgent, not optional.
\n
Figma's horizontal sharding solution had three distinct architectural components that worked together. Colos (colocations) were the conceptual layer — groups of related tables that shared the same sharding key and physical shard layout. Tables within a colo could be joined and queried transactionally as long as the join was on the sharding key. The sharding keys were chosen from a small set: user_id, file_id, or org_id — most tables at Figma could be naturally associated with one of these. Logical sharding was the rollout layer — using Postgres views to simulate sharding behavior without moving any data. DBProxy was the execution layer — intercepting queries, parsing them into an AST, determining which logical shard the query targeted, and routing it to the appropriate physical database.
-- Logical sharding via Postgres views: the key insight\n-- No data moves during logical sharding phase.\n-- Tables behave as if sharded — just views on the same physical table.\n\n-- Single physical table still holds all data:\n-- CREATE TABLE figma_files (file_id UUID, org_id UUID, data JSONB, ...)\n\n-- Logical shards created as views filtered by hash range:\nCREATE VIEW figma_files_shard_0 AS\n SELECT * FROM figma_files\n WHERE hashtext(file_id::text) % 4 = 0;\n\nCREATE VIEW figma_files_shard_1 AS\n SELECT * FROM figma_files\n WHERE hashtext(file_id::text) % 4 = 1;\n\n-- Views accept both reads AND writes in Postgres:\n-- INSERT INTO figma_files_shard_0 (file_id, data) VALUES (...);\n-- → Postgres routes to the underlying table\n-- → DBProxy validates the shard key is in the correct range\n\n-- Physical sharding later:\n-- Data is ACTUALLY moved to separate RDS instances per shard\n-- DBProxy routing stays the same — application code unchanged\n-- Rollback: re-point physical shard back to original instance\n\n\nDBPROXY: THE QUERY ENGINE
\nDBProxy is a Go service sitting between the application layer and PgBouncer. Its query engine has three components: a query parser that transforms SQL into an AST, a logical planner that extracts query type and shard IDs from the AST, and a physical planner that maps logical shard IDs to physical database instances and rewrites queries accordingly. DBProxy also handles scatter-gather queries (fanning out to all shards and aggregating results), dynamic load-shedding, improved observability, and database topology management. Building it took months — but it was the only way to make sharding transparent to application developers.
\n
\n\n\nℹ️
\nLogical Before Physical: The Two-Phase Rollout
Figma's key migration insight was separating logical sharding from physical sharding. Logical sharding (Phase 1) makes the application behave as if tables are sharded — using views, updating DBProxy config — but all data still lives in one physical database. This can be rolled out as a percentage-based config change, validated against production traffic, and rolled back instantly. Physical sharding (Phase 2) actually moves data to separate RDS instances. Much higher risk — but by this point, the logical layer has been running in production for weeks, bugs are fixed, and the team has empirical confidence in the sharding correctness.
\n
\n\n\n✅
\nThe Rollback Guarantee: Even After Physical Sharding
Most horizontal sharding implementations are one-way migrations — once data is on separate physical instances, rolling back requires a complex reverse migration. Figma's team designed their system so that physical shard splits are reversible. They maintained the ability to point physical shards back to the original database instance while the new routing logic was validated. This reduced the risk of being stuck in a bad state when unknown unknowns inevitably occurred.
\n
Figma's Three-Phase Database Scaling Journey: Before and After
| Phase | Architecture | Bottleneck Addressed | Runway Gained |
|---|---|---|---|
| 2020 | Single RDS Postgres instance | Initial growth | Moderate |
| 2021–2022 | Vertical partitioning (12 domain DBs) + read replicas + PgBouncer | CPU, read load, connection pool | ~1 year |
| 2023–2024 | Horizontal sharding via colos + DBProxy + logical/physical migration | Table-level IOPS ceiling, vacuum backlog, billions-of-row tables | Near-infinite scalability |
\n\n\n🔀
\nScatter-Gather: The Necessary Evil
Some queries don't have a shard key — a query like 'get all recently modified files for an admin dashboard' has no natural file_id scope. DBProxy handles these with scatter-gather: fan the query out to every shard in parallel, collect results, merge and sort. It works correctly but is expensive. Figma's engineering team was explicit with product engineers about the scatter-gather tax, encouraging them to refactor access patterns before their tables were sharded. The shadow planning data showed exactly which queries would become scatter-gather — engineering teams had weeks to fix them before cutover.
\n
The before-state of Figma's architecture had application services talking directly to PgBouncer, which connected to RDS Postgres. Vertical partitioning meant multiple databases, but each database was still a single physical instance — and the largest individual tables still had no mechanism to distribute their rows across instances. DBProxy was inserted between the application and PgBouncer layers, adding the query parsing and routing intelligence that made horizontal sharding possible without requiring application code changes.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nCOLOS: THE DEVELOPER-FACING ABSTRACTION
\nThe colo concept is what made horizontal sharding usable for product engineers. A colo is a named group of tables that share a sharding key — for example, the
\nfiles_colocontainsfigma_files,file_nodes,file_comments, and other tables all sharded byfile_id. Within a colo, cross-table joins and full transactions are supported when restricted to a single shard key value. This matches how Figma's application code already accessed the database — most operations concerned a single file or a single user, not cross-colo data. The colo abstraction minimized the number of queries that needed to be refactored.
\n\n\n⚠️
\nThe Scatter-Gather Tax
Queries without a shard key — those that need results from all shards — are handled by DBProxy's scatter-gather mechanism: the query is fanned out to all shards in parallel and results are merged. Scatter-gather is correct but expensive: it multiplies read load by the number of shards. Having too many scatter-gather queries would defeat the purpose of horizontal sharding. The shadow planning framework specifically identified scatter-gather patterns in the production query log before sharding, allowing teams to refactor the most frequent offenders before their tables were migrated.
\n
\n\n\n✅
\nDBProxy: Six Months to Build, Indefinite Value
Building DBProxy — the Go service with an AST parser, logical planner, and physical planner — was the highest-risk engineering bet in the sharding project. It took months to build and required solving problems that existing tools had already solved in different ways. But the payoff was precise control: DBProxy understands Figma's specific query patterns, supports exactly the subset of SQL that Figma uses, and can be extended as Figma's needs evolve. A generic proxy would have required adapting Figma's code to its limitations. DBProxy was adapted to Figma's code.
\n
Figma's sharding story is widely cited because it did something genuinely hard — horizontally sharding a complex relational production database under deadline pressure — and documented the architecture decisions clearly enough for other teams to learn from. The lessons are about sequencing, abstraction, and the courage to build something custom when existing tools genuinely don't fit.
\nWhat to remember
\n\n\n✅
\nThe Post-Migration State: Scale Without Change
After the initial work to make a table horizontal-sharding-compatible, all future shard splits happen transparently. As a table grows again toward limits, the infrastructure team can split shards — updating the physical topology and DBProxy's routing config — without any product engineer touching their code. This is the payoff of the upfront investment: the database can now scale indefinitely at the infrastructure layer, decoupled from the application layer.
\n
\n\n\nWHEN MONTHS OF RUNWAY MEANS NOW
\nFigma's team framed their problem as 'months of runway remaining' — meaning that if they did nothing, they would hit a hard scaling ceiling and likely experience database reliability incidents or outages within months. This framing was not catastrophizing; it was the math of their growth rate applied to their IOPS limit. The urgency drove the decision to build custom rather than migrate to an unfamiliar system. Teams facing similar trajectory should run this calculation early — months of runway sounds like plenty of time until the migration itself takes several months.
\n
\n\nFigma's database grew 100x and a small team fixed it in nine months — which is either very good database engineering or very good use of Postgres views depending on who you ask, and the answer is both.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-18T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.791800+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Databases", "Figma"]}, {"id": "https://techlogstack.com/explore/datadog-systemd-outage-graceful-degradation-2023/", "url": "https://techlogstack.com/explore/datadog-systemd-outage-graceful-degradation-2023/", "title": "Datadog Went Dark for 24 Hours and Came Back With a Different Philosophy", "summary": "How an unsupervised systemd update took down 50–60% of Datadog's Kubernetes nodes globally, cost $5M, and drove a company-wide shift to graceful degradation architec", "content_html": "Datadog · Reliability · 18 May 2026
\nOn March 8, 2023, Datadog — the platform engineers use to know when their own infrastructure is broken — broke. For more than 24 hours, across five regions on three cloud providers, metrics stopped arriving, logs disappeared, and dashboards showed nothing. The people whose job was to fix it couldn't see what was happening. It cost $5 million. It changed how Datadog thinks about building software.
\n\nWe had built with the assumption that the only way to handle failure was to prevent it entirely — or to stop everything — rather than finding ways to degrade gracefully and continue delivering value to customers, even under extreme conditions.
— — Laura de Vesine, Rob Thomas, Maciej Kowalewski — via Datadog Engineering Blog
At 01:31 EST on March 8, 2023, Datadog experienced its first global outage — every region, every cloud provider, simultaneously. The company that monitors the infrastructure of thousands of other companies could not monitor its own. Dashboards loaded but displayed no data. Logs, metrics, alerting, and traces were all unavailable. The engineers whose job was to diagnose and fix the outage were operating without the observability tools that Datadog itself provides. It lasted over 24 hours. It cost $5 million in direct revenue. And it forced a fundamental rethink of how Datadog builds reliable systems.
\nThe immediate cause was disarmingly mundane: an automated systemd (the init system and service manager used by most modern Linux distributions — it starts processes, manages services, and handles system initialization) update was applied to Datadog's Ubuntu-based virtual machines across all regions simultaneously. This was a legacy security patch mechanism — Datadog had since built a modern lifecycle automation system for all nodes — but the legacy channel was still active and executed its update across the global fleet without any staged rollout, any health gates, or any human awareness. The update caused a systemd-networkd (the systemd component responsible for managing network interfaces on Linux hosts) restart interaction that removed network routes from the machines as they came back up. Nodes that had previously been connected to each other's network simply vanished from the cluster.
\n\n\n\nTHE CIRCULAR DEPENDENCY TRAP
\nThe worst part was not that 50–60% of Kubernetes nodes lost network connectivity — it was what those nodes were running. Among the VMs brought down by the network route removal were the VMs powering Datadog's regionalized control planes based on Cilium (a cloud-native networking platform for Kubernetes that uses eBPF to provide networking, security, and observability for containerized workloads). The control plane going down meant Kubernetes couldn't schedule new pods, auto-repair failed nodes, or scale workloads to compensate. The very system that should have responded to the failure was among the first things the failure took down. This circular dependency — the recovery mechanism depending on the infrastructure that failed — is what turned a 50% node loss into a nearly complete platform outage.
\n
A legacy automated Ubuntu security update channel applied a systemd update across Datadog's entire global fleet simultaneously — all five regions, all three cloud providers, all at once. The update caused a systemd-networkd restart interaction that removed network routing tables from nodes as they restarted. 50–60% of Kubernetes nodes lost network connectivity within minutes. Pages loaded but displayed no data. The outage was total from the customer perspective.
\nThe Kubernetes control plane — the cluster management layer responsible for scheduling, auto-repair, and scaling — was among the nodes that lost connectivity. This created a circular dependency: the recovery system needed the cluster to heal, but the cluster could not heal without the recovery system. Additionally, Datadog's multi-region, multi-cloud architecture provided no protection because the update was applied uniformly across all infrastructure simultaneously.
\nRecovery required manual intervention: engineers identified and restarted affected nodes, restoring network routing and bringing Kubernetes control planes back online region by region. The legacy update channel was immediately disabled. But recovery took over 24 hours — far longer than the node loss itself — because services loaded large in-memory caches on startup that were slow to initialize, and the cluster lacked the spare capacity to absorb the sudden recovery surge.
\nFull service restoration after 24+ hours. In the months following, Datadog published a detailed engineering blog describing not just what happened but the architectural shift it drove: away from never-fail systems toward systems designed to degrade gracefully when failure inevitably occurs. Published October 2025, the blog documented two years of architectural work as a direct result of the March 2023 incident.
\n\n\n\n💸
\nDatadog operates on usage-based billing — customers pay for the volume of metrics, logs, and traces they send. During the 24-hour outage, Datadog did not charge customers for data they couldn't send. The $5M revenue loss was direct: one day of global service unavailability translated directly into one day of foregone billing. This number was revealed on an earnings call, making the financial cost of the outage unusually concrete and public.
\n
\n\n\n❌
\nMulti-Cloud Did Not Help
Datadog ran in five regions across three cloud providers — AWS, GCP, and Azure. This architecture is often cited as a reliability best practice. But it provided zero protection in this incident because the failure mechanism — the automated Ubuntu update — operated at the OS layer, uniformly across all infrastructure regardless of cloud provider. Multi-cloud protects against cloud provider failures. It does not protect against failures in your own automation that touch all infrastructure simultaneously.
\n
The 24-hour recovery time was itself a lesson. Even after the Kubernetes control planes came back online and new pods could be scheduled, services were slow to recover. The investigation found two patterns: some services had insufficient compute allocated relative to others, causing them to wait a long time for Kubernetes to schedule their pods after the control plane recovered. Others loaded large, processing-intensive caches into memory at startup — caches that had been optimized for steady-state operation but were extremely expensive to rebuild from scratch after a complete restart. Both of these were design choices that had seemed reasonable in a world where failure was rare and total restarts were rarer still. In a world where failure must be expected, they were traps.
\n\n\n\n⚠️
\nThe Irony of the Observability Platform
There is a particular quality of darkness in losing observability tooling during an outage. Engineers responding to the incident were using Datadog to understand what was happening — and Datadog was the thing that was down. The response team had to work from first principles: SSH into individual hosts, read raw logs, check systemd status directly. The tooling built to abstract away that complexity was unavailable at precisely the moment the complexity needed to be navigated. The incident revealed how dependent Datadog's own oncall rotation was on Datadog itself.
\n
\n\n\nℹ️
\nThe Square-Wave Failure Pattern
Datadog's engineers described the outage as a square-wave failure — the platform went from fully operational to nearly completely down almost instantaneously, rather than degrading gradually. This pattern is characteristic of failures at the infrastructure layer: when Kubernetes nodes lose network connectivity, every pod running on those nodes disappears from service meshes and load balancers at once. There is no gradual ramp. For an observability platform designed around monitoring continuous signals, a square-wave drop to zero looked different from every other failure mode the monitoring systems had been trained on.
\n
\n\n\n🌐
\nDatadog ran infrastructure across five regions on three different cloud providers — a setup specifically designed to avoid single points of failure. It provided no protection at all against this incident because the failure mechanism lived at a layer beneath the cloud provider abstraction: the Ubuntu OS update that ran on every Datadog-managed VM, regardless of which cloud it ran on. The lesson is precise: multi-cloud resilience and OS-level automation independence are orthogonal properties.
\n
\n\n\nTHE POSTMORTEM DELAY
\nDatadog waited over two months to publish a public postmortem — a gap that generated significant industry commentary, particularly after the CEO referenced it on an earnings call before it was publicly available. The eventual postmortem was substantive and technical. But the delay — and the CEO's apparent confusion about whether it had been shared — was widely noted as a departure from the transparency standard set by companies like Cloudflare. Speed of postmortem publication matters for customer trust, especially for a platform whose entire value proposition is reliability and observability.
\n
The deep engineering response to the March 2023 outage was not a list of tactical fixes. It was a philosophical shift. Datadog's engineering teams had, historically, built for reliability through redundancy — designing systems so that individual components never went down. This produced what the postmortem called never-fail architectures: systems where components and services had to be fully functional to serve any user use case. When a component did fail, the entire service path that depended on it failed with it. The incident revealed a hidden assumption: that recovery would be fast and partial failure would be brief. A 24-hour outage broke that assumption completely, and exposed how little thought had gone into what the system should do while broken.
\n\n\n\nWHAT GRACEFUL DEGRADATION ACTUALLY MEANS
\nDatadog's post-incident architectural shift was built on a simple principle: when failure occurs, the system should continue to deliver as much value as possible to as many customers as possible, even if it cannot deliver full value to all customers. This means designing every service with an explicit answer to the question: what does this service do when its dependencies are unavailable? Can it serve stale data? Can it serve a subset of features? Can it serve with degraded accuracy? Or does it have to stop entirely? Most services, when the question is asked honestly, can do better than stop.
\n
# Before: Never-fail architecture (implicit assumption)\nclass MetricsQueryService:\n def query_metrics(self, metric_name, time_range):\n # If storage is unavailable, this raises an exception\n # The exception propagates up — user sees an error page\n raw_data = self.storage.fetch(metric_name, time_range)\n return self.process(raw_data) # no fallback\n\n# After: Graceful degradation architecture\nclass MetricsQueryService:\n def query_metrics(self, metric_name, time_range):\n try:\n # Try live storage first\n raw_data = self.storage.fetch(metric_name, time_range)\n return self.process(raw_data)\n except StorageUnavailable:\n # Fall back to cached/stale data — user sees old data with a warning\n stale_data = self.read_through_cache.fetch(metric_name, time_range)\n if stale_data:\n return DataResponse(data=stale_data, staleness_warning=True)\n # Fall further back — return partial data from other sources\n partial = self.fallback_source.fetch(metric_name, time_range)\n if partial:\n return DataResponse(data=partial, completeness_warning=True)\n # Only now surface an error — and make it informative\n return DataResponse(error='Storage degraded', retry_in=30)\n\n\nℹ️
\nStartup Optimization: Fixing the Recovery Drag
Two changes addressed the slow recovery after node restoration. First, Datadog used Kubernetes priority mechanisms to ensure critical services got compute allocated before lower-priority ones when the cluster came back online — preventing a thundering herd of equal-priority pods all waiting for the same scarce resources. Second, services with large startup caches shortened their lookback windows and changed data formats to eliminate processing-intensive deserialization at startup. Services that had been trying to rebuild six months of cache at startup were redesigned to start with a smaller warm window and build up over time.
\n
\n\n\n✅
\nThe Architectural Patterns That Emerged
Over the two years following the incident, Datadog published a set of graceful degradation patterns applied across its products: persist data early (write to durable storage as early as possible in the pipeline, so recovery is stateless); stale reads (serve cached data with a staleness indicator rather than surfacing an error); partial serving (return what you have rather than nothing); circuit breaking (automatically stop calling a failing dependency, fall back to alternative, re-probe for recovery). None of these patterns were invented by Datadog — they were standard resilience engineering techniques that Datadog had systematically under-applied.
\n
\n\n\n✅
\nPersist Data Early: The Durability Pattern
One of the most concrete architectural changes after the incident was implementing a persist early pattern across Datadog's data pipelines. Instead of holding data in-memory for processing before writing to durable storage, the system was changed to write to durable storage as soon as data arrived — before processing. This meant that even if processing services went down, incoming customer telemetry was safely on disk and could be processed retroactively when services recovered. Recovery no longer required customers to resend data that had arrived during the outage window.
\n
\n\n\n⚠️
\nThe Kubernetes Priority Class Oversight
After the outage, Datadog's investigation found that many services had not been assigned appropriate Kubernetes Priority Classes — a mechanism that tells the Kubernetes scheduler which pods should get compute resources first when the cluster is under resource pressure. In normal operation, this doesn't matter much. After a large failure where the entire cluster restarts simultaneously, priority classes determine recovery order. Services that should start first (database proxies, ingestion pipelines) were waiting for the same CPU allocations as low-priority background jobs. Recovery order is a design decision that should be made explicitly, not left to scheduler defaults.
\n
The architecture that failed in March 2023 had a specific shape: every product feature in Datadog's platform depended on a chain of services, each of which had to be fully healthy for any part of the chain to work. Logs required a log ingestion pipeline, a storage layer, a query layer, and a frontend — all healthy. If any component in the chain was down, the entire feature was down. The never-fail architecture assumed each link in the chain would always be up. The March 2023 incident showed what happens when multiple links go down simultaneously.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nMULTI-CLOUD IS NOT A RELIABILITY SILVER BULLET
\nDatadog's global, multi-cloud infrastructure — five regions, three cloud providers — provided zero protection against this incident. The lesson generalizes: multi-cloud protects against cloud provider failures. It does not protect against failures in your own configuration management, your own automation, your own deployment systems, or your own service design. An automated update that runs across all infrastructure uniformly bypasses all multi-cloud redundancy. Organizations that invest heavily in multi-cloud while neglecting the uniformity of their own automation are addressing the wrong failure vector.
\n
\n\n\n⚠️
\nThe Legacy Channel Problem
The update that caused the outage went through a legacy security update mechanism — a channel that Datadog's security team had kept active while building a modern replacement. The modern system had been built; the legacy system had not been decommissioned. This is one of the most common failure patterns in infrastructure: a replaced system that was never actually turned off. The old system executed one last time at the worst possible moment. Every team with legacy automation that still runs in production should audit whether it could execute in a way that bypasses the modern system's safety gates.
\n
\n\n\n🔬
\nRoot Cause Archaeology: Finding the Network Route Bug
The technical root cause was subtle: when systemd-networkd restarted during the OS update, it cleared the network routing table for container workloads that had been set up by Kubernetes's networking plugin (Cilium). New nodes starting up for the first time don't have this problem — they start with an empty routing table and Cilium populates it correctly. But nodes that were already running had existing routing entries that were erased by the systemd-networkd restart. This was a previously unobserved interaction that only manifested when restarting a running node rather than provisioning a new one.
\n
The March 2023 Datadog outage is extraordinary for two reasons: the irony of an observability platform going dark, and the depth of the architectural response it drove. The lessons here are not primarily about the incident itself but about the philosophy that emerged from it.
\nWhat to remember
\n\n\nℹ️
\nTwo Years to Publish the Engineering Post
The March 2023 outage happened in March 2023. The detailed engineering blog documenting the architectural response was published in October 2025 — two and a half years later. This timeline reflects the depth of the work: the blog described real architectural changes that had been implemented and validated in production across Datadog's entire product portfolio, not aspirational plans. Publishing only after the work was done is the responsible version of transparency — claiming to have fixed something before you've fixed it erodes trust faster than silence.
\n
\n\n\nGRACEFUL DEGRADATION AS A DESIGN PRINCIPLE, NOT A FEATURE
\nThe deepest lesson from Datadog's post-incident work is that graceful degradation is not a feature you add to a service after it's built — it's a design principle that shapes how the service is architected from the beginning. A service designed to gracefully degrade will have different internal boundaries, different cache strategies, different dependency contracts, and different SLOs than one designed to always succeed. Retrofitting graceful degradation into a never-fail architecture is expensive. Building for it from the start is cheaper. After two years of retrofitting, Datadog's engineering organization now treats the question 'how does this service degrade?' as a required design review criterion.
\n
\n\nDatadog's monitoring platform went down for 24 hours — which means the engineers had to debug a global infrastructure failure using SSH, intuition, and the kind of raw log reading skills that got them into engineering in the first place.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-18T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.876392+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "Datadog"]}, {"id": "https://techlogstack.com/explore/openai-postgresql-scaling-2026/", "url": "https://techlogstack.com/explore/openai-postgresql-scaling-2026/", "title": "OpenAI Runs ChatGPT for 800 Million Users on One PostgreSQL Instance — and It Works", "summary": "How OpenAI scales ChatGPT's PostgreSQL database to 800 million users with a single primary instance, 50 read replicas, connection pooling, and ruthless query discipl", "content_html": "OpenAI · Databases · 18 May 2026
\nChatGPT has 800 million users. It handles millions of database queries per second. And it runs on a single primary PostgreSQL instance on Azure — one writer, backed by about fifty read replicas. No sharding. No distributed SQL. Just Postgres, pushed further than almost anyone thought possible through obsessive optimization and ruthless operational discipline.
\nThe conventional wisdom about database scaling at 800 million users is straightforward: you shard. You move to a distributed SQL system. You decompose into microservices each with their own database. You do not run a single primary PostgreSQL instance. OpenAI's ChatGPT does not follow this conventional wisdom. It runs on one Azure PostgreSQL Flexible Server that handles all writes — backed by approximately 50 read replicas spread across multiple regions. The system handles millions of queries per second at low double-digit millisecond p99 latency and has maintained five-nines availability. In twelve months, they had one SEV-0. The story is not that Postgres is magic. The story is that relentless optimization of a boring, proven technology can outperform premature architectural complexity.
\n\n\n\nWHY SINGLE-PRIMARY WORKS AT THIS SCALE
\nChatGPT's workload is overwhelmingly read-heavy. When 800 million users open the app, browse their chat history, or load their settings, those are reads. Writes happen on message submission and account updates — a much smaller fraction of the total traffic. This access pattern is exactly what a single-primary with many read replicas handles well: the write path stays narrow, the read load fans out horizontally across replicas. The architecture is not brilliant. It is appropriate for the workload. That fit is what makes it work.
\n
OpenAI's blog published at PGConf.dev 2025 was unusually candid about both the decisions that worked and the ones that nearly broke the system. The database load grew by more than 10x in a single year following ChatGPT's viral growth. The team responded with aggressive optimization at every layer: connection management, query design, caching, write path discipline, and schema change governance. Each of these deserves examination — not because the techniques are novel, but because executing all of them simultaneously, under extreme growth pressure, with production at risk, is far harder than any one technique in isolation.
\n\n\n\n🔌
\nOpenAI's Azure PostgreSQL Flexible Server has a maximum of 5,000 concurrent connections. At ChatGPT's scale, application servers would easily exhaust this limit without connection pooling. Before deploying PgBouncer (a lightweight connection pooler for PostgreSQL that multiplexes many application connections into a smaller pool of real database connections, dramatically reducing connection overhead), average connection time was 50ms. After deployment in statement-pooling mode: 5ms. A 10x improvement from one infrastructure change.
\n
ChatGPT's viral growth — 100 million users in two months at launch, 800 million by 2025 — drove database load up more than 10x in a single year. Connection exhaustion became a recurring threat. A 12-table ORM-generated join was causing multiple high-severity incidents when traffic spiked. Write pressure on the single primary was approaching dangerous levels during high-demand events.
\nORMs ORM (Object-Relational Mapping — a framework layer like Django or SQLAlchemy that automatically generates SQL from application code, abstracting away the database — convenient but capable of generating complex, inefficient queries that are invisible until they cause production incidents) generate SQL automatically, hiding complexity from developers. Under low load, even a 12-table join is fast enough to not notice. Under 10x load, the same query saturates database CPU. Meanwhile, write-heavy workloads that could be migrated to sharded systems like Azure Cosmos DB remained on the single primary longer than optimal.
\nOpenAI implemented PgBouncer connection pooling (cutting connect time 10x), a cache-locking mechanism to prevent thundering herd on cache misses, multi-layer rate limiting at application, proxy, and query levels, surgical elimination of the worst ORM-generated queries, strict schema change governance (5-second DDL timeout), and a policy of migrating all new write-heavy workloads to sharded systems by default.
\nOne SEV-0 in twelve months — triggered by the viral launch of ChatGPT ImageGen, which caused a 10x write surge as over 100 million users signed up within a week. Postgres recovered by design. p99 latency held at low double-digit milliseconds. The single-primary architecture remained viable at a scale that surprised the entire database engineering community.
\n\n\n\n⚠️
\nThe ORM Query That Caused Multiple SEV-0s
OpenAI's engineers discovered that a single ORM-generated SQL query was joining 12 tables. Under normal load, the query executed in acceptable time. Under traffic spikes, it saturated the primary database's CPU and caused multiple high-severity incidents. The query had been auto-generated by the ORM framework and never explicitly reviewed. ORMs are excellent for developer productivity and terrible for query performance visibility. OpenAI now requires that all ORM-generated queries against high-traffic tables be reviewed and analyzed with EXPLAIN ANALYZE before deployment.
\n
OpenAI's schema change governance is one of the most operationally distinctive aspects of their Postgres setup. They enforce a strict rule: schema changes that trigger a full table rewrite are prohibited in production. Postgres's MVCC (Multi-Version Concurrency Control — Postgres's mechanism for allowing readers and writers to operate concurrently without blocking each other, at the cost of retaining multiple versions of each row and requiring periodic vacuum to reclaim space) model means that operations like ALTER TABLE ADD COLUMN DEFAULT on large tables can hold an exclusive lock for hours while rewriting billions of rows. This would be catastrophic at ChatGPT's scale. All DDL operations have a 5-second timeout: if the schema change cannot acquire a lock within 5 seconds, it is cancelled automatically. Long-running queries that would block vacuum or DDL are automatically terminated.
\n\n\nℹ️
\nThe Hot Standby in High-Availability Mode
OpenAI runs the primary database in High-Availability mode with a hot standby — a continuously synchronized replica specifically designated as the failover target. If the primary goes down, the hot standby can be promoted to primary with ~30–60 seconds of downtime. During a primary failure, read traffic on replicas is unaffected — since most ChatGPT requests are reads, a primary failure is not a SEV-0 (because reads remain available). Writes fail until promotion completes. This asymmetry between read and write availability is a conscious architectural tradeoff: the 800 million users who are just browsing conversation history continue being served.
\n
\n\n\nℹ️
\nWhy Not Shard? The Honest Answer
The engineering question 'why didn't OpenAI shard PostgreSQL?' has a straightforward answer: sharding is expensive and their workload didn't require it yet. Horizontal sharding introduces cross-shard transaction complexity, scatter-gather query patterns, operational overhead of multiple database instances, and application-layer awareness of shard routing. For a read-heavy workload that can be served from replicas, these costs are not justified. OpenAI chose to pay the operational cost of extreme Postgres optimization rather than the architectural cost of sharding — and the math worked out. The 'no new tables' policy ensures this calculation will be revisited for write-heavy workloads as they emerge.
\n
\n\n\nIDLE TRANSACTION TIMEOUTS: THE QUIET KILLER
\nOpenAI identified a subtle but devastating Postgres pattern at scale: idle transactions. When application code opens a database connection, starts a transaction, does unrelated work (calling an external API, waiting for user input), and only then commits — the transaction holds locks for the entire duration. At ChatGPT's scale, applications that hold open transactions for seconds can block vacuum, block DDL, and degrade query performance for all other connections. OpenAI enforces strict idle_in_transaction_session_timeout settings — any connection idle inside a transaction for more than a few seconds is automatically terminated. This breaks poorly-written code immediately in staging rather than causing incidents in production.
\n
\n\n\n📊
\nDespite having ~50 read replicas across multiple geographic regions, OpenAI reports near-zero replication lag on most replicas under normal conditions. This is achieved by co-locating PgBouncer, application servers, and replicas in the same region (minimizing network latency in the replication path) and by keeping primary write load within the replication throughput capacity of the replicas. Heavy write events — like the ImageGen launch surge — temporarily increase replication lag, which is why read-your-own-write operations are always routed to the primary.
\n
\n\n\n⚠️
\nThe Write Ceiling Is Real
OpenAI's single-primary architecture has an acknowledged limit: write-heavy events can overwhelm it. The ImageGen SEV-0 was caused by a write surge, not a read surge. The architecture is not defended against arbitrary write load — it is defended against the current write profile, which remains manageable because most new write-heavy workloads are being routed to Cosmos DB. If write load grows faster than the migration effort proceeds, the single-primary architecture will face a harder ceiling. The 'no new tables in Postgres' policy is the operational discipline that buys time.
\n
OpenAI's Postgres scaling is not one clever trick — it is seven mutually reinforcing operational practices applied simultaneously. Any one of them in isolation would help marginally. Together they have produced an architecture that handles a scale that its underlying technology was not originally designed for. The practices are: connection pooling, thundering herd prevention, multi-layer rate limiting, hot standby failover, write offloading, query surgery, and DDL governance. Each addresses a specific failure mode that appeared as ChatGPT grew.
\n# Cache-locking pattern: prevents thundering herd on cache misses\n# When cache expires, only ONE request repopulates it — others wait\n\nimport threading\n\n# Simplified cache-lock implementation\n_cache = {}\n_locks = {}\n_lock_mutex = threading.Lock()\n\ndef get_with_cache_lock(key: str, fetch_fn, ttl_seconds: int):\n \"\"\"Get value from cache. On miss, only one thread fetches;\n others block and receive the result once available.\"\"\"\n \n # Fast path: cache hit\n if key in _cache:\n return _cache[key]\n \n # Slow path: cache miss — acquire per-key lock\n with _lock_mutex:\n if key not in _locks:\n _locks[key] = threading.Event()\n should_fetch = True\n else:\n should_fetch = False\n event = _locks[key]\n \n if should_fetch:\n try:\n # This thread does the database read\n value = fetch_fn() # ONE database query\n _cache[key] = value # populate cache\n return value\n finally:\n with _lock_mutex:\n event = _locks.pop(key)\n event.set() # wake all waiters\n else:\n # Other threads wait for the fetching thread to complete\n event.wait(timeout=5) # don't wait forever\n return _cache.get(key) # return from cache\n\n\nℹ️
\nThe Cosmos DB Migration Policy
OpenAI's most forward-looking operational decision is a standing policy: no new tables are created in PostgreSQL. All new workloads default to sharded systems — primarily Azure Cosmos DB. Existing write-heavy workloads that can be horizontally partitioned are gradually migrated out. This policy doesn't fix the current architecture; it fixes the future architecture. Over time, the Postgres primary handles a smaller and smaller share of writes while remaining the canonical store for core user and conversation data. The single-primary architecture is not defended forever — it's being gracefully phased toward a hybrid model.
\n
\n\n\nREVIEW ORM-GENERATED SQL IN PRODUCTION
\nOpenAI's most actionable advice: add ORM-generated SQL review to your production deployment process. ORM frameworks are brilliant for development velocity. They are silent performance landmines at scale. A query that joins 12 tables, a query that does a full table scan on an unindexed column, a query that triggers N+1 loads — none of these are visible in code review because the ORM generates them at runtime. OpenAI now requires that SQL generated by ORM frameworks for high-traffic tables be logged, analyzed with EXPLAIN ANALYZE at peak load, and reviewed by a database engineer before the code ships. This practice is cheap. Not having it costs SEV-0s.
\n
\n\n\n✅
\nLazy Writes: Smoothing Write Spikes
OpenAI introduced lazy writes for certain workloads — deferring non-critical writes instead of executing them immediately. For example, updating a user's last-seen timestamp or incrementing a view counter doesn't need to hit the database synchronously with every request. Batching these writes and flushing them periodically smooths write traffic from a spiky real-time pattern to a steadier background pattern. Lazy writes reduced write load on the primary meaningfully without any change to user-visible behavior.
\n
\n\n\n✅
\nCovering Indexes: The Query Surgery Tool
Beyond eliminating bad ORM queries, OpenAI invested heavily in covering indexes — indexes that contain all columns needed by a query, so Postgres can answer it from the index alone without reading table rows. A covering index on a high-frequency query can reduce query cost from a sequential scan of billions of rows to a few hundred index lookups. OpenAI's database engineers regularly audit slow query logs and apply targeted index improvements, particularly after any traffic increase reveals latent query inefficiencies that weren't visible at lower load.
\n
\n\n\nℹ️
\nFeature Traffic Isolation
OpenAI isolates low-priority features from critical traffic paths. If a secondary feature — say, a background data analysis job — starts behaving poorly and consuming database resources, it should not degrade ChatGPT's core conversational experience. This isolation is implemented through separate connection pools for different traffic classes, Kubernetes resource quotas for background workloads, and rate limiting that gives core product queries priority access to database capacity. The principle: a misbehaving low-priority feature should degrade itself, not the entire platform.
\n
OpenAI's Postgres architecture is simple at the macro level — one writer, many readers — but densely engineered at the micro level. The simplicity is intentional: every additional layer of infrastructure complexity is a potential failure mode. The dense engineering at the application and proxy layers is what makes the simple macro architecture viable at unprecedented scale. Understanding why this architecture works requires understanding both its strengths (read-heavy workload perfectly matched to replica fan-out) and its known limits (write spikes, ORM-generated queries, connection exhaustion).
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE REPLICATION LAG TRADEOFF
\nAsynchronous replication to read replicas introduces a tradeoff: reads may return slightly stale data. For most ChatGPT operations — loading conversation history, displaying user settings, browsing the interface — a few hundred milliseconds of staleness is imperceptible and acceptable. For the small fraction of requests that require current data (a write followed immediately by a read-your-own-write pattern), OpenAI routes those reads to the primary. This explicit differentiation between 'reads that can tolerate lag' and 'reads that cannot' is a design discipline, not an accident — and it's what allows the read load to be distributed across 50 replicas.
\n
\n\n\n⚠️
\nThe Backfill Rate Limit: So Slow It Takes a Week
OpenAI enforces strict rate limits on database backfill operations — migrations that populate new columns or update existing rows across large tables. These rate limits are aggressive enough that a large backfill can take over a week to complete. This is deliberate: a fast backfill on a billion-row table would compete with live traffic for I/O, degrade query latency, and risk triggering the DDL timeout. Slow backfills are boring and invisible. Fast backfills cause incidents. OpenAI chose boring.
\n
\n\n\n✅
\nThe Five-Nines Achieved
OpenAI reports achieving 99.999% availability on their Postgres infrastructure — five nines, which means less than 5.26 minutes of downtime per year. This is achieved despite running a single primary, primarily because most customer traffic is read-only (served by replicas even during primary downtime), write failures during primary maintenance are brief (hot standby promotion in 30–60 seconds), and the defensive layers prevent the most common failure modes from escalating. Five nines on a single-primary setup requires more engineering discipline, not less, than achieving the same availability on a distributed system.
\n
OpenAI's PostgreSQL story is the strongest available evidence that conventional wisdom about 'you must shard at scale' is not a law — it's a heuristic that depends heavily on workload shape. The lessons here are about operational discipline, honest workload analysis, and knowing the limits of your architecture before they find you.
\nWhat to remember
\n\n\nℹ️
\nThe ImageGen Launch: The One That Got Through
In twelve months of operation with the fully hardened architecture, OpenAI had one SEV-0: the launch of ChatGPT ImageGen. Over 100 million new users signed up within a week, driving a >10x spike in write traffic — specifically new account creation and preference storage — that temporarily overwhelmed the primary's write capacity. The system recovered by design, but the event validated the 'no new tables in Postgres' policy. Write surges at viral launch scale are the known limit of single-primary architecture. The Cosmos DB migration is the known fix.
\n
\n\n\nTHE HYBRID MIGRATION STRATEGY
\nOpenAI's hybrid approach — keep Postgres for what it does well, migrate write-heavy workloads to Cosmos DB, enforce 'no new tables in Postgres' — is a template for any team running a successful legacy database under growth pressure. The alternative extremes (migrate everything at once, or never migrate anything) are both wrong. Incremental migration guided by workload characteristics is boring, slow, and correct. The discipline is in writing down the policy and enforcing it before the crisis arrives.
\n
\n\nOpenAI runs ChatGPT for 800 million users on one Postgres instance and the most complex part of their database engineering is telling people not to use ORMs without reading the SQL they generate.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-18T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.880910+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Databases", "OpenAI"]}, {"id": "https://techlogstack.com/explore/uber-multicloud-secrets-2025/", "url": "https://techlogstack.com/explore/uber-multicloud-secrets-2025/", "title": "Uber Had 150,000 Secrets Scattered Across 25 Vaults — So They Built One Platform to Rule Them", "summary": "How Uber consolidated 150,000 secrets from 25 fragmented vaults into 6 centrally managed vaults and built an automated Secret Management Platform handling 20,000 rot", "content_html": "Uber · Security · 18 May 2026
\n150,000 secrets. 25 separate vaults. Hundreds of teams managing their own credentials in their own ways, some in plain text in version control. At Uber's scale — 5,000 microservices, 5,000 databases, 500,000 analytical jobs per day — secrets sprawl is not a compliance problem. It is an incident waiting to happen. A team of ten engineers decided to fix it.
\nSecrets sprawl is the entropy of infrastructure security. Left to its own devices, every team builds its own vault, stores credentials however is convenient, and shares secrets in whatever way is fastest. At a startup with ten engineers, this is manageable. At Uber — 5,000 microservices, 5,000 databases, 400+ third-party integrations, 500,000 analytical jobs per day — it becomes a systemic security risk. By the time Uber's Secrets team began their consolidation project, the company had 150,000 secrets scattered across 25 separate vault systems, operated by different teams, with different security standards, different rotation practices, and inconsistent access controls. Some secrets were in plain text in codebases. Others lived in databases that had never been audited for credential exposure. Cyberattacks targeting exposed credentials were rising industry-wide. The question was not whether Uber should fix this — it was how.
\n\n\n\n🔐
\nBefore the consolidation, Uber's 25 separate vault systems were operated by various teams across engineering. Some were standard HashiCorp Vault (an open-source secrets management tool that provides a secure, centralized store for tokens, passwords, certificates, and encryption keys) deployments. Others were custom databases. Others were cloud-specific secret managers for AWS, GCP, and Azure. None of them talked to each other. None of them had a unified view of what credentials existed where.
\n
The Secrets team's strategy had two phases. Phase 1 was consolidation: take ownership of all vault infrastructure, standardize on a small number of canonical vault systems (one per cloud provider plus one on-premises HashiCorp Vault), and migrate all secrets from the 25 fragmented vaults into these six. This was the foundation work — unglamorous, involving hundreds of engineers across different teams, and requiring careful coordination to avoid breaking services that depended on existing vault paths. Phase 2 was the platform: building a Secret Management Platform on top of the consolidated vaults — a metadata model, lifecycle automation, unified API, and real-time scanning — that turned six vaults into a governed, auditable, self-service system.
\n\n\n\nTHE FIVE PROBLEMS THEY HAD TO SOLVE
\nAs the Secrets team consolidated vaults, five common problem patterns emerged that any future platform would need to address: (1) no unified metadata model — no way to know what a secret was for, who owned it, when it was last rotated; (2) no cross-vault CRUD — managing secrets across different vault types required different tools and APIs; (3) no developer self-service — engineers filed tickets to create or rotate secrets; (4) no inventory — no way to generate a complete list of secrets for security incident response; (5) no automated rotation — credential rotation required manual coordination, so it was delayed or skipped.
\n
Uber's infrastructure had grown faster than its secrets governance. 25 vault systems operated by different teams meant no single team had visibility into the company's complete credential inventory. Shadow IT vaults with no central oversight created audit gaps. Secrets were shared insecurely, rotated rarely, and sometimes stored in version control. With cyberattacks targeting credential exposure rising industry-wide, the status quo was untenable.
\nAt Uber's scale, decentralized secrets management doesn't produce diversity and resilience — it produces inconsistency and risk. Each of 25 vaults had its own standards, its own rotation schedule (usually none), its own access model. There was no way to answer basic security questions: who has access to which credentials? When were they last rotated? Are any credentials in source code? The scale that made the problem urgent also made it hard to fix without a dedicated team and platform.
\nPhase 1 consolidated 25 vaults into 6 centrally managed vaults (one per cloud provider plus on-prem HashiCorp Vault). Phase 2 built the Secret Management Platform: a metadata model, a unified API abstracting across all vault types, a Cadence-orchestrated Secret Lifecycle Manager (Uber's automation system that handles the complete lifecycle of secrets — creation, rotation, distribution to workloads, and eventual decommissioning — using Uber's Cadence workflow engine), real-time scanning across git/Slack/CI pipelines, and self-service developer tooling.
\nA team of 10 engineers now drives 20,000 automated monthly secret rotations — up from manual rotation that happened rarely. Secrets exposed in CI pipelines dropped by 90%. The platform generates a complete inventory of all 150,000 secrets on demand, enabling rapid response to security incidents. Uber is actively pursuing secretless authentication — replacing long-lived credentials with ephemeral, automatically-issued tokens wherever possible.
\n\n\n\n⚠️
\nThe Migration Scale Problem
Migrating secrets from 25 vaults to 6 involved hundreds of engineers whose workloads depended on existing vault paths. A secret migration is not just a data copy — it is a coordination problem. Every service reading a secret from vault path A needs to be updated to read from vault path B. In a monolith, that's one codebase. Across Uber's 5,000 microservices, that's 5,000 potential update targets. The team built tooling to discover which services were reading from which vault paths, generated migration checklists automatically, and used feature flags to switch services over gradually with rollback capability.
\n
The metadata model (a structured representation of a secret's properties — owner, purpose, rotation schedule, associated services, expiry date, security classification — that enables automated governance and incident response) was the architectural cornerstone of the Secret Management Platform. Before consolidation, a secret was just a key-value pair in a vault with no context. After the platform was built, every secret had a structured record: who owned it, which services used it, when it was last rotated, what its rotation policy was, and what its security classification was. This metadata made automated governance possible: the platform could identify secrets that hadn't been rotated in 90 days, generate compliance reports, and automatically alert owners of soon-to-expire credentials. It also made incident response practical: when a security team needed to identify all credentials that could have been exposed in a compromise, they could query the inventory rather than interviewing 250 engineering teams.
\n\n\n\nℹ️
\nReal-Time Scanning: Catching Secrets Before They Ship
One of the most impactful platform features was real-time scanning across Uber's code repositories, CI pipelines, and internal Slack messages. The scanner looks for patterns matching API keys, database passwords, and private key formats. When detected, it automatically revokes the exposed credential and alerts the owning team. Before the platform, a credential committed to git might live there for months — or forever. Now, exposure is measured in seconds. The 90% reduction in secrets found in pipelines reflects this detection-and-revocation automation.
\n
\n\n\n❌
\nShadow IT Vaults: The Security Debt Multiplier
Perhaps the most dangerous aspect of Uber's pre-platform state was what the team called shadow IT vaults: secret storage systems created by individual teams outside the knowledge of the central Secrets team. These vaults had no security baseline review, no rotation policy, no access audit, and no inventory. When a team built a shadow vault, they optimized for their immediate convenience — and created a security liability that the company didn't know existed. You cannot rotate credentials you don't know about. You cannot audit access to vaults you don't know exist. Shadow IT vaults are the point where 'move fast' becomes 'incur unquantifiable risk.'
\n
\n\n\nWHY 400 THIRD-PARTY INTEGRATIONS MATTER
\nUber's 400+ third-party vendor integrations are a significant factor in the secrets management challenge. Each integration requires credentials — API keys, OAuth tokens, database passwords — that must be rotated when vendors change their systems or when Uber's access policy changes. Before the platform, vendor credential rotation required manual coordination: someone had to get the new credentials from the vendor, find which services used them, update each service's configuration, and verify nothing broke. At 400 integrations, this manual process consumed disproportionate engineering time and rotations were often delayed. The Secret Lifecycle Manager automated the rotation for most standard integrations.
\n
\n\n\n🔄
\nBefore the Secret Management Platform, secret rotation at Uber required a service owner to coordinate with the Secrets team, obtain a new credential from the upstream provider, update their service's configuration, and verify the rotation succeeded. At 150,000 secrets across 5,000 services, this process ran rarely — not because security was a low priority but because the operational overhead was prohibitive at scale. Most secrets were rotated only when forced by a security incident or vendor requirement. The platform inverts this: rotation is the default, manual coordination is the exception.
\n
\n\n\nℹ️
\nKubernetes Native Injection
One of the most seamless developer experiences in the platform is Kubernetes-native secret injection. Rather than requiring services to call an API to retrieve their credentials at startup, the platform can inject secrets directly as environment variables or mounted files into Kubernetes pods at deploy time. This is transparent to application code — the service sees its credentials as normal environment variables, with no awareness of which vault they came from or how they were rotated. When a rotation occurs, the platform can trigger a pod restart with the new credentials injected automatically.
\n
The Secret Lifecycle Manager (SLM) is the operational core of Uber's Secret Management Platform. Built on Cadence (Uber's open-source distributed workflow engine, designed for long-running, fault-tolerant business processes — the same engine that powers Uber's ride dispatch and payment workflows), SLM orchestrates the complete lifecycle of every secret: initial creation, distribution to consuming services, periodic rotation, and eventual decommissioning. Using Cadence's durable workflow model means that secret rotation operations are fault-tolerant — if the rotation workflow fails midway through, it can resume from where it left off rather than leaving credentials in a half-rotated, potentially inconsistent state.
\n# Simplified Secret Lifecycle Manager rotation workflow (conceptual)\n# Real implementation uses Cadence's durable workflow primitives\n\nfrom cadence.workflow import workflow_method\n\nclass SecretRotationWorkflow:\n @workflow_method\n async def rotate_secret(self, secret_id: str):\n \"\"\"Cadence ensures this completes even if individual steps fail.\"\"\"\n \n # Step 1: Generate new credential from upstream provider\n new_credential = await self.generate_new_credential(secret_id)\n \n # Step 2: Write new credential to canonical vault\n # (Durable: if this step completes, Cadence records it)\n await self.write_to_vault(\n secret_id=secret_id,\n value=new_credential,\n version='new' # old version still readable during transition\n )\n \n # Step 3: Signal consuming services to reload credential\n # Each service has a registered reload handler\n consuming_services = await self.get_consumers(secret_id)\n for service in consuming_services:\n await self.signal_reload(service, secret_id)\n \n # Step 4: Verify all services are using new credential\n # (Wait for health checks to confirm)\n await self.verify_rotation_complete(secret_id, consuming_services)\n \n # Step 5: Expire old credential in upstream provider\n await self.revoke_old_credential(secret_id)\n \n # Step 6: Update metadata: last_rotated, next_rotation_due\n await self.update_metadata(secret_id, rotated_at=now())\n # Cadence schedules next rotation based on policy\n\n\nSECRETLESS AUTHENTICATION: THE NEXT FRONTIER
\nThe logical endpoint of Uber's secrets management journey is secretless authentication — a model where services don't hold long-lived credentials at all. Instead, they use their identity (a Kubernetes service account, a Spiffe/SPIRE identity, a cloud provider IAM role) to dynamically request short-lived tokens at runtime. When a token expires in 1 hour, there is nothing to steal, nothing to rotate, nothing to audit. Uber is actively building toward this model as the long-term replacement for static credential management. The Secret Management Platform is both the current solution and the bridge to the secretless future.
\n
\n\n\n✅
\nSelf-Service Developer Tooling
Before the platform, creating a new secret required filing a ticket with the Secrets team. The turnaround could be days. After the platform, developers can create, update, and delete secrets through a self-service API, CLI, and web UI — all of which enforce the metadata requirements and policy compliance automatically. The Secrets team's workload shifted from manual secret operations (which scaled linearly with the number of services) to platform maintenance and governance (which scales much more slowly). A team of 10 can now serve 5,000 microservices because the services serve themselves.
\n
\n\n\nℹ️
\nThe Multi-Vault Abstraction Layer
Uber's infrastructure spans AWS, GCP, Azure, and on-premises HashiCorp Vault. Each environment has its own native secret manager with a different API. The Secret Management Platform includes a unified abstraction layer that presents a single API for secret CRUD operations regardless of which underlying vault the secret lives in. Application code interacts with the platform API; the platform handles routing the operation to the correct vault (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or HashiCorp Vault) and translating the response. This abstraction decouples application code from vault topology — when Uber migrates a secret from one vault to another, no application code changes.
\n
\n\n\nℹ️
\nThe Unified API: One SDK, Four Vaults
Uber's unified abstraction layer exposes a single SDK that application developers use regardless of which underlying vault stores their secret. The SDK handles routing: an AWS-deployed service's database password might live in AWS Secrets Manager; an on-prem service's certificate might live in HashiCorp Vault. The developer writes
\nsecrets.get('myservice/db_password')and receives the credential — the SDK consults the metadata catalog to find which vault holds that secret and retrieves it via the appropriate vault API. Application code is decoupled from vault topology, making future vault migrations transparent.
\n\n\n✅
\nCompliance Reporting on Demand
Before the metadata model, answering a compliance auditor's question — 'show me all credentials with access to our payment processing systems' — would have required interviewing dozens of engineering teams over days. After the platform, the same question is answered by a metadata query: filter by associated_system='payment_processing', return all matching secrets with their rotation history, access policies, and owner contacts. Compliance reporting that took days now takes seconds. The metadata model was built for developer self-service but it turns out to be equally valuable for security operations and compliance.
\n
The Secret Management Platform sits as an orchestration layer above Uber's six canonical vault systems. Applications and services no longer talk directly to specific vaults — they interact with the platform's unified API or use Kubernetes-native injection (where secrets are automatically mounted into pods at deployment time). The platform maintains the metadata catalog, handles lifecycle automation via the Secret Lifecycle Manager, runs real-time scanning, and provides the developer self-service tools. The vault systems themselves are the authoritative stores; the platform is the governance and automation layer on top.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nCADENCE: WHY UBER CHOSE WORKFLOWS FOR ROTATION
\nSecret rotation is a multi-step process with real failure modes: the upstream provider might be unavailable, the vault write might fail, a downstream service might not acknowledge the new credential. A simple cron job or Lambda function that fails midway leaves the system in an unknown state — is the old credential still valid? Is the new one active? Cadence's durable workflow model provides exactly-once execution semantics: each step is recorded, and if the workflow fails partway through, it resumes from the last successful step. This makes secret rotation reliable enough to run 20,000 times per month without manual oversight.
\n
\n\n\n⚠️
\nThe Migration Coordination Challenge
Migrating a secret from its old vault path to the new centralized platform sounds like a database copy operation. In practice it's a distributed coordination problem across hundreds of teams. Every service reading the secret needs to be updated simultaneously (or with a dual-read transition period). Uber built tooling to discover all consumers of a vault path, generate migration checklists, and track completion status. Services that hadn't migrated within the target window were flagged for the owning team. The tooling made a migration problem tractable across an organization of thousands of engineers.
\n
\n\n\n🛡️
\nSPIFFE/SPIRE: The Path to Secretless
Uber's secretless authentication initiative builds on the SPIFFE/SPIRE framework — an open standard for issuing cryptographic workload identities. Every service at Uber has a unique SPIFFE identity that is automatically issued, cryptographically verifiable, and short-lived. Services that can authenticate using their SPIFFE identity don't need to hold long-lived credentials at all — the identity proves who they are, and the system issues time-limited tokens dynamically. As more of Uber's infrastructure adopts SPIFFE-based authentication, the number of long-lived secrets that need to be managed by the platform shrinks toward zero.
\n
Uber's secrets management story is about the organizational and engineering cost of decentralization without governance — and the compounding returns of building the right platform once.
\nWhat to remember
\n\n\n✅
\n10 Engineers, 5,000 Microservices
The most striking number in Uber's secrets story is the ratio: 10 engineers managing secrets governance for 5,000 microservices. This 500:1 leverage ratio is only possible because the platform does the work that used to require human coordination. Automated rotation, self-service tooling, policy enforcement in the platform layer — all of these shift work from the coordination model (each secret operation requires a human) to the automation model (each secret operation executes itself). Platform teams that want to scale should measure their leverage ratio and ask what automation would improve it.
\n
\n\n\nSECRETS IN VERSION CONTROL ARE NOT A MISTAKE; THEY'RE AN ARCHITECTURE PROBLEM
\nEvery engineering organization has discovered credentials accidentally committed to git. The standard response is to educate developers about the risk. Uber's analysis found that the root cause was not developer carelessness — it was the absence of a convenient alternative. When getting a credential into a service requires filing a ticket and waiting days, developers find a shortcut: put it in the config file. The secure path needs to be the easy path. The self-service API and Kubernetes injection that the Secret Management Platform provides made the secure approach easier than the shortcut.
\n
\n\nUber built a platform to manage 150,000 secrets at scale — and the most important feature turned out to be a metadata field that just says 'who owns this thing?'
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-18T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.885535+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Security", "Uber"]}, {"id": "https://techlogstack.com/explore/hotstar-ipl-scaling-2019/", "url": "https://techlogstack.com/explore/hotstar-ipl-scaling-2019/", "title": "When MS Dhoni Got Out: How Hotstar Survived 25 Million Concurrent Users", "summary": "How Hotstar built Project HULK, abandoned AWS autoscaling, and survived 25.3M concurrent users during the 2019 Cricket World Cup semi-final.", "content_html": "Hotstar · Live Streaming · 17 May 2026
\nJuly 9th, 2019. India vs New Zealand, Cricket World Cup semi-final. MS Dhoni walks to the crease and 1.1 million new viewers join Hotstar every single minute. Then he gets run out — and 24 million people hit the back button almost simultaneously.
\n\n\n\n🏏
\nAt the peak of the 2019 ICC World Cup semi-final between India and New Zealand, 25.3 million people were simultaneously streaming live on Hotstar — a global record for any streaming platform. The platform consumed 5.7 Tbps of bandwidth: roughly 70% of India's total internet capacity at the time.
\n
Cricket is not just a sport in India — it is synchronized national emotion. When MS Dhoni walks to the crease, tens of millions of people who had given up on the match suddenly reconsider. They reach for their phones. They open Hotstar. At one point during the semi-final, viewership was growing at 1.1 million users per minute — a rate that would overwhelm most cloud architectures before the first over was complete. Hotstar's engineering team had spent months anticipating exactly this moment, building an entirely custom infrastructure stack designed around one insight: the biggest danger in live sports streaming is not the peak load itself, but the speed of arrival at that peak.
\n\n\n\nTHE DHONI PROBLEM
\nThe growth rate was not the hardest engineering challenge — it was the drop. When Dhoni was run out, Hotstar went from 25.3 million concurrent viewers to under 1 million in minutes. Millions of users hit the back button simultaneously. They didn't leave the app — they returned to the homepage, where personalization, recommendation, and content-discovery APIs were suddenly hammered by a traffic tsunami that had nothing to do with video streaming. A system built only for video delivery would have collapsed on the exit.
\n
The platform's architecture in 2019 ran on AWS EC2 instances with Akamai as the primary CDN (Content Delivery Network — a distributed system of edge servers that caches and delivers content from locations close to the user, reducing latency and offloading origin servers), backed by Apache Kafka for real-time event streaming and Apache Flink for stream processing. Hotstar had already made the strategic shift to microservices (an architecture where an application is built as a collection of small, independently deployable services, each responsible for a specific function), migrating from a monolith in 2018, which allowed individual services to be scaled independently. But AWS's native Auto Scaling Groups (AWS's built-in mechanism for automatically adding or removing EC2 instances based on metrics like CPU usage) had a fundamental problem: when you need to go from 1.5 million to 15 million concurrent viewers in under ten minutes, sequential instance provisioning isn't fast enough.
\n\nDuring the IPL Final and World Cup, Hotstar's traffic grew at +500,000 concurrent users per minute during peak moments — triggered by push notifications sent by the marketing team when the match became exciting. AWS native auto-scaling groups couldn't provision new EC2 instances fast enough: they experienced insufficient capacity errors in specific availability zones, and their step-size mechanism added nodes in fixed batches rather than responding to the rate of change.
\nAvailability Zone (an isolated, physically separate data center within an AWS region, designed to be independent of failures in other zones) imbalance meant adding capacity to one zone could leave another undersupplied. ASG step size — adding a fixed number of instances per scaling action — couldn't respond fast enough to 1.1M user/minute growth. And insufficient capacity errors in a zone at peak were unrecoverable: there was simply no spare inventory to allocate on-demand during a national cricket final.
\nHotstar abandoned AWS ASG for live events entirely. Instead, they pre-warmed the full expected infrastructure before each match based on Project HULK predictions, maintaining a 2-million-user capacity buffer at all times. A custom internal scaling engine — driven by request rate and concurrency, not CPU — could spin up new capacity in under 90 seconds. A secondary ASG (the backup auto-scaling group kept on standby to provide a different instance type mix if the primary cluster hits limits) provided a different instance-type mix as a fallback if the primary cluster hit hard limits.
\nThe IND vs NZ semi-final became the largest concurrent live stream in history at the time: 25.3 million viewers, zero reported downtime. The platform handled both the spike to peak and the catastrophic drop when Dhoni got out — the homepage API layer, pre-scaled and tested via chaos engineering, absorbed the sudden exit traffic without incident. Hotstar's engineering approach became a canonical reference talk for scaling live event infrastructure on AWS.
\nThe most consequential engineering decision Hotstar made was building Project HULK — an internal load-testing platform with a footprint bigger than most companies' production environments. At full scale, Project HULK deployed 108,000 CPUs, 216 TB of RAM, and 200 Gbps of outbound network across 8 geographically distributed AWS regions, running geo-distributed load generators to simulate realistic user journeys. It performed four categories of tests: load generation to establish baseline capacity, Tsunami tests that simulated the sudden spike-and-drop profile of a Dhoni innings, chaos engineering (the practice of deliberately injecting failures into a system to discover weaknesses before they manifest in production) to test resilience when an availability zone went down, and ML-driven traffic pattern modelling to predict load curves for upcoming events. The insight that emerged was that Hotstar needed to face the real game before the actual game.
\n\n\n\nℹ️
\nBandwidth at the Edge of India's Internet
At 25.3 million concurrent users, Hotstar's bandwidth consumption hit 5.7 Tbps — approximately 70% of India's total available internet capacity at the time. This is not a platform metric; it is a national infrastructure metric. Hotstar engineers were not just managing application scale, they were managing demand at the level of physical network capacity across an entire country.
\n
\nRace against time: +500K growth rate per minute concurrency. Fully baked AMIs: 4 minutes. Application boot-up time: 90 seconds. Reaction time: push notifications.
— — Gaurav Kamboj, Cloud Architect at Hotstar — AWS Community Day Bengaluru 2018
\n\n\n⚠️
\nThe Push Notification Trap
The marketing team's push notifications — sent to bring users back to the app during exciting match moments — were a hidden traffic generator that engineering had no advance warning of. Every notification blast created an immediate, synchronized spike in both video and API traffic. Hotstar had to build a feedback loop where marketing intent was translated into infrastructure capacity decisions before the notification was sent, not after.
\n
\n\n\n🔄
\nThe Graceful Degradation Contract
When a service hit its capacity limits, Hotstar implemented panic mode (a defined degradation state where non-essential services are deliberately disabled to preserve capacity for the most critical user path — live video delivery). Recommendations, personalization, and social features were the first to shed load. Video streaming was always the last service standing. The contract was explicit: degrade gracefully rather than fail catastrophically.
\n
The Infradashboard — Hotstar's internal capacity planning tool — gave the operations team a real-time view of infrastructure headroom and allowed proactive scale-up decisions hours before a match began. The team maintained a permanent buffer of 2 million concurrent user capacity above current load at all times during live events. Because AWS ASG couldn't add new nodes fast enough during the actual match, the buffer had to already exist before the first ball was bowled. The combination of pre-warming, buffer maintenance, and custom scaling logic turned a reactive system into a predictive one — and it held at 25.3 million.
\n\n\n\n✅
\nRecord After Record — The Architecture That Kept Scaling
The 2019 record of 25.3 million concurrent viewers was the first proof of concept. By 2023, Hotstar hit 59 million during the Cricket World Cup final. In 2025, the ICC Champions Trophy Final drew 61 million simultaneous streams — more than the entire population of Italy, watching live on a single app. Each generation of the architecture was built directly on the engineering lessons of the one before.
\n
AWS Auto Scaling Groups were built for the average web application: traffic grows gradually, CPU rises, new instances are added over minutes. Live sports streaming is the opposite. Traffic can double in 90 seconds when MS Dhoni appears on screen. The AWS ASG step-size mechanism — adding instances in fixed batches — was simply too slow for this pattern. Hotstar's engineering team identified three specific failure modes in ASG behavior during live events: insufficient capacity errors (AWS returning an error when the requested EC2 instance type has no available inventory in a specific availability zone at that moment) during peak demand, availability zone skew (a condition where one AZ accumulates more load than others, exhausting its capacity while other AZs still have headroom) when scaling across AZs, and the lag between a scaling trigger and a serving-ready instance that could run to 4+ minutes when including AMI bake time and application boot.
\n# Hotstar's custom scaling logic — conceptual pseudocode\n# Key insight: scale on REQUEST RATE and CONCURRENCY, not CPU utilization\n\nclass HotstarScaler:\n CAPACITY_BUFFER = 2_000_000 # always maintain 2M concurrent user headroom\n REACTION_TIME_TARGET = 90 # seconds to new serving capacity\n BOOT_TIME = 75 # seconds for app boot from pre-baked AMI\n\n def compute_required_capacity(self, current_concurrency, request_rate):\n # Project forward 5 minutes using ML traffic model for this match\n projected_peak = self.ml_model.predict_peak(\n current=current_concurrency,\n rate=request_rate,\n event_type='cricket_live'\n )\n # Add mandatory buffer — never let buffer drop below 2M\n return projected_peak + self.CAPACITY_BUFFER\n\n def pre_warm_for_event(self, event_metadata):\n # Called hours before match start — pre-bake AMIs in all regions\n # Uses HULK traffic model predictions, not current load\n predicted_peak = self.ml_model.predict_peak_from_event(event_metadata)\n target_capacity = predicted_peak + self.CAPACITY_BUFFER\n\n for region in ACTIVE_REGIONS:\n # Launch PRIMARY ASG with main instance type mix\n primary_asg.scale_to(target_capacity * 0.8, region)\n # Launch SECONDARY ASG with diverse instance types as fallback\n # Protects against insufficient capacity in any single instance family\n secondary_asg.scale_to(target_capacity * 0.2, region)\n\n def scale_on_signal(self, concurrency_metric, request_rate):\n required = self.compute_required_capacity(concurrency_metric, request_rate)\n current = infrastructure.get_active_capacity()\n if current < required:\n delta = required - current\n # SNS alert triggers Lambda to activate secondary ASG immediately\n sns.publish('scale_required', delta=delta)\n lambda_handler.trigger_secondary_asg(delta)\n\n\nTHE SECONDARY ASG PATTERN
\nHotstar ran two auto-scaling groups simultaneously for every live event. The primary ASG held the bulk of capacity, pre-warmed before the match. The secondary ASG — with a different mix of instance types — acted as an emergency reserve. An AWS SNS alert triggered a Lambda function to activate the secondary ASG when the primary hit limits. This avoided the single-point failure of depending on one instance family being available in a specific AZ during peak national demand.
\n
The Infradashboard was the operational nerve centre during live events. Engineers could see capacity headroom in real time, trigger manual scale-up actions hours in advance of anticipated spikes, and monitor which services were approaching their degradation thresholds. The key operational insight was that live sports infrastructure management begins the morning of the match, not when the concurrency alert fires. Pre-warming full capacity ahead of push notifications — the external traffic amplifiers that marketing controlled — meant the infrastructure was already serving at near-peak levels before the first viewer joined. The Infradashboard made proactive capacity management a team sport between engineering and marketing.
\n\n\n\n✅
\nKubernetes Migration: The Long-Term Fix
The 2019 architecture was built on raw EC2 instances with a custom autoscaler. The engineering team recognized this was not indefinitely scalable. In 2018, Hotstar had begun migrating to containerized microservices on a self-managed Kubernetes cluster, which enabled pod-level scaling in seconds rather than minutes. By 2023, this evolution reached EKS with Data Center Abstraction — the infrastructure that would handle 61 million concurrent viewers four years later.
\n
\n\n\nℹ️
\nGraceful Degradation Protocol
Hotstar defined a tiered degradation order for when services approached capacity limits. Tier 1 (shed first): recommendations, social features, personalization. Tier 2: match statistics, secondary content APIs. Tier 3 (never shed): live video delivery, playback APIs, authentication. The protocol was automated and tested via Project HULK's chaos engineering scenarios — so when the real moment came, the system degraded on its own without human intervention.
\n
\n\n\n📡
\nThe Client-Side Backoff Contract
When backend latency exceeded thresholds, Hotstar's client applications were programmed to increase the interval between retry requests — backing off rather than hammering the server. This client-side behaviour was the last line of defense: when 25 million devices all experience a glitch simultaneously, the difference between a recoverable spike and a cascading failure can come down to whether the clients know to wait before retrying.
\n
Hotstar's architecture in 2019 was built for one governing constraint: traffic does not arrive smoothly. A cricket match involving India can go from zero to 10 million concurrent viewers in under ten minutes — faster than any traditional auto-scaling system can respond. The platform ran on AWS EC2 across multiple regions, with Akamai as the CDN (Content Delivery Network — a distributed network of edge servers that caches video segments and static assets close to users, absorbing the majority of load before it reaches origin servers) delivering video segments. Apache Kafka ingested 10 billion-plus clickstream events per match day. Microservices handled video playback, match statistics, personalization, and authentication independently — each able to be scaled or degraded without affecting the others. The three-layer architecture — CDN edge, application tier, data tier — had to be specifically engineered so that no layer became a bottleneck at the velocity of a cricket crowd.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nPROJECT HULK: THE PRODUCTION REHEARSAL
\nProject HULK was not a staging environment — it was a separate production-scale load testing infrastructure that ran geo-distributed tests from 8 AWS regions simultaneously. Its load generation cluster alone used c5.9xlarge instances (36 vCPUs, 72 GB RAM each) to generate realistic concurrent user traffic. Simulations ran four test types: baseline load, tsunami testing (a stress test pattern that simulates sudden extreme spikes and drops in traffic, matching the pattern of a cricket match when a wicket falls), chaos engineering with AZ failures, and ML-trained traffic pattern replay of previous match profiles. The goal: there should be no mode of failure in production that HULK has not already triggered in a test.
\n
View interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\n⚠️
\nThe Bandwidth Ceiling No One Talks About
At 5.7 Tbps peak, Hotstar was approaching a hard physical limit — not a software limit. Adding more servers would not help if India's total internet capacity couldn't carry the bytes. This ceiling forced Hotstar's engineers to think about CDN efficiency and adaptive bitrate streaming (a technology that dynamically switches video quality based on the viewer's network conditions, delivering the best quality the connection can support at any moment) not just as user experience decisions, but as existential infrastructure constraints.
\n
\n\n\n🌐
\nThe Microservices Migration That Made It Possible
Hotstar migrated from a monolith to microservices in 2018 — just one year before the 25.3M record. This was the architectural prerequisite for everything else: without independent service scaling, graceful degradation tiers, and per-service capacity controls, the custom autoscaling and Infradashboard capabilities described here would have been impossible to build. You cannot surgically shed load from a monolith.
\n
Hotstar's 2019 story is not about surviving a traffic spike — it is about re-designing the relationship between infrastructure and time. The core lesson is that reactive systems cannot serve live events. When your traffic can double in 90 seconds, you need infrastructure that is already there. Every engineering decision Hotstar made — Project HULK, pre-warming, custom ASG, the buffer, graceful degradation — was aimed at the same goal: transforming a system that responds to load into one that anticipates it.
\nWhat to remember
\n\n\nTHE DHONI EFFECT — AND WHAT CAME NEXT
\nThe 2019 record of 25.3 million concurrent viewers held until 2023, when Hotstar hit 59 million during the Cricket World Cup — then broke its own record again in 2025 with 61 million during the ICC Champions Trophy Final. Each leap required a new generation of architecture: from EC2 to Kubernetes, from Kubernetes to EKS, from EKS to Data Center Abstraction with Envoy-based gateways. The 2019 engineering story was chapter one.
\n
\n\n\n🌍
\nThe Template That Changed Streaming Architecture
Hotstar's 2019 engineering approach — pre-warming, custom autoscaling based on concurrency, graceful degradation tiers, and game-day chaos testing — became a reference architecture cited in AWS re:Invent talks and engineering conference circuits worldwide. The project that started as a solution to one cricket match became the blueprint for live-event streaming infrastructure globally.
\n
\n\nOne cricket match used a quarter of India's internet. The engineering team called that a success.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-17T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.089953+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Live Streaming", "Hotstar"]}, {"id": "https://techlogstack.com/explore/github-mysql-8-upgrade-2023/", "url": "https://techlogstack.com/explore/github-mysql-8-upgrade-2023/", "title": "How GitHub Upgraded 1200 MySQL Hosts Without Dropping a Single Query", "summary": "GitHub's year-long mission to upgrade 1,200+ MySQL hosts from 5.7 to 8.0 while maintaining 5.5 million queries per second and keeping rollback available at every ste", "content_html": "GitHub · Databases · 17 May 2026
\nMySQL 5.7 was hitting end-of-life, and GitHub's production database fleet spanned 1,200 hosts, 300 terabytes of data, and 5.5 million queries every second. Getting from here to MySQL 8.0 without disrupting 100 million developers was going to take more than a weekend.
\nGitHub started as a Ruby on Rails application with a single MySQL database over 15 years ago. Since then, MySQL had become the foundation of everything GitHub stores: repositories, pull requests, issues, code review comments, user accounts, billing data, and the entire social graph of 100 million developers. By 2022, MySQL 5.7 (the production MySQL version GitHub had been running for years, which Oracle officially declared end-of-life in October 2023, meaning no more security patches or bug fixes) was approaching end-of-life — Oracle had announced support would end in October 2023. The GitHub database team made a simple calculation: stop receiving security patches on the database that holds every line of code pushed to GitHub, or upgrade. The only real question was how to upgrade 1,200 hosts, 300+ TB of data, and 5.5 million queries per second without disrupting a single user-visible transaction.
\nPreparation began in July 2022 — a full year before any production host was promoted to 8.0. The team added MySQL 8.0 to CI (Continuous Integration — the automated system that runs tests against every code change before it merges, ensuring the codebase is always in a shippable state) for all applications using MySQL, running 5.7 and 8.0 side-by-side to catch regressions early. They built MySQL 8.0 Codespaces (GitHub's cloud development environment that spins up isolated VM workspaces for debugging, allowing engineers to test against specific MySQL versions without affecting production) debug containers so developers could test their queries against the new version. They created an internal GitHub Project board to track every cluster's upgrade status across the entire fleet. And they did all of this before upgrading a single production host. The discipline of the preparation phase is what made the execution phase look routine.
\n\n\n\nTHE HIDDEN BREAKING CHANGE
\nMySQL 8.0 changes the default character set to utf8mb4 and its default collation to `utf8mb4_0900_ai_ci` — a newer Unicode specification that MySQL 5.7 does not support. This created a problem: when an 8.0 primary replicates writes to a 5.7 replica, the collation metadata in the binary log (the record MySQL maintains of every data modification, used to replicate changes to replica hosts and to reconstruct data state for point-in-time recovery) can cause replication to break entirely on the downstream 5.7 nodes. GitHub's rollback strategy depended on maintaining backward replication from 8.0 to 5.7 — so this had to be solved before a single production primary was promoted.
\n
Oracle announced MySQL 5.7 end-of-life for October 2023, cutting off security patches and bug fixes. GitHub's 1,200+ host fleet running at 5.5M QPS could not safely continue on an unsupported database version. The challenge was executing a major version upgrade across a mixed fleet of Azure VMs and bare-metal hosts without a maintenance window or service disruption.
\nTesting revealed two breaking changes: MySQL 8.0's new default utf8mb4_0900_ai_ci collation (a Unicode 9.0 character sorting specification supported in MySQL 8.0 but absent from 5.7, causing replication to break when an 8.0 primary writes to a 5.7 replica) broke downstream 5.7 replicas, and the new MySQL 8.0 roles feature caused permission-expansion scripts to generate 8.0-syntax statements that 5.7 replicas could not parse. Both had to be patched before any primary promotion.
\nGitHub built a 5-step playbook: upgrade replicas one data center at a time, reconfigure the replication topology (the tree of primary and replica MySQL hosts through which write changes propagate — the primary receives writes and replicas receive a stream of changes to stay in sync) to create parallel 5.7 and 8.0 chains, promote an 8.0 host to primary via graceful failover, keep 5.7 standbys ready for rollback, then clean up after 24 hours of successful traffic.
\nEvery cluster upgraded without a single SLO violation. The rollback path was preserved throughout the entire year-long process — a 5.7 standby was always available. The project delivered not just the MySQL 8.0 upgrade but a repeatable automation framework for future major version upgrades, so the next one will be faster.
\n\n\n\n⚠️
\nThe Vitess Complication: Version Advertisement
Vitess (YouTube's open-source MySQL sharding layer that GitHub uses for its highest-traffic product domains) adds an extra layer of complexity: its proxy component VTgate (Vitess's query router that intercepts MySQL connections and directs them to the correct shard) advertises the MySQL version to client applications. One Java client was checking the advertised version to decide whether to disable the MySQL query cache — a feature that was completely removed in 8.0. As soon as even one shard in a Vitess keyspace was upgraded, VTgate's version advertisement had to be updated, otherwise the Java client would generate blocking errors. Timing the VTgate version bump to coincide exactly with the first shard promotion became a critical coordination step.
\n
\nUpgrading the fleet with no impact to our Service Level Objectives (SLO) was no small feat — planning, testing and the upgrade itself took over a year and collaboration across multiple teams within GitHub.
— — Jiaqi Liu, Daniel Rogart, Xin Wu — via GitHub Engineering Blog
\n\n\n🔄
\nGitHub's engineers discovered a replication bug in MySQL 8.0 that only manifested under intensive load over long periods — a host could eventually run out of commit-order sequence numbers and stall. The bug had been patched in MySQL 8.0.28. This meant GitHub had to ensure all hosts were on 8.0.28 or later before any long-running cluster was considered safe, adding a version-pinning requirement to an already complex upgrade matrix.
\n
The upgrade process for each cluster was designed to preserve the rollback option at every single step. Promoting an 8.0 replica to primary was never an irreversible action until after 24 hours of clean traffic had confirmed success. During the brief window of dual replication chains, GitHub maintained a set of offline 5.7 replicas specifically for rollback — not serving traffic, not receiving new promotion candidates, just sitting ready. Orchestrator (GitHub's open-source MySQL topology management tool that handles automated failover, replication topology visualization, and candidate promotion) was configured to blacklist all 5.7 hosts as failover candidates during this window, preventing an automated failover from accidentally rolling back to 5.7 during an unplanned outage. The architecture of the rollback path was as carefully designed as the architecture of the upgrade path itself.
\n\n\n\n🔧
\ngh-ost: Schema Changes Without Table Locks
GitHub's in-house tool gh-ost (GitHub Online Schema Migrations) was a critical part of the upgrade preparation. It enabled schema changes required for MySQL 8.0 compatibility to be applied to production tables without locking them — essential when those tables receive millions of queries per second. Without gh-ost, applying schema changes to GitHub's largest tables would have required multi-hour maintenance windows that users would have noticed.
\n
\n\n\nℹ️
\nWhat MySQL 8.0 Actually Unlocked
Beyond escaping end-of-life, MySQL 8.0 delivered features GitHub's database team genuinely wanted. Instant DDLs allow many schema changes to be applied without rebuilding the entire table — critical for a 300+ TB fleet where traditional ALTER TABLE could take hours. Invisible indexes let engineers create an index, test it under production traffic without it being used by the query planner, and only then make it active — dramatically safer index deployment. Compressed binary logs reduce replication bandwidth between primary and replicas, a meaningful saving at 5.5M queries per second.
\n
The hardest technical problem in this upgrade was not moving forward — it was preserving the ability to move backward. MySQL officially supports replication from a lower version to the next higher version but does not support reverse replication from 8.0 down to 5.7. When GitHub tested this in staging, promoting an 8.0 host to primary caused replication to break on all downstream 5.7 replicas immediately. Two root causes: MySQL 8.0's new default collation (a set of rules that determines how character strings are compared and sorted; different collations can produce different sort orders for the same strings) `utf8mb4_0900_ai_ci` was not recognized by 5.7's replication parser, and MySQL 8.0's new ROLE management syntax generated statements in the binary log (the sequential log of all data-modifying SQL statements that MySQL writes to enable replication and point-in-time recovery) that 5.7 could not execute. Both required surgical fixes before any production promotion could proceed.
\n-- The collation incompatibility fix:\n-- MySQL 8.0 defaults to utf8mb4_0900_ai_ci (Unicode 9.0)\n-- MySQL 5.7 only supports up to utf8mb4_unicode_520_ci\n-- Fix: explicitly set database/table collations to a 5.7-compatible value\n\n-- On the 8.0 primary, before promotion:\nALTER DATABASE github_production\n CHARACTER SET utf8\n COLLATE utf8_unicode_ci; -- 5.7-compatible collation\n\n-- Verify that new tables inherit the correct collation\nSHOW CREATE TABLE repositories\\G\n-- Should show utf8_unicode_ci, NOT utf8mb4_0900_ai_ci\n\n-- Confirm replication is running on downstream 5.7 replicas\n-- after a test write to ensure no Seconds_Behind_Master growth\nSHOW SLAVE STATUS\\G\n-- Expected: Seconds_Behind_Master: 0\n-- Slave_SQL_Running: Yes\n-- Last_Error: (empty)\n\n-- The roles fix: temporarily strip role-expansion from permission grants\n-- during the upgrade window so no ROLE syntax appears in the binlog\n\n\nℹ️
\nThe Dual Replication Chain Architecture
During the critical promotion window, GitHub maintained two parallel replication chains downstream of a single 8.0 replica: one chain of offline 5.7 standbys ready for rollback, and one chain of serving 8.0 replicas handling production traffic. This dual-chain state lasted only hours per cluster — long enough to confirm 8.0 health before decommissioning the 5.7 standby chain. The temporary cost: double the replica infrastructure per cluster during the promotion window.
\n
\n\n\nORCHESTRATOR: PREVENTING ACCIDENTAL ROLLBACK
\nOrchestrator (an open-source MySQL high-availability tool co-created by GitHub that manages replication topology and automated failover) is configured to make automated failover decisions when a primary fails. During the upgrade, GitHub added an explicit blacklist of all 5.7 hosts as failover candidates. Without this, an unplanned primary failure during the upgrade window could have caused Orchestrator to promote a 5.7 host as the new primary — an automated rollback that would undo hours of upgrade work and potentially confuse application behavior with a sudden version downgrade. The blacklist was the safety guard against automation working against the upgrade.
\n
After each primary promotion, GitHub's policy required at least one complete 24-hour traffic cycle before declaring a cluster successfully upgraded and decommissioning the 5.7 standby chain. This was not arbitrary — GitHub's traffic has strong diurnal patterns, with dramatically different load profiles between business-hours peak traffic and overnight lows. A cluster that behaved well during off-peak hours might reveal latency regressions during the morning rush of developers opening pull requests in Europe and North America. The 24-hour window caught several edge cases in early clusters that were fixed before the team moved to the next one.
\nGitHub's 5-Step MySQL 8.0 Upgrade Playbook Per Cluster
| Step | Action | Rollback Available? |
|---|---|---|
| 1 | Upgrade replicas one DC at a time; route read traffic to 8.0 replicas | Yes — disable 8.0 replicas, re-enable 5.7 |
| 2 | Reconfigure topology: split into dual 8.0 and 5.7 replication chains | Yes — fail back to 5.7 chain |
| 3 | Promote 8.0 replica to primary via Orchestrator graceful failover | Yes — 5.7 chain still in sync |
| 4 | Monitor for 24 hours of complete traffic cycle at full load | Yes — promote 5.7 standby if needed |
| 5 | Decommission 5.7 standbys after 24h success confirmation | No — rollback window closed |
\n\n\n✅
\nThe Automation Investment That Pays Forward
GitHub's database team explicitly designed this upgrade to produce a reusable automation framework for future major MySQL versions. The tooling for mixed-version CI, the dual-chain promotion scripts, the rollback procedures, the checklist issue templates — all of it was built as a library, not a one-time script. When MySQL 9.0 eventually needs to be adopted, the playbook already exists. The year of effort became infrastructure.
\n
\n\n\n🏷️
\nThe Mixed-Version CI Safety Net
Running MySQL 5.7 and 8.0 side-by-side in CI for all applications throughout the entire year-long upgrade was the single most important safety investment GitHub made. Application teams discovered query incompatibilities, deprecated feature usage, and reserved keyword conflicts in automated tests rather than in production promotions. This meant by the time each cluster was promoted, the application code was already known-compatible — the upgrade was validating infrastructure, not discovering application bugs.
\n
GitHub's MySQL fleet is not a single cluster — it's a network of over 50 independent clusters, each serving a specific product domain (repositories, issues, pull requests, billing, etc.), with larger domains horizontally sharded via Vitess. Each cluster has its own primary-replica topology. The upgrade had to be executed independently per cluster, each following the same 5-step playbook, with the dual replication chain state existing only during the transition window. Understanding this topology is essential to understanding why the upgrade took a year and why that was the right timeline, not the wrong one.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE TOOLING ECOSYSTEM
\nGitHub's MySQL reliability depends on a suite of open-source and in-house tools: Orchestrator manages replication topology and automated failover; gh-ost applies online schema changes without table locks; freno throttles schema migration speed based on replica lag to prevent migrations from disrupting production reads; and Percona Toolkit provides checksumming and replication verification. Without this ecosystem, the year-long upgrade would have required dozens of maintenance windows instead of zero.
\n
\n\n\nℹ️
\nVitess Sharding: One Shard at a Time
GitHub's Vitess clusters required upgrading one shard at a time rather than one cluster at a time, adding an inner loop to the upgrade playbook. For each sharded keyspace, the VTgate version advertisement had to be updated immediately after the first shard was promoted to 8.0 — otherwise client applications checking the advertised version would behave incorrectly. This timing constraint added coordination overhead but was resolved with explicit upgrade checklist items per keyspace.
\n
\n\n\n📋
\nGitHub Projects as the Upgrade Control Plane
GitHub used its own GitHub Projects tool to build a rolling calendar that tracked every cluster's upgrade status, scheduled upcoming cluster promotions, and coordinated between the database team and application teams. Issue templates gave application teams a standardized checklist for validating their service before and after each promotion. Meta-note: GitHub building GitHub with GitHub to upgrade GitHub's database is either very on-brand or very circular, depending on your disposition.
\n
GitHub's MySQL 8.0 upgrade is one of the cleanest examples of large-scale infrastructure migration executed with discipline. The lessons here are as much about process and architecture as they are about database mechanics.
\nWhat to remember
\n\n\nTHE END-OF-LIFE FORCING FUNCTION
\nMySQL 5.7's end-of-life announcement was the external forcing function that gave GitHub's database team the organizational priority to execute this migration. Security patch cutoffs are one of the most effective levers for getting cross-team infrastructure migrations approved and resourced. If your team has been deferring a major version upgrade, check when your current version's security support ends — it may already be overdue.
\n
\n\n\n⚠️
\nThe Replication Bug in 8.0 Pre-0.28
GitHub's testing surfaced a MySQL bug where replica_preserve_commit_order under intensive load could cause a host to exhaust commit-order sequence numbers and stall replication. The fix was in 8.0.28. This meant every host had to be on at minimum 8.0.28 — adding a version-pinning constraint to an already complex upgrade matrix. Moral: always scan the target version's release notes for known bugs before committing to that specific build in production.
\n
\n\nThey upgraded 1,200 database hosts without a single user noticing — which means they either did extraordinary engineering or extraordinary documentation, and based on the blog post, it was both.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-17T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.176139+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Databases", "GitHub"]}, {"id": "https://techlogstack.com/explore/slack-cellular-architecture-2023/", "url": "https://techlogstack.com/explore/slack-cellular-architecture-2023/", "title": "Slack Built a Big Red Button to Drain an Entire Data Center in Five Minutes", "summary": "How a 2021 availability zone outage led Slack to spend 1.5 years migrating to a cellular architecture with an AZ drain capability that works in under 5 minutes.", "content_html": "Slack · Reliability · 17 May 2026
\nOn June 30, 2021, a network link connecting one AWS availability zone failed — and Slack users felt it, despite Slack running in multiple availability zones. The postmortem question was brutal: why did a single AZ failure affect users at all? The answer drove 18 months of architecture work.
\nSlack runs most of its core infrastructure in the AWS us-east-1 region across multiple availability zones (isolated data centers within the same geographic region, designed so that a failure in one AZ does not affect others — each AZ has independent power, cooling, and networking) (AZs). The cloud infrastructure guarantee is clear: AZs should provide failure isolation. A problem in one AZ should not cascade to others. On June 30, 2021, at 11:45am PDT, an intermittent fault developed in a network link connecting one AZ to its neighbors. From a physical hardware perspective, this was an unremarkable incident — a flaky network link that was automatically removed from service at 12:33pm, restoring full connectivity 48 minutes after it first showed symptoms. What was remarkable was that Slack's users felt it at all.
\n\nWe were led to wonder why, in fact, this outage was visible to our users at all. Slack operates a global, multi-regional edge network, but most of our core computational infrastructure resides in multiple Availability Zones within a single region, us-east-1.
— — Slack Engineering — via Slack's Migration to a Cellular Architecture blog post
The answer was gray failure (a failure mode where different components have different views of system availability — some servers see one AZ as fully available while servers in that AZ see the others as unavailable, creating an inconsistent state that is much harder to detect and respond to than a clean hard failure). When the network link became intermittent, Slack's systems within the impacted AZ believed they had full connectivity to everything inside that AZ. Systems outside the AZ saw it as unavailable. Even clients within the same AZ had inconsistent views depending on whether their specific network flow traversed the failed equipment. This partial, view-dependent failure was far harder to detect and respond to than a clean hard failure. No single alert could capture it. No automated remediation was precise enough to act on it. The answer, the team concluded, was not to solve automated remediation of gray failures — it was to make the computers' job easier by relying on human judgment.
\n\n\n\nTHE BUTTON WE NEEDED
\nDuring the June 2021 incident, engineers monitoring the outage could see clearly on their dashboards that one AZ was the problem — nearly every graph segmented by target AZ told the same story. If there had been a button to tell all systems 'this AZ is bad; avoid it,' they would have pressed it immediately. So the goal became: build that button. Design requirements: drain an AZ within 5 minutes with no user-visible errors, operable from outside the affected AZ itself.
\n
A network link connecting one AWS AZ to the others experienced intermittent faults for 48 minutes. Despite Slack running in multiple AZs, users experienced degraded service — because Slack's core infrastructure was monolithically distributed across AZs without AZ-aware traffic isolation. No single switch could route traffic away from the affected AZ.
\nGray failures (partial failures where different components have inconsistent views of system availability, making it impossible to detect or respond to them with simple binary health checks) are uniquely dangerous in multi-AZ architectures. When a network link is intermittently faulty, not flaky, the failure depends on which specific flow traverses the bad equipment. Automated health checks often cannot detect this — they pass most of the time and fail occasionally, making the system appear healthy by aggregated metrics.
\nSlack spent 1.5 years migrating its most critical user-facing services to a cell-based architecture — with 3-4 independent instances of each service, one per AZ. An AZ drain button was built that, when pressed by an operator, reroutes all traffic away from the targeted AZ within 5 minutes. The drain mechanism was designed to operate from outside the affected AZ so it remains usable even when the AZ's own control plane is degraded.
\nAn AZ failure that previously required 48 minutes of user impact can now be mitigated in under 5 minutes via the drain button. Slack's 99.99% availability SLA (less than 1 hour downtime per year) makes 5-minute mitigation operationally viable; 48 minutes does not. The cellular architecture also brought independent deployment, testing, and monitoring for each cell.
\nThe cellular architecture migration forced Slack to make hard decisions about which services could be siloed cleanly and which could not. The key dividing line was statefulness. Stateless services — those that hold no long-lived data and process requests independently — are natural candidates for full siloing: run 3-4 independent instances, one per AZ, and route requests to the closest healthy instance. Stateful services — those that are the system of record for data — are harder. Distributing state across cells introduces consistency challenges under CAP theorem (the theoretical result stating that a distributed data store can provide at most two of: Consistency (all nodes return the same data), Availability (every request receives a response), and Partition tolerance (the system continues operating despite network partitions)) tradeoffs. Slack's team used CAP theorem analysis as a principled framework for categorizing each service and selecting the appropriate cell architecture for it.
\n\n\n\n🔴
\nSlack's AZ drain button has a design requirement that many engineers overlook: it must not rely on the impacted AZ to function. During large network outages, SSH-ing into servers in the affected AZ to make them 'lame-duck' themselves is unreliable. The drain mechanism works from outside — using control plane infrastructure that is deliberately hosted in AZs other than the one being drained.
\n
\n\n\nℹ️
\nIncremental Traffic Recovery: 1% at a Time
The drain capability is bidirectional — Slack can also gradually reintroduce traffic to a recovering AZ rather than dumping all traffic back at once. Starting at 1% and monitoring for errors before increasing gives engineers confidence that the recovery is real before exposing the full user base. This incremental re-introduction is as important as the drain itself: a hard full restore after an AZ incident often triggers a second cascade as cold caches and restarted services encounter sudden full-load.
\n
\n\n\n📐
\nWhy Not Automatic AZ Failover?
Automatic remediation of gray failures is technically hard because the signal is ambiguous — partial connectivity, inconsistent views, intermittent errors that don't trigger clean alert thresholds. Slack's architects chose to rely on human judgment for the drain decision while making the execution automated and fast. An operator who can see the dashboards and understand that 'one AZ is bad' is a more reliable detector than any automated system for this class of failure.
\n
\n\n\n⚠️
\nThe 99.99% SLA Math
Slack's 99.99% availability SLA means less than 1 hour of downtime per year is tolerable. The June 2021 AZ incident lasted 48 minutes — nearly the entire annual budget in a single event. A second incident of similar duration would breach the SLA. The cellular architecture and AZ drain button are not aspirational reliability improvements; they are the technical prerequisites for Slack to honor its contractual commitments to enterprise customers.
\n
\n\n\n🏗️
\nShipping Deep Changes Across Connected Services
One of the most underappreciated challenges of the cellular migration was coordinating changes across services that have live dependencies on each other. Converting Service A to a cellular model while Service B still calls it monolithically requires careful sequencing and temporary compatibility shims. Slack's engineering team wrote extensively about the 'ship the change' problem: making sweeping architectural changes to a live system without disrupting the engineers working on it daily.
\n
\n\n\nWHAT 'GOOD ENOUGH' ENABLED
\nSlack's architecture team made an explicit decision to embrace a 'good enough' cellular model rather than pursuing perfect cell isolation. Some services couldn't be fully siloed without years of additional work. A cell with 80% AZ isolation that can be built in 6 months is more valuable than a perfectly isolated cell that requires 3 years. The pragmatic threshold — drain within 5 minutes with no user errors — guided every architecture decision.
\n
The cellular architecture migration is notable not just for what it produced but for how long it took. 1.5 years of engineering effort across dozens of services, with careful sequencing to avoid disrupting a platform that millions of professionals depend on every day. The team decomposed the problem by service type, migrated services incrementally starting with those most amenable to siloing, and built the AZ drain infrastructure before migrating all services to depend on it. The project combined the operational discipline of a database migration with the architectural ambition of a complete infrastructure overhaul.
\n# Simplified AZ drain logic (conceptual)\n# Real implementation uses load balancer weight APIs and health check manipulation\n\nclass AZDrainButton:\n def drain_az(self, target_az: str):\n \"\"\"Drain all traffic from target_az within 5 minutes.\n Operable from any AZ — does not rely on target_az control plane.\"\"\"\n \n # Step 1: Update load balancer weights to 0 for target_az\n # Uses the cloud provider API — operates outside the AZ itself\n for service in self.critical_services:\n self.lb_api.set_weight(\n service=service,\n az=target_az,\n weight=0 # no new traffic; existing connections drain naturally\n )\n \n # Step 2: Update internal service discovery to prefer other AZs\n self.consul_api.set_az_preference(\n preferred_azs=[az for az in ALL_AZS if az != target_az],\n avoid_az=target_az\n )\n \n # Step 3: Monitor drain progress — connections should complete within 5 min\n return self.monitor_drain_progress(target_az, timeout_minutes=5)\n \n def gradual_restore(self, target_az: str, start_pct: float = 0.01):\n \"\"\"Incrementally restore traffic to recovering AZ starting at 1%.\n Monitor for errors before increasing allocation.\"\"\"\n current_pct = start_pct\n while current_pct <= 1.0:\n self.lb_api.set_weight(target_az, weight=current_pct)\n if self.error_rate_acceptable(target_az):\n current_pct = min(current_pct * 2, 1.0) # double until 100%\n else:\n self.lb_api.set_weight(target_az, weight=0) # back to zero\n break\n\n\nSTATEFUL VS STATELESS CELL STRATEGY
\nSlack's cellular migration required a principled decision for each service: can this service be independently siloed per AZ? Stateless services (no persistent data) are straightforward — run 3-4 independent instances. Stateful services (system of record) require more nuance. Services optimizing for availability during a partition get AZ-isolated instances that can serve stale data. Services requiring consistency stay centralized with careful cross-AZ replication. CAP theorem is not an abstract thought experiment here — it is a deployment decision for each individual service.
\n
\n\n\n✅
\nIndependent Deployment Per Cell
An underappreciated benefit of cellular architecture is that each cell can be deployed and updated independently. A canary deployment that goes wrong in Cell A does not affect Cell B or Cell C. This dramatically reduces the blast radius of bad deploys — one of the leading causes of production incidents. Cellular architecture is not just a reliability pattern; it's a deployment safety pattern.
\n
\n\n\nℹ️
\nTesting the Drain Before You Need It
Slack's engineering team explicitly built tooling to test the AZ drain mechanism regularly — not just in staging but in production via controlled drains of individual services. This is chaos engineering applied to the mitigation tool itself: if the drain button is only tested during incidents, its failure modes will be discovered at the worst possible moment. The drain mechanism is exercised regularly to ensure it works when it matters.
\n
\n\n\n🌐
\nSlack has a global multi-regional edge network that handles user connections near the user's geographic location. This edge layer is already highly distributed. The cellular architecture migration focused on Slack's core computational infrastructure in us-east-1 — the tier where message storage, fanout, and business logic lives. Fixing the core tier was the key to eliminating AZ-level blast radius for the majority of user-visible failures.
\n
\n\n\n✅
\nThe Independent Cell Deployment Dividend
Post-migration, Slack's oncall teams reported that bad deploys could be isolated to a single cell before being promoted to the full fleet. A canary that degrades performance in Cell A triggers an alert while Cells B and C continue healthy — giving the deploying team clear signal and a clean blast-radius boundary. Reliability and deployment safety turned out to be the same investment.
\n
Before the migration, Slack's core platform was a monolithic service topology with components distributed across AZs but not isolated within them. A request might be load-balanced to a webapp in AZ-1, which calls a backend service in AZ-3, which reads from a database in AZ-2. This cross-AZ traffic pattern meant that any AZ degradation could affect the latency of any request — the system had no natural blast-radius boundary at the AZ level. Post-migration, each cell contains a complete serving stack: its own webapp instances, its own backend service instances, and its own cache tier. Cross-cell traffic exists only for data that genuinely requires global consistency.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE CAP THEOREM AS A DEPLOYMENT GUIDE
\nEvery service Slack migrated required a CAP tradeoff decision. Cooper Bethea's QCon talk summarizes it clearly: services can choose to partition-tolerate with availability (serve potentially stale data from an isolated cell, keeping users active) or partition-tolerate with consistency (refuse to serve data if it can't verify it's current, protecting correctness). For Slack, the messaging delivery path chose availability — users can still send messages even if the cell is partitioned. Billing and auth services chose consistency — better to reject a request than to process it on stale data.
\n
\n\n\nℹ️
\nObservability Must Be Cell-Scoped
One of the operational requirements that emerged from the cellular migration was cell-scoped monitoring dashboards. Aggregated metrics across cells can hide cell-specific degradation — if Cell B is struggling but Cells A and C are fine, a global average error rate might look acceptable. Each cell needs its own metrics, alerts, and dashboards so operators can detect and act on cell-level issues without noise from the healthy cells masking the signal.
\n
\n\n\n📊
\nPer-Cell Metrics: The Observability Requirement
After cellularizing Slack's services, global aggregated metrics became misleading. A global p99 latency of 120ms might hide that Cell B has a p99 of 400ms while Cells A and C are at 90ms. Every cell now has its own dashboard, its own alerts, and its own error budget tracking. This per-cell observability investment was not optional — without it, the drain mechanism's input (operator judgment about which cell is degraded) would be unreliable.
\n
Slack's cellular architecture migration is a landmark case study in proactive reliability engineering: a team that experienced an incident, asked 'why did this affect users at all?', and then spent 18 months building the answer into the infrastructure.
\nWhat to remember
\n\n\nTHE COST OF CORRECTNESS
\nSome services at Slack could not be cleanly siloed because they require strong consistency across AZs — and consistency under partition is expensive. These services were migrated last, required the most architectural work, and ended up with more complex cell topologies (cross-cell replication, global coordination). This is the honest cost of CAP theorem reality: perfect AZ isolation is achievable for stateless services and very expensive for stateful ones. Acknowledge the cost early and budget accordingly.
\n
\n\n\n⚠️
\nMetastable States Can Emerge in Cell-Based Systems Too
Cellular architecture reduces blast radius but does not eliminate the risk of metastable failures. A cell-level cascade — where a single cell degrades in a self-sustaining way — is still possible. The AZ drain mechanism helps by allowing operators to route traffic away from a degraded cell, but the metastable failure patterns described in Slack's 2-22-22 incident postmortem can still occur within an individual cell. Defense in depth requires both architectural isolation and operational practices for cascade detection and recovery.
\n
\n\nSlack spent 18 months building a button so an operator could drain an entire data center in five minutes, which is either a lot of work for a button or exactly the right amount of work for that button.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-17T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.179907+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "Slack"]}, {"id": "https://techlogstack.com/explore/cloudflare-react-config-outage-2025/", "url": "https://techlogstack.com/explore/cloudflare-react-config-outage-2025/", "title": "Cloudflare Fixed a React Security Vulnerability and Broke the Entire Network", "summary": "How Cloudflare's rollout of a React security fix triggered a global killswitch bug that caused HTTP 500 errors across their network — the third configuration-related", "content_html": "Cloudflare · Reliability · 17 May 2026
\nIn late 2025, Cloudflare was rolling out a fix for a React security vulnerability. To do so, they needed to disable an internal testing tool with a global killswitch. The killswitch, unexpectedly, triggered a bug that sent HTTP 500 errors across Cloudflare's entire global network. This was the third major configuration-related global outage in two years.
\nBy December 2025, Cloudflare had experienced two major configuration-related global outages — the November 2023 Bot Management outage and various incidents in between — and had identified staged configuration rollouts as the primary systemic fix. That fix was still not fully implemented. Then came the React security vulnerability outage. Cloudflare was deploying a fix for a React CVE (a Common Vulnerabilities and Exposures report for a security flaw in the React JavaScript library — CVEs trigger mandatory patching workflows across the industry) in their internal tooling. The patch introduced an error in an internal testing tool. The team disabled the testing tool with a global killswitch. That killswitch, unexpectedly, triggered a bug in an unrelated code path — causing HTTP 500 errors across Cloudflare's network.
\n\nIn this latest outage, Cloudflare was burnt by yet another global configuration change. The previous outage in November happened thanks to a global database permissions change. This change would make it so that Cloudflare's configuration files do not propagate immediately to the full network, as they still do now. But making all global configuration files have staged rollouts is a large implementation that could take months. Evidently, there wasn't time to make it yet, and it has come back to bite Cloudflare.
— — The Pragmatic Engineer newsletter analysis of the Cloudflare December 2025 outage
The pattern was now impossible to ignore. Cloudflare had experienced multiple major outages in the 2023–2025 period, each with the same root-cause category: a configuration change that propagated globally and instantly, without staged rollout, caused unexpected systemic failures. The November 2023 Bot Management outage's primary action item — implement staged configuration rollouts — was explicitly identified as a large implementation that could take months. Each new outage was paying the price of that implementation not yet being complete. The React outage was the industry's most documented illustration of technical debt from unimplemented postmortem action items.
\n\n\n\nTHE KILLSWITCH THAT WASN'T JUST A KILLSWITCH
\nA killswitch is a simple concept: disable something. But in a complex distributed system, disabling one component can have unexpected dependencies. The internal testing tool that was disabled via global killswitch was apparently connected to a code path that, when the tool was absent, triggered a bug causing HTTP 500 errors. Killswitches are configuration changes. All the same rules apply: validate them, stage them, monitor them. A killswitch deployed globally and instantly is a global instant configuration change.
\n
Cloudflare was rolling out a fix for a React security vulnerability in internal tooling. The fix caused an error in an internal testing tool, prompting the team to disable the tool. The disable was executed as a global configuration change via killswitch.
\nThe global killswitch that disabled the testing tool unexpectedly triggered a bug in a connected code path. The bug caused HTTP 500 errors across Cloudflare's network. Because the killswitch was propagated globally and instantly, the impact was immediate and global.
\nThe fix was to revert the killswitch configuration — undoing the disable of the testing tool that had triggered the bug. This brought Cloudflare's network back to its pre-fix state. The React CVE patch then needed to be reworked to avoid triggering the testing tool error.
\nService was restored after reverting the configuration. The postmortem was published on the same day. CTO Dane Knecht acknowledged the pattern publicly and committed to making enhanced rollouts and versioning 'the first priority across the organization' — the same commitment made after the 2023 outages.
\n\n\n\n❌
\nThe Third Configuration-Related Outage in Two Years
The React security fix outage was the third major configuration-related global outage in Cloudflare's 2023–2025 period. The November 2023 Bot Management outage, subsequent incidents, and the December 2025 React outage all shared the same fundamental cause: a configuration change propagated globally and instantly without safety validation. The same fix had been identified after the first outage. That the fix hadn't been implemented by the third outage is a case study in the organizational cost of deprioritizing postmortem action items.
\n
\n\n\n⚛️
\nThe React vulnerability that started this chain of events was a security patch that Cloudflare was doing the right thing by deploying. Security vulnerability patching is mandatory and time-sensitive. The outage wasn't caused by bad intentions or negligence — it was caused by a security response that didn't account for all of its dependencies.
\n
One of the most challenging aspects of Cloudflare's staged rollout implementation is the security-versus-safety tension. Cloudflare's configuration distribution system was designed to be fast because security changes need to be fast. When a new attack pattern is detected, Cloudflare needs to push mitigation rules globally as quickly as possible. Slowing down configuration propagation has real security costs: the window between an attack being detected and the mitigation being globally deployed gets longer. The engineering challenge is building a system that can be fast for security-critical changes but staged for everything else — which requires distinguishing between change types at the infrastructure level.
\n\n\n\n⚠️
\nCTO Dane Knecht's Public Commitment
Following the December 2025 outage, Cloudflare CTO Dane Knecht was quoted in the postmortem: 'Global configuration changes rolling out globally remains our first priority across the organization.' This was the same commitment made after the 2023 outages. The public, repeated commitment to the same fix — without the fix having been implemented — created accountability that was difficult to ignore. The staged rollout project was given resources and deadline commitment following the December 2025 outage.
\n
\n\n\nℹ️
\nSame-Day Postmortem: The Third Time
Cloudflare published their postmortem for the December 2025 React outage on the same day the incident resolved — maintaining their remarkable transparency standard for the third major outage in two years. The postmortem's candor was notable: it explicitly referenced the November 2023 action item that hadn't been completed, and included CTO Dane Knecht's public acknowledgment that staged configuration rollouts 'remains our first priority.' Three same-day postmortems, three public commitments to the same fix, growing organizational accountability.
\n
\n\n\n🔄
\nThe Pattern: Configuration Changes That Break Things
Looking across Cloudflare's 2023–2025 incidents, a precise pattern emerges: (1) a routine operational change is made to production infrastructure, (2) the change has unexpected downstream effects, (3) the affected configuration or rule is propagated globally and instantly, (4) the impact is global and immediate. The fix to this pattern is not 'be more careful' — it's staged rollout infrastructure that makes global instant propagation impossible for non-security-critical changes.
\n
\n\n\nWHAT MAKES CLOUDFLARE'S CASE UNIQUE
\nMost organizations have configuration-related incidents. What makes Cloudflare's case unusual is the scale: a configuration change at Cloudflare affects infrastructure serving a significant fraction of all internet traffic. The blast radius is not one company's systems — it's millions of websites and their users globally. This scale makes configuration safety not just an operational concern but a responsibility to the broader internet ecosystem. Cloudflare's staged rollout implementation is infrastructure for global internet resilience.
\n
Cloudflare's CTO described the required fix as 'Enhanced Rollouts and Versioning' — applying the same safety and blast mitigation features to configuration data that Cloudflare already applies to software deployments. Software at Cloudflare is deployed gradually, with strict health validation at each stage. Configuration changes had no equivalent safety system. The fix required building one: a configuration versioning system that could tag changes, a rollout engine that could apply them to staged percentages, and health checks that could catch problems before wider propagation.
\n# The required Enhanced Rollouts and Versioning system\n# Differentiates security-critical changes (fast) from configuration changes (staged)\n\nclass ConfigRolloutEngine:\n def deploy_change(self, change: ConfigChange):\n # Security-critical changes (DDoS mitigations, attack signatures)\n # Still fast — but with validation gate\n if change.type == ConfigChangeType.SECURITY_CRITICAL:\n self._validate_config(change) # validation must pass\n self._deploy_global_fast(change) # then deploy fast\n return\n \n # All other changes: staged rollout with health gates\n self._validate_config(change)\n \n # Stage 1: 1% canary\n self._deploy_to_percentage(change, pct=0.01)\n self._wait_and_check_health(minutes=5)\n \n # Stage 2: 10% cohort \n self._deploy_to_percentage(change, pct=0.10)\n self._wait_and_check_health(minutes=5)\n \n # Stage 3: 50% cohort\n self._deploy_to_percentage(change, pct=0.50)\n self._wait_and_check_health(minutes=10)\n \n # Stage 4: Full rollout\n self._deploy_global(change)\n \n def _validate_config(self, change: ConfigChange):\n # Size limits, schema validation, semantic checks\n # Catches the oversized ClickHouse fallback config\n # Catches malformed configs before any propagation\n pass\n \n def _wait_and_check_health(self, minutes: int):\n # Error rate, latency, traffic metrics\n # Auto-rollback if thresholds exceeded\n pass\n\n\nTHE SECURITY-SPEED TENSION
\nThe core tension in Cloudflare's configuration safety problem is that their configuration system was designed for security use cases where speed matters. Staged rollouts introduce latency that's unacceptable for DDoS mitigation rules. The solution requires distinguishing between change types: security responses (fast propagation + validation) versus configuration updates (staged propagation + health gates). This distinction is architecturally complex — the system needs to know the change type, enforce the right deployment mode, and maintain separate pipelines without creating a new single point of failure.
\n
\n\n\n✅
\nThe Three-Outage Forcing Function
If the staged rollout implementation had been deprioritized after the November 2023 outage, the December 2025 outage provided an undeniable forcing function. Three configuration-related global outages in two years, with the same root cause, creates organizational pressure that cannot be managed with further prioritization discussions. The December 2025 outage finally resulted in resources, deadline commitment, and executive ownership for the staged rollout project.
\n
\n\n\n⚠️
\nPostmortem Action Items Need Priority Enforcement
The Cloudflare staged rollout story is one of the industry's clearest examples of what happens when postmortem action items are treated as backlog items rather than critical debt. The November 2023 postmortem identified the fix. The December 2025 outage demonstrated the cost of not implementing it. Engineering organizations need mechanisms to track postmortem action items with urgency, not just completeness — including escalation paths when critical action items age without progress.
\n
\n\n\n✅
\nResources Finally Allocated After Three Incidents
The December 2025 outage served as the organizational forcing function that earlier incidents hadn't fully achieved. Following the third configuration-related global outage in two years, Cloudflare allocated dedicated engineering resources, a named project lead, and a committed delivery timeline for the Enhanced Rollouts and Versioning system. The system is now being built as production infrastructure rather than a backlog item.
\n
\n\n\n⚠️
\nThe Security-Critical Fast Path
One of the hardest engineering problems in the staged rollout system is the security-critical fast path. When Cloudflare detects a new DDoS attack pattern or zero-day exploit, they need to push mitigations to every PoP globally within seconds — not within the staged rollout window of 30+ minutes. The system must distinguish at the protocol level between security-critical changes (which maintain fast propagation) and configuration updates (which go through staged rollout). Building this distinction correctly — without creating a bypass that regular configuration changes can be misclassified into — is the core engineering challenge.
\n
The React outage sits in a chain of failures that reveals a systemic architectural vulnerability in Cloudflare's control plane. At the data plane level — PoPs, traffic routing, DDoS mitigation — Cloudflare's architecture is highly resilient. At the configuration plane level — the system that distributes rules and settings to the data plane — the architecture was designed for speed rather than safety. Three outages in two years from the same root cause is the empirical evidence that speed without safety is not viable at global infrastructure scale.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTHE ORGANIZATIONAL LESSON: ACTION ITEMS NEED OWNERS
\nCloudflare's staged rollout work was identified as a priority after three separate incidents. Each time, it was described as a large implementation requiring months. In hindsight, the organizational failure was not the identification — it was the lack of a named owner with authority, resources, and a committed deadline. Postmortem action items without named owners, resource allocation, and deadline accountability often age in backlogs until a subsequent incident forces the conversation again.
\n
\n\n\nℹ️
\nCloudflare's Transparency as Industry Standard
Despite three major outages with related root causes, Cloudflare's consistent same-day postmortem publication is widely recognized as an industry best practice. The transparency builds trust even when the incidents themselves erode it. Companies that publish honest postmortems attract and retain engineers who want to learn from failures, and they establish accountability mechanisms that internal-only postmortems don't create. The public commitment to fixing staged rollouts after the December 2025 outage has an accountability dimension that an internal action item does not.
\n
\n\n\nℹ️
\nCloudflare's Scale Makes the Problem Harder
Staged configuration rollout at Cloudflare's scale (300+ PoPs, millions of configuration updates per year, microsecond-sensitive security decisions) is genuinely difficult infrastructure engineering. The problem is not that Cloudflare doesn't know how to build staged rollouts — they already do this for software deployments. The problem is retrofitting staged rollout semantics onto a configuration distribution system that was designed for a different set of requirements (fast propagation, consistency, global reach) without disrupting the security use cases that depend on that speed.
\n
The React security fix outage is the third chapter in a two-year story about the cost of not completing a known critical infrastructure fix. The lessons are organizational as much as technical.
\nWhat to remember
\n\n\nTHE TRANSPARENCY COMPOUNDING EFFECT
\nCloudflare's pattern of same-day postmortem publication for major incidents has created a compounding transparency dividend: each postmortem increases customer trust, each public commitment creates accountability, each incident with the same root cause raises the organizational urgency. The third outage with the same root cause forced a resource and timeline commitment that the first and second outages hadn't achieved. Transparency accelerates accountability.
\n
\n\n\n⚠️
\nTesting Infrastructure for Operational Safety Changes
The React CVE fix that started this chain of events was a security response — the right thing to do. But deploying it through a testing tool that hadn't been validated for that specific change created the downstream error. Operational safety infrastructure (testing tools, killswitches, monitoring systems) needs the same testing rigor as application code. When safety infrastructure fails, it often does so during incidents — exactly the moment it's needed most.
\n
\n\nCloudflare fixed a React security vulnerability and accidentally broke the global internet, which is both very on-brand for React and a reminder that security patches are just change management with higher stakes.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-17T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.183704+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "Cloudflare"]}, {"id": "https://techlogstack.com/explore/discord-cassandra-scylladb-2022/", "url": "https://techlogstack.com/explore/discord-cassandra-scylladb-2022/", "title": "How Discord Migrated Trillions of Messages and Fired Their Garbage Collector", "summary": "How Discord's engineering team eliminated JVM GC pauses, cut their database fleet from 177 to 72 nodes, and migrated 4 trillion messages in 9 days.", "content_html": "Discord · Databases · 17 May 2026
\nIt is 2022 and Discord's on-call engineers are babysitting a 177-node database cluster, manually rebooting nodes after Java GC pauses spiral out of control. The system holding every message ever sent is becoming the thing everyone fears touching most.
\n\nOur Cassandra cluster exhibited serious performance issues that required increasing amounts of effort to just maintain, not improve.
— — Bo Ingram, Senior Software Engineer — via Discord Engineering Blog
Discord launched in 2015 with a mission to build the best voice and text chat platform for gamers. By 2017 they had outgrown MongoDB and migrated their entire message store to Apache Cassandra (a distributed wide-column NoSQL database designed for high availability across many nodes without a single point of failure). Cassandra's promise was compelling: write anywhere, replicate everywhere, scale horizontally forever. For a few years it held. By 2022, however, the promises had curdled into a maintenance nightmare that consumed engineering cycles every single week. The database cluster had grown to 177 nodes holding trillions of messages, and keeping it alive required the kind of expertise and vigilance that should be reserved for nuclear reactor operators, not chat app engineers.
\n\n\n\n🔥
\nAt peak, Discord's Cassandra cluster required engineers to manually reboot individual nodes after JVM GC pauses (Java Virtual Machine garbage collection — periodic stop-the-world pauses where the JVM freezes all threads to reclaim memory) spiraled long enough to drop the node from the cluster. This was not a rare emergency — it was routine on-call work.
\n
The core problem was architectural. Cassandra is written in Java, and Java's garbage collector periodically halts all threads in the JVM to reclaim heap memory — a moment engineers call a stop-the-world pause. Under Discord's workloads, these pauses could last long enough to cause cascading latency spikes visible to users, and in severe cases, the JVM's consecutive GC pauses got so bad that a node would effectively fall out of the cluster entirely. An on-call engineer would then have to manually reboot it and babysit it back to health. The p99 latency on historical message reads ranged between 40 and 125 milliseconds depending on whether compaction was running — an unpredictability that made SLO planning impossible. Every time someone tried to improve the cluster rather than merely maintain it, they risked triggering a cascade.
\nDiscord's message data model organized messages by channel ID and a fixed time window called a bucket (a fixed time slice, e.g. 10 days, used as part of the partition key so messages are spread across multiple Cassandra partitions rather than one per channel). This was efficient for write distribution and replication, but created a painful read problem. Cassandra performs writes cheaply by appending to a commit log (a sequential on-disk journal where writes are recorded before being applied to the in-memory structure, enabling fast writes at the cost of read complexity) and an in-memory structure called a memtable (an in-memory write buffer in Cassandra that is flushed to disk as SSTables when it fills up). Reads, however, must query the memtable and potentially multiple SSTables (Sorted String Tables — immutable on-disk files in Cassandra that hold flushed memtable data, which must all be merged on read to reconstruct the current value), a dramatically more expensive operation. When a popular Discord server made a major announcement and thousands of users simultaneously opened their apps to read it, every single one of those reads would hammer the same partition. The cluster called these hot partitions, and they were Discord's most common and painful operational incident.
\n\nBy early 2022, Discord's on-call rotation was spending more time nursing Cassandra than building features. GC pause alerts fired multiple times a week, and the p99 latency on reads ranged from 40ms to 125ms depending on whether compaction was running on the affected node — an unpredictability engineers had simply learned to live with.
\nThe root cause split into two layers: JVM garbage collection (Java's memory management system that periodically pauses all threads to reclaim heap memory — in large heaps, these pauses could last hundreds of milliseconds) on write-heavy nodes created latency cliffs, while Cassandra's read path — requiring merges across multiple SSTables — meant any popular channel partition would spike latency under concurrent user load. The combination made the cluster inherently unpredictable at scale.
\nDiscord chose ScyllaDB, a Cassandra-compatible database rewritten in C++ with a shard-per-core architecture (a design where each CPU core is assigned its own exclusive subset of data and handles requests independently, avoiding cross-core coordination and lock contention). They also built a Rust-based data services layer between the API and the database to absorb hot-partition spikes via request coalescing. The migration tool was rewritten in Rust to achieve 3.2 million records per second transfer speed.
\nThe migration completed in nine days — the original estimate using ScyllaDB's Spark migrator had been three months. The cluster footprint shrank from 177 nodes to 72, each ScyllaDB node running with 9 TB of disk versus the average 4 TB on Cassandra. P99 latency for historical reads settled at a stable, predictable 15 milliseconds.
\n\n\n\n⚠️
\nWhy cassandra-messages Was the Last to Move
By 2020 Discord had migrated every other database to ScyllaDB — the messages cluster was the lone holdout. They deliberately waited to last because it was the most critical dataset: trillions of messages, nearly 200 nodes, and the one cluster whose failure would be immediately visible to every user. They used the other migrations to tune ScyllaDB for their access patterns first, including filing and waiting on performance improvements to ScyllaDB's reverse query support.
\n
The migration nearly ended in drama rather than triumph. After running the Rust migrator for days at 3.2 million records per second, the progress bar hit 99.9999% — and stopped. The migrator was timing out trying to read the last few token ranges (in Cassandra, data is distributed across the ring by assigning each partition a hash token, and a token range is a contiguous slice of that ring assigned to a node) because they contained gigantic ranges of tombstones (deletion markers in Cassandra — when data is deleted, a tombstone is written instead of the row being removed, because immutable SSTables cannot be modified in-place; these tombstones must be read and skipped during every subsequent read until compaction removes them) that had never been compacted away. Engineers had to manually trigger compaction on that token range; seconds later, the migration hit 100%. Automated data validation confirmed correctness by sending a sample of reads to both databases and comparing results. Discord switched to ScyllaDB in May 2022.
\n\n\n\nTHE WORLD CUP TEST
\nThe real stress test came months after go-live: the 2022 FIFA World Cup Final between Argentina and France. Every goal by Messi, every equalizer by Mbappé, every moment in the shootout created a massive spike of simultaneous message reads across Discord's biggest servers. Under the old Cassandra architecture this would have triggered hot-partition alerts and cascading latency. Under ScyllaDB with the Rust data services layer, the monitoring dashboards showed nothing unusual. The system held flat through 120 minutes of football and a penalty shootout.
\n
The Rust data services layer was the architectural insight that made ScyllaDB viable, not just the database choice alone. When a popular server makes an announcement and thousands of users simultaneously open their clients, all those read requests arrive at the data service within milliseconds of each other — all asking for the same messages in the same channel. Without coalescing, each request would hit the database separately, creating a hot partition. With request coalescing (a pattern where the first incoming request for a piece of data triggers an active lookup, and all subsequent requests for the same data subscribe to that lookup's result rather than issuing their own query, reducing N database hits to 1), only one query goes to ScyllaDB; every subsequent request subscribes to the in-flight result and receives the answer when the single database query returns. The data services layer also used consistent hashing (a ring-based routing scheme where each data service instance is responsible for a specific subset of channel IDs, ensuring all requests for a given channel are routed to the same service instance to maximize coalescing effectiveness) to route requests for the same channel to the same service instance, maximizing coalescing opportunity.
\n\nThe fix had three distinct components, and Discord was deliberate about not rushing any of them. First, they spent years migrating every other database to ScyllaDB to build operational expertise before touching the one cluster that mattered most. Second, they collaborated with the ScyllaDB team to improve reverse query performance — a blocker they hit in early testing — and waited until that was production-grade before proceeding. Third, they built the Rust data services layer before starting the migration, so the new database would go live already protected from hot-partition load patterns. This sequencing was the engineering discipline that made the migration look easy in retrospect.
\nThe turning point in the migration timeline was a one-day engineering sprint. ScyllaDB's off-the-shelf Spark migrator (an Apache Spark-based tool provided by ScyllaDB for bulk data migration that reads token ranges from Cassandra and writes them to ScyllaDB) estimated three months to move the message data — three months of dual-running two massive database clusters, three months of operational complexity, and three months of potential failure modes. Bo Ingram decided that was three months too long. He and two colleagues rewrote the migrator in Rust in a single day. The new migrator read token ranges from a database, checkpointed them locally via SQLite for crash recovery, and fired them into ScyllaDB as fast as possible. The result: 3.2 million records per second. The new estimate was nine days, and the team chose to do a single-flip cutover instead of a phased time-based approach entirely.
\n// Simplified version of Discord's request coalescing logic in the Rust data service\n// Real implementation uses Tokio async runtime\n\nuse std::collections::HashMap;\nuse tokio::sync::broadcast;\n\nstruct CoalescingDataService {\n // Map from cache_key -> active broadcast sender\n // If a task is in flight, subscribers receive the result\n in_flight: HashMap>,\n}\n\nimpl CoalescingDataService {\n async fn get_messages(\n &mut self,\n channel_id: u64,\n before_id: u64,\n ) -> Result> {\n // Build a stable cache key for this exact query\n let key = format!(\"{}:{}\", channel_id, before_id);\n\n if let Some(sender) = self.in_flight.get(&key) {\n // A query for this channel is already in flight\n // Subscribe and wait — NO second database round-trip\n let mut rx = sender.subscribe();\n return Ok(vec![rx.recv().await?]); // receive the shared result\n }\n\n // No existing task — we are the first; create the broadcast channel\n let (tx, _rx) = broadcast::channel(16);\n self.in_flight.insert(key.clone(), tx.clone());\n\n // Execute the single database query to ScyllaDB\n let results = self.scylladb.query_messages(channel_id, before_id).await?;\n\n // Broadcast to ALL waiting subscribers at once\n let _ = tx.send(results.clone()); // every subscriber wakes up\n self.in_flight.remove(&key); // clean up the in-flight tracker\n\n Ok(results)\n }\n} \n\n\nℹ️
\nThe SuperDisk: Custom Hardware for Cloud Durability
ScyllaDB is optimized for NVMe SSDs (Non-Volatile Memory Express solid-state drives — extremely fast local storage that dramatically reduces I/O latency) but in cloud environments NVMe is ephemeral — a node restart wipes the disk. Discord engineered a custom RAID 1 configuration they called the Superdisk: writes go to both fast local NVMe and slower persistent network-attached storage simultaneously; reads prefer the NVMe for speed. This gave them NVMe-level read performance with cloud-level data durability.
\n
\n\n\n✅
\nZstandard Compression: 50–60% Disk Reduction
Alongside the database migration, Discord enabled Zstandard compression on their ScyllaDB tables. Message data compresses extremely well. The result was a 50–60% reduction in raw disk usage compared to uncompressed Cassandra storage — effectively giving each physical node far more useful capacity at zero hardware cost.
\n
\n\n\nTHE VALIDATION STRATEGY
\nDiscord ran automated correctness validation throughout the migration by sending a small percentage of reads to both databases simultaneously and comparing results. Only when reads matched across Cassandra and ScyllaDB was a partition considered successfully migrated. This shadow-read approach caught data inconsistencies without any user-visible impact, and gave the team confidence to flip the cutover switch as a single atomic event rather than a long, hedged transition.
\n
Cassandra vs ScyllaDB at Discord: Before and After Migration
| Metric | Cassandra (Before) | ScyllaDB (After) |
|---|---|---|
| Cluster Nodes | 177 | 72 |
| Disk per Node (avg) | 4 TB | 9 TB |
| p99 Read Latency | 40–125ms (variable) | ~15ms (stable) |
| GC Pauses | Frequent stop-the-world | None (C++, no GC) |
| Hot Partition Risk | High — no coalescing | Mitigated by Rust data services |
| On-Call Toil | Weekly node babysitting | Dramatically reduced |
Before the migration, Discord's message write and read path ran through a monolithic API server directly into the Cassandra cluster. There was no intermediary — every user action that read messages translated directly into a database query, with no protection against fan-out or hot partition amplification (when many users simultaneously request data stored in the same database partition, causing that node to receive far more traffic than its neighbors, creating latency spikes and potential instability). The API server held connection pools to Cassandra, handled CQL queries (Cassandra Query Language — a SQL-like interface for querying Cassandra) for message pagination, and relied on Cassandra's own internal mechanisms (memtable, SSTables, compaction) to handle read pressure. Under normal load this worked. Under peak load — a major announcement, a viral moment, a World Cup Final — it did not.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nSHARD-PER-CORE ARCHITECTURE
\nThe fundamental reason ScyllaDB handles concurrent reads so much better than Cassandra is its shard-per-core architecture. Each CPU core is assigned its own exclusive slice of the data and handles all requests for that data without coordination with other cores. In Cassandra's JVM-based model, all threads compete for heap memory under a single garbage collector. In ScyllaDB's C++ model, each core is an independent actor: no cross-core locking, no GC, no stop-the-world. When one partition gets hot, it affects only the core assigned to that shard — it cannot cascade to neighbors.
\n
\n\n\nℹ️
\nConsistent Hashing: Routing Channels to Service Instances
Each Rust data service instance is responsible for a deterministic subset of channel IDs via consistent hashing (a routing scheme where each channel_id is mapped to a specific service instance using a hash ring, so all requests for channel #12345 always go to Data Service Instance B — maximizing the chance that an in-flight coalescing task for that channel already exists). This means if 1,000 users simultaneously load the same popular channel, all 1,000 requests arrive at the same service instance and collapse into one database query.
\n
View interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\n🦀
\nWhy Rust for the Data Services Layer
Discord chose Rust for data services because it offered C-level throughput with memory safety guarantees that prevent entire classes of concurrency bugs common in C++ — exactly what you want in a layer handling millions of concurrent subscribers. The Tokio async runtime gave them non-blocking I/O without the GC overhead that had plagued their Cassandra setup. As Bo Ingram noted with characteristic candor: it also let them say they rewrote it in Rust.
\n
Discord's migration took years of preparation and nine days of execution. The long preparation was not waste — it was the reason the execution was clean. The lessons here are as much about sequencing and courage as they are about database choice.
\nWhat to remember
\n\n\n✅
\nThe World Cup Validation
The 2022 FIFA World Cup Final was Discord's unplanned load test — and the system passed cleanly. Every goal, every save, every penalty created message spikes across thousands of servers simultaneously. The combination of ScyllaDB's shard-per-core architecture and Rust data services coalescing kept latency flat through all 120 minutes plus penalties. No hot partition alerts. No on-call pages. No post-match war rooms.
\n
\n\n\nSHADOW READ VALIDATION
\nDiscord's validation strategy during migration was elegantly simple: send a small percentage of reads to both Cassandra and ScyllaDB simultaneously, compare results automatically, and flag any discrepancy. This meant correctness was continuously verified during the nine days of data transfer — not checked at the end in a tense manual review. Any database migration touching production data should implement this pattern before flipping the final switch.
\n
\n\nThey migrated four trillion messages in nine days, and the most stressful moment was the progress bar stopping at 99.9999% — because even tombstones refuse to die quietly.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-17T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.396336+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Databases", "Discord"]}, {"id": "https://techlogstack.com/explore/discord-cloud-dev-environments-2023/", "url": "https://techlogstack.com/explore/discord-cloud-dev-environments-2023/", "title": "Discord Killed the MacBook Dev Environment and Never Looked Back", "summary": "Discord's journey from MacBook chaos to cloud development environments — two migrations, a Kubernetes dead-end, and how Tailscale finally made remote dev feel local.", "content_html": "Discord · Reliability · 17 May 2026
\nDiscord's engineering team had tripled in size and was drowning in a swamp of 'works on my machine' bugs — some engineers running macOS, some Ubuntu, all of them slowly. The solution was radical: no one gets a local dev environment anymore.
\n\n\n\n🖥️
\nDiscord's engineering organization tripled in size over a few years, and the Internal Developer Experience team was spending more time debugging niche, unreproducible, engineer-specific environment issues than actually improving the developer toolchain. The same code would fail on one MacBook and pass on another — and the DevEx team had no systematic way to fix that.
\n
For most of Discord's early history, backend engineers set up development environments on their personal laptops — primarily MacBooks, but some preferred Ubuntu. This dual-environment world was manageable when the engineering team was small and most people sat near each other in San Francisco. As Discord's product grew and the engineering organization tripled in headcount, the cracks became structural failures. Homebrew (a popular package manager for macOS that installs open source software but lacks guarantees of reproducibility across machines) upgrades would silently break an engineer's dev setup. A new team member would spend their first week not shipping code but untangling environment issues unique to their laptop. The tooling team accumulated a growing backlog of one-off tickets that amounted to: your environment is subtly different from everyone else's, and we have to figure out why. There was no single source of truth for what a correct Discord development environment looked like.
\nThe solution the DevEx team landed on was radical in its simplicity: stop maintaining two local environments and move all backend and infrastructure development to a single Linux-based Cloud Development Environment (a remote machine running in a cloud provider's data center that developers access via their editor's remote extension, giving them full Linux capabilities without managing local hardware). Discord evaluated Coder (an open-source platform for creating and managing cloud development environments at scale, providing templated workspace provisioning, lifecycle management, and developer access controls) in late 2020 — a natural fit given that Coder's team were avid Discord users. The alignment was easy: Discord was already a heavy Kubernetes user, and Coder's V1 product was entirely Kubernetes-native. The partnership began, and Discord started the first of what would turn out to be two separate migrations.
\n\nAs Discord's engineering organization tripled in size, the DevEx team found itself firefighting unreproducible environment issues specific to individual MacBooks. Brew upgrades broke setups silently. Ubuntu engineers had subtly different library versions. No environment was truly identical to another, making debugging a game of 'is this a code bug or a local environment bug?'
\nThe SDLC (Software Development Lifecycle — the full process from writing code to shipping it, including build, test, and deployment) had grown too complex for unmanaged local environments. Discord's backend requires a highly complex environment with many moving parts — running it inside Sysbox (a container runtime that allows running full operating systems inside Docker containers by emulating kernel features) on Kubernetes V1 introduced layers of virtualization that were difficult to debug when things went wrong.
\nThe initial Coder V1 migration moved engineers to Kubernetes-based container environments. Networking latency and frequent disconnections plagued engineers outside San Francisco. In 2023, Discord migrated to Coder V2, which replaced the Kubernetes-container model with full VMs using Tailscale and WireGuard for networking — dramatically more stable and performant.
\nAfter V2, Discord stopped receiving support tickets and questions about high latency and connection drops entirely. Engineers reported that development felt faster and smoother. The DevEx team stopped spending time on 'works on my machine' debugging and started spending time on tooling improvements that actually moved the needle.
\n\n\n\n⚠️
\nThe V1 Kubernetes Problem: Layers on Layers
Coder's V1 product ran development environments in Docker containers orchestrated by Kubernetes. Discord quickly found that developing inside Sysbox containers on Kubernetes introduced so many layers of virtualization — container runtime, kernel emulation, cloud networking — that debugging environment failures became genuinely difficult. When something broke, the question was always: is this a Discord bug, a Coder bug, a Kubernetes bug, or a network issue? The layers made attribution nearly impossible and resolution slow.
\n
The V1 to V2 migration fixed the worst problems. Coder's V2 abandoned the Kubernetes-container model and delivered full virtual machine (a software emulation of a complete computer that runs inside a cloud provider's physical host, giving tenants full OS access and eliminating container layering complexity) provisioning instead, giving engineers direct access to the Linux host without the indirection of container runtimes. The networking stack was rewritten to use Tailscale and WireGuard — a mesh VPN (a peer-to-peer virtual private network where devices connect directly to each other rather than through a central gateway, reducing latency and eliminating bottlenecks) approach where developer machines connect directly to their cloud VMs via encrypted tunnels. The combination of VM simplicity and direct WireGuard networking eliminated the latency and stability issues that had made V1 frustrating for engineers outside the Bay Area office.
\n\n\n\nTHE PANDEMIC TIMING ADVANTAGE
\nDiscord began its CDE migration in late 2020 — just as the pandemic forced most tech companies to distribute their engineering teams globally. Because Discord had already committed to the cloud dev environment path, they were better positioned than most to operate as a fully distributed engineering organization. Engineers could spin up identical development environments from anywhere in the world without IT shipping them a configured MacBook. The organizational investment paid off in ways the team had not initially anticipated.
\n
One pragmatic concession emerged: frontend engineers who worked heavily with large HTML and JavaScript files found that the network overhead of transferring those assets during live editing created noticeable latency in their save-and-rebuild loops. The DevEx team made a deliberate exception — frontend work was excluded from the mandatory CDE migration, with those engineers continuing to develop locally on their MacBooks. This was not a failure of the approach; it was an honest acknowledgment that different workloads have different locality requirements, and optimizing for 100% consistency at the cost of 30% of engineers' daily experience is not engineering, it's ideology.
\n\n\n\n📁
\nThe /home Directory Persistence Strategy
One of Discord's key architectural decisions for developer experience was keeping the /home directory persistent across VM restarts and template updates. Engineers could update templates and base images without losing their repositories, settings, personal tools, and workspace customizations. This made CDEs feel like a machine that belonged to them rather than an ephemeral container that could be wiped at any time.
\n
\n\n\n❌
\nThe Communication Gap They Would Fix
Despite all-hands announcements, early signaling, and a thorough beta period, Discord's DevEx team acknowledged they could have communicated the migration better. Engineers discovered gaps in documentation only after cutover — a pattern common in large infrastructure migrations. Requesting hundreds of engineers to overhaul their entire development workflow is a major ask, and the team would invest more in change management communications next time.
\n
\n\nDespite the challenges and the need for two migrations (Mac→V1→V2), our move to remote dev machines using Coder has been remarkably successful. The timing was fortuitous, as we embarked on this journey before the pandemic began.
— — Denbeigh Stevens, Senior Software Engineer — via Discord Engineering Blog
Discord's CDE story is not a clean one-migration success. It required two complete migrations: from MacBooks to Coder V1 (Kubernetes), then from Coder V1 to Coder V2 (VMs). This is worth naming directly because it reframes the story from 'we had a good idea and executed it' to 'we had a good idea, hit a wall, and had the organizational courage to do it again better.' The decision to rebuild entirely on VMs rather than patch the Kubernetes architecture was the right engineering call, and it required Discord to invest in yet another migration cycle even as engineers were still adjusting to the first one.
\n\n\n\nWHY VMS BEAT KUBERNETES CONTAINERS FOR DEV ENVIRONMENTS
\nKubernetes is excellent for production workloads but creates friction as a development environment host. Running in containers adds layers of virtualization that complicate debugging, restrict host access needed for development tooling, and create networking abstractions that can introduce latency. VMs eliminate this friction: engineers get a real Linux machine with full host access, simpler networking, and predictable behavior. Coder V2's choice to move to VMs was the architectural insight that made Discord's CDE program successful.
\n
# Simplified Coder V2 workspace template (Terraform)\n# Discord provisions Linux VMs for each engineer via this kind of template\n\nresource \"coder_workspace\" \"discord_backend\" {\n # Each engineer gets their own dedicated VM\n name = \"${data.coder_workspace.me.name}-backend\"\n\n # VM provisioned in cloud, not a container\n instance_type = \"n2-standard-8\" # 8 vCPU, 32GB RAM\n disk_size_gb = 100\n\n # /home is persistent — survives template updates and restarts\n # Engineers keep their repos, configs, and customizations\n persistent_home = true\n\n # Tailscale/WireGuard handles secure network tunnel\n # Engineer's laptop <--> VM via encrypted peer-to-peer mesh\n network = \"tailscale-mesh\"\n\n # Standard Discord development image\n image = \"discord/devenv:latest\"\n\n # IAM via cloud provider — no VPN required\n service_account = \"dev-env-sa@discord-dev.iam.gserviceaccount.com\"\n}\n\n# VS Code remote extension connects to this VM\n# Engineers experience it as if it were a local machine\n\n\nℹ️
\nChampions: The Adoption Accelerator
Discord explicitly recruited CDE champions from across engineering departments — engineers who were enthusiastic about the new tooling and willing to beta-test it early. These champions provided a diversity of feedback from different daily development loops (backend, infrastructure, mobile), helping the DevEx team surface issues that a homogeneous beta group would have missed. Identifying and empowering internal champions is one of the most effective change management tactics for large-scale developer tooling migrations.
\n
\n\n\n✅
\nImmutability: The End of 'Works on My Machine'
The structural benefit of cloud development environments is immutability and reproducibility. Every engineer's environment is provisioned from the same base image and template. When the DevEx team fixes a bug or adds a tool, it ships to everyone simultaneously via image update. No more per-engineer debugging sessions. No more Homebrew version drift. No more 'what's your local Python version?' — the question itself ceases to be meaningful.
\n
The remaining engineering challenge after V2 was building better tooling to understand network conditions under different global setups. Discord's engineers are distributed across the US and internationally, and while Tailscale/WireGuard dramatically improved average-case latency, the DevEx team admitted they didn't build enough diagnostics during migration to understand the worst-case network experience. Those tools came after go-live rather than before — a gap they noted they would prioritize earlier next time. The lesson is subtle: when you migrate to a networked development environment, network observability for the developer experience layer must be treated as a first-class requirement, not an afterthought.
\n\n\n\n🔄
\nTemplate Updates Without Workspace Rebuilds
One of the most underappreciated features of Discord's CDE setup is the ability to update base templates and images without requiring engineers to rebuild their workspaces from scratch. Because the /home directory is persistent, an image update ships the new tooling to every engineer's VM while leaving their repositories, configurations, and in-progress work completely untouched. This makes infrastructure maintenance feel less like a forced migration and more like an automatic software update.
\n
View interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nTAILSCALE + WIREGUARD: WHY THIS NETWORKING WON
\nTraditional VPN solutions route all traffic through a central gateway — every packet from a distributed engineer's laptop goes hub-and-spoke to HQ before reaching the dev VM. Tailscale's mesh networking routes traffic directly peer-to-peer between the engineer's machine and their cloud VM via encrypted WireGuard (a modern, minimal VPN protocol built directly into the Linux kernel, providing lower latency than OpenVPN or IPSec with a drastically smaller codebase) tunnels. For globally distributed engineers, the difference in feel is enormous: direct-path WireGuard feels like working on a local machine; gateway VPNs feel like working through treacle.
\n
\n\n\nℹ️
\nFrontend Exception: A Pragmatic Carve-Out
Not all workloads are equal in a networked dev environment. Discord's frontend engineers work with large HTML/JS asset bundles whose save-and-rebuild loop requires frequent large file transfers across the network. The latency was noticeable enough to hurt developer experience, so frontend development was explicitly kept on local machines. This was not a failure — it was an honest architectural boundary that preserved developer happiness for a workload where locality genuinely mattered.
\n
\n\n\n📦
\nImmutability as Infrastructure
By running development environments as templated VMs rather than managed laptop configurations, Discord transformed dev environment maintenance from a support burden into an infrastructure deployment problem — and infrastructure deployment is something engineering teams know how to do well. Template updates ship like container image updates. Rollbacks are possible. Every engineer gets the same foundation, and deviations from it are explicit configuration, not accidental drift.
\n
\n\n\n⚠️
\nThe V1 Sysbox Trap: When Container-in-Container Breaks
Coder's V1 used Sysbox containers to simulate a full Linux environment inside Kubernetes pods. Developing Discord's complex backend required running Docker containers inside those containers — a level of nesting that Sysbox enabled but made debugging treacherous. When something broke, the failure could originate in the application code, the container runtime, Sysbox's kernel emulation, the Kubernetes networking layer, or the cloud provider. Four layers of virtualization made every incident investigation significantly harder than it needed to be.
\n
Discord migrated their entire engineering development environment twice in three years. The first migration was necessary; the second was a correction. Both were worth doing. Here is what the journey teaches engineers who are considering similar moves.
\nWhat to remember
\n\n\n✅
\nThe Unexpected Pandemic Dividend
Discord began the CDE migration in late 2020 — right as the pandemic forced most tech companies to scramble for distributed-work solutions. Because Discord had already invested in cloud dev environments, their engineers could work from anywhere in the world with the same environment as their colleagues. What looked like an infrastructure investment turned out to be a resilience investment too.
\n
\n\n\nTHE COST OF DOING IT TWICE
\nDiscord needed to migrate twice because the first migration solved the wrong problem at the infrastructure level. Kubernetes containers gave them cloud hosting but not true machine equivalence. VMs gave them the latter. The lesson: validate your CDE architecture on a small cohort of power users before committing the whole organization, with explicit evaluation criteria around host access, networking latency, and debuggability — not just 'it runs Linux.'
\n
\n\nThey asked 'what if we just gave every engineer the same Linux box in the cloud?' and then had to do it twice before the cloud box stopped lying about its Wi-Fi signal.
TechLogStack — built at scale, broken in public, rebuilt by engineers
This case is a plain-English retelling of publicly available engineering material.
\nRead the full case on TechLogStack → (interactive diagrams, source links, and the full reader experience).
", "date_published": "2026-05-17T00:00:00+00:00", "date_modified": "2026-06-13T18:53:04.416552+00:00", "authors": [{"name": "TechLogStack Editorial"}], "tags": ["Reliability", "Discord"]}, {"id": "https://techlogstack.com/explore/netflix-maestro-workflow-2024/", "url": "https://techlogstack.com/explore/netflix-maestro-workflow-2024/", "title": "Netflix Hit the AWS Instance Ceiling and Built a Workflow Engine That Scales Forever", "summary": "How Netflix hit the vertical scaling ceiling on its Meson workflow orchestrator and built Maestro — a horizontally scalable system handling 2 million jobs a day.", "content_html": "Netflix · Distributed Systems · 17 May 2026
\nNetflix's Meson orchestrator was handling hundreds of thousands of daily data and ML jobs — and running out of machine. Vertically scaling on AWS had a hard ceiling, and the workflows were doubling in size every year. The only way out was a complete architectural rethink.
\nNetflix is not just a streaming platform — it is a data factory that runs thousands of ML pipelines (automated sequences of data processing, model training, and validation jobs that produce the recommendation algorithms and personalization signals driving Netflix's content strategy) and data engineering workflows every single day. Recommendation models, A/B test analyses, content quality pipelines, ad-targeting models — every one of these is a graph of interdependent jobs that must be orchestrated, retried on failure, scheduled on time, and monitored continuously. By 2020, Netflix was running all of this through Meson, an in-house workflow orchestrator built around a single-leader architecture that the team described as having achieved high availability — but at the cost of requiring continuous vertical scaling as usage grew.
\n\n\n\n📈
\nThe number of workflows in Meson was doubling year over year, and the sizes of individual workflows were growing too — some containing tens of thousands of interdependent jobs. Vertical scaling on AWS had a hard physical ceiling: you can only get so many CPUs and so much RAM in a single EC2 instance type.
\n
The core problem with Meson was structural. A single-leader architecture (a system design where one node (the leader) is responsible for all decisions and coordination — providing simplicity and consistency but creating a vertical scaling bottleneck as load increases) means all orchestration decisions flow through one machine. For low-throughput systems, this is fine. For a platform running hundreds of thousands of daily ML workflows with projected 100% year-over-year growth, it meant the infrastructure team was perpetually fighting against the ceiling of what the largest available AWS instance type could handle. During peak usage — typically when multiple large training runs coincided with end-of-month reporting pipelines — Meson experienced slowdowns that required on-call engineers to closely monitor the system, especially during off-hours. The system was not broken, but it was clearly approaching the limits of how it was designed.
\n\n\n\nTHE VERTICAL SCALING WALL
\nAWS instance types top out. In 2020, Netflix's Meson orchestrator was approaching the compute limits of the largest available EC2 instances. The team could keep picking bigger and bigger machines, but the ceiling was visible and the timeline to hit it was predictable. Horizontal scaling — distributing load across many commodity machines — was the only architectural solution that could match indefinitely growing workflow volumes.
\n
During peak usage, Meson's single-leader architecture struggled under the combined load of hundreds of thousands of daily jobs. On-call engineers were monitoring the orchestrator during off-hours to prevent cascading delays. The system worked — but it required human attention that scale demands cannot sustain.
\nMeson's single-leader model meant all orchestration state — job status, dependency tracking, retry logic, scheduling — lived on one machine. As Netflix's DAG (Directed Acyclic Graph — a workflow representation where jobs are nodes and dependencies are directed edges, ensuring no circular dependencies) workloads grew 100% year-over-year, vertical scaling was no longer a strategy; it was a countdown.
\nNetflix designed Maestro from first principles for horizontal scalability. The architecture decomposed orchestration into independent stateless workers, event-driven step execution, and purpose-built state management. Hundreds of thousands of workflows migrated from Meson to Maestro with minimal user disruption, and by 2024 Maestro was open-sourced under Apache 2.0.
\nMaestro handles 2 million jobs on busy days across hundreds of thousands of workflows, with support for individual workflows containing hundreds of thousands of jobs. Scaling is now horizontal — add more workers, handle more load — without the system design itself becoming a constraint.
\n\nUnlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflows, and conditional branches.
— — Jun He, Natallia Dzenisenka et al. — via Netflix Technology Blog
Traditional workflow orchestrators like Apache Airflow assume that workflows are DAGs (Directed Acyclic Graphs — graphs where edges have direction and no cycles exist, meaning no job can eventually depend on itself). Netflix's data engineering reality is messier: some workflows are genuinely cyclic, some need foreach loops that spawn thousands of child workflow instances, some need conditional branching that determines entire execution paths at runtime. Maestro was built to support all of these natively within the engine, rather than requiring users to simulate them with workarounds. The foreach pattern is particularly powerful: each iteration is internally treated as a separate workflow instance, scaling identically to any other Maestro workflow. A single foreach over a thousand content items spawns a thousand parallel workflow instances, each tracked independently, each retried independently, each reportable independently.
\n\n\n\nℹ️
\nWorkflow-as-a-Service: Abstracting Infrastructure from Users
Maestro is designed as a fully managed workflow-as-a-service for Netflix's data practitioners — data scientists, ML engineers, content producers, and business analysts. Users define their business logic in Docker images, notebooks, bash scripts, SQL, or Python, and Maestro handles scheduling, queuing, dependency resolution, retries, and monitoring. Users never configure infrastructure. The engineering investment lives entirely in the platform layer, freeing thousands of practitioners to focus on what their workflows do rather than how they run.
\n
\n\n\n🎯
\nStrict SLOs During Traffic Spikes
Maestro is designed to maintain its service level objectives even during spikes in workflow submission traffic. This matters because Netflix's data platform is non-uniform: large content drops, end-of-quarter analyses, and major live events all generate burst workflow submissions that can be orders of magnitude above the baseline. A workflow orchestrator that degrades its own SLOs during the exact moments when reliability matters most is worse than useless.
\n
Netflix's Meson-to-Maestro migration was deliberate about user disruption. The team migrated hundreds of thousands of workflows on behalf of users — users didn't rewrite anything. The platform team built migration tooling, validated that migrated workflows produced equivalent outputs, and performed the migration atomically for each user's pipeline. This is a crucial organizational point: the people running the workflows were data scientists and analysts, not infrastructure engineers. Asking them to relearn an orchestration system and rewrite their pipelines would have consumed months of productive time across the company. By owning the migration completely, the Maestro team enabled a seamless transition that felt, from the user perspective, like a platform upgrade that just worked.
\n\n\n\n⚠️
\nThe On-Call Tax of Vertical Scaling
Meson's vertical scaling strategy created an operational pattern where engineers had to manually monitor the orchestrator during off-hours around peak usage — particularly when end-of-quarter reporting pipelines coincided with large ML training runs. This on-call tax is a leading indicator of approaching architectural limits. When your infrastructure requires human attention to avoid failure during predictable peak events, the architecture is telling you it cannot scale further without change.
\n
Maestro's architecture replaces Meson's single-leader with three independently scalable services. The Workflow Engine manages the full lifecycle from definition through execution — it is the core state machine that tracks which steps have completed and which are ready to run. The Step Runtime Workers are stateless executors that pick up individual step executions from a queue, run them on the appropriate compute engine (Spark, Trino, Kubernetes, etc.), and report results back. The Signal Service enables event-driven orchestration — workflows can wait for external signals (data availability, upstream pipeline completion) rather than polling or using fixed schedules. Crucially, each of these three layers scales independently: need more throughput on step execution? Add more workers. Need more scheduling capacity? Scale the engine tier. No single bottleneck.
\n// Simplified Maestro workflow definition example\n// Users define their logic; Maestro handles all execution mechanics\n{\n \"workflow_id\": \"ml_model_training_pipeline\",\n \"steps\": [\n {\n \"id\": \"data_prep\",\n \"type\": \"spark\", // run on Netflix's Spark cluster\n \"image\": \"netflix/etl:v2.3\",\n \"dependencies\": [] // no upstream dependencies — runs first\n },\n {\n \"id\": \"feature_engineering\",\n \"type\": \"foreach\", // Maestro native pattern: spawn parallel sub-workflows\n \"items\": \"${data_prep.output.segments}\", // iterate over upstream output\n \"steps\": [\n // each item spawns its own workflow instance — scales to thousands\n { \"id\": \"compute_features\", \"type\": \"spark\" }\n ],\n \"dependencies\": [\"data_prep\"]\n },\n {\n \"id\": \"model_train\",\n \"type\": \"kubernetes\", // different compute engine — Maestro routes it\n \"image\": \"netflix/trainer:v1.1\",\n \"dependencies\": [\"feature_engineering\"]\n }\n ],\n // Signal-based trigger: run when upstream data is ready, not on a fixed clock\n \"trigger\": { \"type\": \"signal\", \"signal_name\": \"daily_data_ready\" }\n}\n\n\nℹ️
\nWhy Not Airflow, Temporal, or Conductor?
The Netflix team evaluated off-the-shelf alternatives before building Maestro. Apache Airflow lacked native support for cyclic workflows and struggled with Maestro's target scale. Netflix Conductor was an option but offered more state-engine features than required, and its overhead was disproportionate to the need. Temporal was optimized for inter-process orchestration via external service calls — at Maestro's million-tasks-per-day scale with many long-running workflows, coupling the DAG engine to an external service call introduced unnecessary reliability weak spots. The team concluded that for their specific requirements — lightweight state transitions at massive scale — a purpose-built engine was the right call.
\n
\n\n\nTHE OPEN SOURCE DECISION
\nIn July 2024, Netflix open-sourced Maestro under the Apache 2.0 license. The decision reflected confidence in the system's production maturity after 4+ years of internal operation and Netflix's long tradition of contributing infrastructure tooling to the engineering community. Maestro joins a list of Netflix open-source projects — Chaos Monkey, Eureka, Hystrix — that have shaped how the industry thinks about distributed systems. The GitHub repository includes the full Java 21 + Gradle codebase and curl-based quickstart examples.
\n
The signal service is one of Maestro's most operationally valuable features. In traditional time-based scheduling, a pipeline runs at midnight regardless of whether the data it depends on is ready. If upstream processing is slow, the pipeline waits, fails, or wastes compute resources scanning for data that isn't there yet. Signal-based triggers invert this: Maestro listens for an event published when upstream data is confirmed ready, and only then starts the dependent workflow. For Netflix's data platform, where hundreds of pipelines have complex dependency chains on upstream tables and streams, this eliminates entire categories of pipeline failures caused by timing assumptions that don't hold under variable upstream load.
\n\n\n\n✅
\nZero-Downtime Migration Strategy
Maestro's team built tooling to migrate Meson workflows in place, validating equivalence of outputs before switching. Each workflow's migration was atomic: Meson and Maestro ran side-by-side during transition, with a configurable percentage router determining which system executed any given workflow instance. Users saw no interruption to their pipelines during the migration period.
\n
\n\n\n🔌
\nPluggable Compute Backends
Maestro's step runtime workers are designed to be compute-engine agnostic. A step definition specifies the engine (Spark, Trino, Kubernetes, Python) and the business logic image; the worker handles routing, execution, and result reporting without the workflow engine knowing which compute platform ran the job. This loose coupling means Netflix can add new compute backends — or retire old ones — without touching workflow definitions.
\n
Meson's architecture concentrated all orchestration intelligence in a single process: state management, scheduling, dependency tracking, retry logic, and compute routing all ran in one JVM. This was comprehensible and debuggable but fundamentally unscalable. Maestro separates these concerns across three independently deployable services, with Apache Kafka (a distributed event streaming platform used by Netflix for high-throughput, fault-tolerant message passing between Maestro's internal services and to external downstream consumers) as the message bus connecting them and PostgreSQL (the relational database Maestro uses for durable workflow state storage, chosen for its ACID guarantees which are essential for exactly-once execution semantics) as the durable state store.
\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\nView interactive diagram on TechLogStack →
\nInteractive diagram available on TechLogStack (link above).
\n\n\n\n\nFOREACH: SCALING TO 100K JOBS IN ONE WORKFLOW
\nMaestro's foreach step is the capability that makes truly massive workflows possible. When a foreach iterates over 10,000 content segments, Maestro internally creates 10,000 independent sub-workflow instances — each scheduled, executed, monitored, and retried independently. From the user's perspective, it's a single foreach loop. From Maestro's perspective, it's 10,000 parallel workflow instances scaling identically to any other workflow in the system. This is not possible in flat DAG systems where 10,000 nodes would overwhelm the scheduler.
\n
\n\n\n✅
\nExactly-Once Execution: The Deduplication Guarantee
Maestro's scheduler includes explicit deduplication logic — even if the scheduler fires a trigger multiple times for the same workflow (due to retries or race conditions), Maestro guarantees the workflow is executed exactly once. This is a critical property for ML training pipelines and financial reporting workflows where duplicate execution could produce incorrect results or waste significant compute resources. The guarantee is implemented via PostgreSQL's ACID transactions on workflow state transitions.
\n
\n\n\n📡
\nExternal Event Publishing: Beyond Netflix
Maestro publishes execution events to external messaging systems — Kafka, SNS, SQS — enabling downstream consumers to react to workflow state changes. A data warehouse system can start preparing tables when a pipeline completes. A monitoring system triggers alerts on critical workflow failures. An analytics dashboard updates in real time. This event bus design transforms Maestro from an orchestrator into a first-class component of Netflix's data platform event fabric.
\n
The story of Maestro is ultimately a story about recognizing architectural limits before they become production disasters, and building the replacement with the same care that you'd apply to user-facing products. Platform infrastructure is a product too.
\nWhat to remember
\n