GitHub Engineering Case Studies

16 min

How GitHub Upgraded 1200 MySQL Hosts Without Dropping a Single Query

MySQL 5.7 was hitting end-of-life, and GitHub's production database fleet spanned 1,200 hosts, 300 terabytes of data, and 5.5 million queries every second. Getting from here to MySQL 8.0 without disrupting 100 million developers was going to take more than a weekend.

1,200+ MySQL hosts upgraded 300+ TB data migrated 5.5M queries/sec maintained +2

Read full story

GitHub Distributed Systems

19 min

GitHub Built the Internet's Code Platform — Then AI Agents Broke It

Between May 2025 and April 2026, GitHub experienced 257 incidents — 48 of them major outages. That's roughly one significant disruption every single week. The culprit wasn't a security breach, a botched deployment, or a rogue engineer. It was the thing GitHub had spent years celebrating: AI. Specifically, agentic AI workflows that turned one human developer's footprint into hundreds of commits, thousands of CI minutes, and a dozen simultaneous PR operations — all at once, across millions of accounts. GitHub had been built for humans. Agents are not human.

257 incidents — May 2025 to April 2026 48 major outages, 112+ hours total downtime 57 GitHub Actions outages in 12 months +1

Read full story

GitHub Databases

4 min

GitHub's Settings Cache Went Stale and Took Authentication, Actions, and Copilot Down With It

A configuration change to the user settings cache triggered a global invalidation. Every AI model preference and policy setting hit the database at once — and the database that stored them also handled login.

2 hr 43 min disruption 2 outage windows Auth, Actions, Copilot affected +1

Read full story

GitHub Reliability

3 min

GitHub Actions Froze for 95% of Workflows When a Redis Cluster Hit Its Limit

On March 5, 2026, GitHub's CI job queue stopped dispatching. Developers pushed code, saw their checks queued — and then nothing happened for 30 minutes.

95% workflows blocked 30 min avg delay Redis cluster failure +1

Read full story

GitHub Security

3 min

GitHub Lost Telemetry and Its Own Security System Started Blocking Real Developer VMs

A telemetry pipeline went silent. GitHub's security automation treated the silence as a threat signal — and locked every Codespaces VM out of its own metadata service.

~6 hr Codespaces outage All regions affected Copilot, CodeQL blocked +1

Read full story

GitHub Performance

2 min

A Fiber Cut in Seattle Slowed GitHub Clone Speeds to Under 1 MB/s for Eight Hours

Nothing changed in GitHub's codebase. A cable under Seattle got cut, and git clone on the US west coast went from fast to dial-up speed — and TCP made it worse.

<1 MiB/s clone speed 800 Gbps → 3.2 Tbps ~8 hr disruption +1

Read full story

GitHub Databases

2 min

Dependabot Silently Skipped 10% of Security PRs Because a Failover Landed on a Read-Only Database

For 42 hours, Dependabot appeared to be running fine. It was quietly failing to create security pull requests — and nobody got an alert.

42 hr degraded window 10% of PRs silently failed Zero visible errors shown +1

Read full story

GitHub Distributed Systems

4 min

GitHub Was Built for 2008. AI Agents Demanded 30x More Scale in 2026 — and the Platform Broke

In October 2025, GitHub set a goal of 10x capacity growth. By February 2026, the CTO was publicly saying that wasn't enough. AI-assisted development had changed the load model entirely.

257 incidents in 12 months 48 major outages 30x redesign target +1

Read full story

GitHub Databases

16 min

GitHub's Database Split Its Brain in 43 Seconds. Cleaning It Up Took 24 Hours.

On October 21, 2018, a technician in a GitHub data center accidentally disconnected a 100G optical cable during routine maintenance. The connection was restored in 43 seconds. In those 43 seconds, GitHub's automated database failover tool had already promoted a new primary database on the West Coast — before the East Coast primary had finished replicating its recent writes. For the next 24 hours, GitHub had two sources of truth for its most critical metadata, neither complete, and no clean way to merge them.

duration of network partition: 43 sec service degradation after: 24+ hrs webhooks queued for replay: 5M+ +1

Read full story

GitHub Reliability

17 min

The Test That Broke GitHub: A Failover Drill Goes Live

June 29, 2023, 17:39 UTC: GitHub engineers initiate a planned live failover test of their brand-new second Internet edge facility — six months of infrastructure work designed to eliminate a single point of failure. Within seconds, instead of validating their redundancy, they've created an outage that takes GitHub offline for millions of developers across North America and South America.

32-minute outage 2-min detect-to-revert ~100M devs affected

Read full story