Topic

Messaging

LinkedIn needed to move data between systems and nothing on the market could do it at their scale, so they built Kafka. Discord had trillions of messages and a garbage collector that couldn't keep up, so they rewrote it. These are the stories of how message queues, event buses, and async pipelines became the infrastructure the internet runs on.

LinkedIn Messaging

21 min

LinkedIn Needed a Message Queue. They Built the One the Entire Internet Runs On.

In 2010, LinkedIn was drowning in data it couldn't move. Every ML model, every recommendation engine, every real-time feature was starving because there was no reliable way to get activity data from the website into the systems that needed it. Jay Kreps, Jun Rao, and Neha Narkhede spent a year building a fix. They named it after Franz Kafka. The rest of the internet adopted it.

1B events/day at launch (2011) 1T messages/day by 2015 7T messages/day by 2019 +1

Read full story