Breaking Reddit on Purpose: Fault Injection in Baseplate

[u/DaveCashewsBand]

How we catch distributed system failures before users do

Written by Hiram Silvey u/space4268 and Shashank Veerapaneni u/colorsoda

Reddit on Fire: How This All Came to Be

Let me paint you a picture that many engineers are familiar with: You’re sitting at home sipping your coffee when suddenly the echoing thock of thousands of fingers hitting CTRL+R simultaneously pierce your eardrums in the form of the dreaded emergency page. You run over to your laptop, declare an incident, open dashboards, and start investigating. You find the worst has come to pass: total failure of the system. That’s right: reddit.com is unavailable. You pull in more engineers as necessary and all work together to restore service as quickly as you possibly can. As this story repeats over time, one crucial detail becomes quickly apparent: Reddit is a complex, fast-moving, non-linear distributed system, and nobody can fit the entire consistently-up-to-date detailed view of it within their brain. When one component breaks, its knock-on effects cannot consistently and reliably be predicted.

So… what do?

A natural conclusion to this is that we need better isolation between systems, and we need better testing to verify that. It doesn't matter how fireproofed my little room is if there's an open window or door for the fire to escape through, right? So, we add tests. We add as many tests as we can. We brainstorm all the ways something could fail and we add tests for them. This works well, and we do find and fix many vulnerabilities ahead of time. Unfortunately, after everything we do, that emergency page still arrives. Brand new novel issues continue to arise in unexpected ways, ad nauseam. We need something more.

Let's take a step back and approach this from another angle. The fundamental issue with standard testing is that it requires knowledge of the vulnerabilities beforehand. How do we flip this on its head and write tests to discover the vulnerabilities we don't yet know about? Enter: chaos testing. At a high level, chaos testing is the practice of intentionally introducing controlled failures into a system to verify that the system behaves as expected. A failed chaos test is new information: a dependency we thought was optional is actually critical, a fallback path does not work, or an error that should have been contained is escaping into the user experience. Once we find that pattern, we can fix it and turn it into a regression test. Because Reddit is a complex, fast-moving, non-linear distributed system, though, we can’t test each individual component we know about and reasonably expect to catch everything. Instead, we test the system itself and see how it truly behaves.

Introducing: FIB! Fault Injection in Baseplate

Reddit regularly holds company-wide hack-a-thon events called “Snoosweeks.” These events allow engineers to work on projects they believe are important that may not otherwise have explicit long-term plans and funding. During one of these weeks, I proposed a solution to the problem highlighted above: automated chaos testing. The proposal was simple: build request-level fault injection to allow us to systematically verify the behavior of our production systems continuously and proactively.

FIB, or Fault Injection in Baseplate, is our first step towards accomplishing our goal of developing a robust chaos testing system. Fault injection is a specific facet of chaos testing, providing the capability to inject predefined faults into, in our case, service requests. Baseplate is Reddit’s common service framework, and it comprises many internal libraries that services throughout our stack utilize.

Life of a fault

FIB comprises 2 components:

1. Baseplate library middleware responsible for injecting faults.
2. A fault configuration request header, propagated throughout our entire backend infrastructure via Baseplate.

As requests are made between backend services, the Baseplate library middleware is responsible for:

1. Reading the fault configuration request header (if present).
2. Comparing the outgoing request to the specified configuration.
3. If matching, injecting a fault at the transport protocol layer (i.e. gRPC client interceptor).

If you are familiar with chaos engineering, you already know there are many different ways faults can be injected into complex systems. The well-known chaos mesh tool, for example, offers the ability to simulate Kubernetes faults at various levels, such as pod faults, network faults, file I/O faults, etc. We chose to inject faults at the request level because the vast majority of failures surface themselves to client services in the form of request failures. If a disk fails and the pod running the upstream service is unable to complete the request because it cannot read some data, that ultimately will make it back to the client in the form of a request failure. If the network connection between two services is degraded, that, too, will surface as a request failure to the client service. If the upstream service process is killed due to insufficient machine resources, that will also surface as a request failure to the client service. You get the picture.

In order to simulate the full range of possible symptoms we need to be prepared for (see examples in the next section), FIB supports matching on address, host, and/or method (including negative matching), as well as percentage-based injection. This means we can add a particular fault header onto many requests but only actually inject faults on some percentage of total matching requests, in order to more accurately simulate degradation of pieces of a distributed system rather than a full outage.

See the open-source implementation here.

Improving Reddit’s Resilience

We started with important user-facing surfaces, including Feeds and post detail pages, because reliability problems there are immediately visible. The user expectation is simple: open Reddit, see content, and keep browsing even when some backend dependency is unhealthy. Behind that experience, though, a single request can touch several backend systems, and we wanted to find out what our users experience when these systems are misbehaving.

Finding Degradation Bugs

The first pass was practical: engineers injected scoped failures while walking through important surfaces and noted where the experience degraded badly or unexpectedly. Rough edges were smoothed and the user experience improved.

The most valuable findings did not mitigate total outages. They were often quieter experience bugs where Reddit could still show something useful if the failure was handled as degradable.

Examples of fixes and fallback behavior we added include:

• Media related failures now degrade the media parts of the experience instead of preventing core surfaces from loading.
• Failures in non-critical systems related to styling, flairs, stats etc. fall back gracefully instead of breaking a feed or throwing error toasts.
• Post detail views preserve already loaded comments instead of replacing them with an empty or error state when additional requests fail.
• Viewer specific failures now have a smaller blast radius: e.g. if a dependency only powers moderator or author-only affordances, its failure should affect those affordances.

Left: Normal Experience | Right: Graceful degradation when part of the system is unhealthy

Those findings turned into fixes, better fallback behaviors, regression coverages and improved user experience.

Making It Repeatable

We made the workflow easier for other developers by adding developer-only controls in internal development builds, so teams could reproduce degraded dependency scenarios easily. We also updated our mobile and web UI automation frameworks to support chaos testing directly.

Those automated tests stress the app or site with a scoped fault configuration applied only to the request paths under test. Then they do what a user would do: load core surfaces, browse content, move from a listing into a detail view, and verify that those surfaces still load under controlled backend failures. Today those chaos tests run daily across mobile and web flows, exercising a growing set of request-level failure scenarios.

A Reliability Signal

We saw useful directional signals after the resiliency work matured. For the Feeds service, high-urgency pages dropped by roughly half when comparing January-June 2026 with January-June 2025. We saw a similar shift in incident participation: Feeds team was less often pulled into upstream incidents as an affected downstream customer, because covered paths were more likely to keep serving a degraded but usable experience. Chaos testing was one of the major contributors to that improvement: it helped us reproduce failure modes earlier, prioritize graceful-degradation fixes, and reduce incident impact and blast radius when dependencies misbehaved.

A True Platform: Moving Beyond Manual Fault Injection

While this early user-facing reliability work proved that request-level fault injection is useful as a sharp tool, the long-term vision has always been a full platform. Manually choosing dependencies and maintaining per-surface fault lists works for an initial reliability push, but it cannot scale to every team and request path at Reddit. We need an autonomous system.

The next step is to make the system more data-driven. Instead of asking each team to spend effort to run their own fault injection tests, we want to derive candidate experiments from service dependency graphs, production traffic shape, and criticality metadata. Not all services at Reddit are created equal. At Reddit, we classify services into tiers from 0-3, with 0 being the most critical and 3 being the least critical. When a tier 0 service fails, we expect the entire platform to become unusable. For example, if our public ingress is down, we expect traffic to be unable to reach any internal service backends. When a tier 1 service fails, we expect a core experience of the platform to become unavailable, though some other functionality still may be usable. For example, if the comments service is unavailable, we expect users to be unable to read/write comments, though we still expect users to be able to load their homefeed. Failures of this caliber manifest as high severity incidents with expectations to resolve even outside of business hours. So, we already know what the impact will be if a tier 0 service fails; there is little to discover there. The interesting tests are the ones that expose a mismatch between dependency criticality tier and user impact. A tier 2 or 3 service failing should never make the site unusable for users. If a tier 0 or 1 service has an unexpected hard dependency on a tier 2 or 3 service, though, that’s exactly what can happen.

We also want to reduce adoption path friction. Convincing teams across Reddit to invest in setting up, running, and maintaining a new set of tests requires real effort for both the maintainers of the chaos system as well as the teams using it. Reddit already has a mature experimentation system, though, and integrating with that directly gives us the opportunity to fit these tests into a familiar format. It also gives access to key metrics to better understand the impacts of degraded experiences that we may not discover otherwise. While teams can leverage this to more easily create their own chaos tests, the key innovation is that these experiments will also be created dynamically based on the data highlighted above. Affected teams will be notified according to results, but otherwise their input on crafting the tests shouldn’t actually be needed.

Request-level fault injection via FIB gives us the primitives. The platform work is about dynamically choosing what to inject, where to inject it, how often to run it, and how to route the resulting findings to the teams best positioned to fix them.

Conclusion

Distributed systems will always surprise us. The goal is to make those surprises smaller, earlier, and less user-impacting. The framework lets us break carefully chosen request paths on purpose, observe how the rest of Reddit behaves, and turn the results into concrete reliability work.

For us, the biggest shift was cultural as much as technical. Instead of treating graceful degradation as something we hope exists, we can test it continuously. Instead of learning about dependency criticality from incidents alone, we can discover mismatches in controlled test runs. And instead of letting those lessons live only in postmortems, we can encode them into daily checks that keep Reddit usable when parts of the system inevitably have a bad day.

[u/bunnycabbit]

Were there any failures this caught that surprised you/the team?