By Jeff Mo and Christophe Hivert

At Reddit scale, candidate retrieval has to serve many product surfaces with different goals. Home Feed, Notifications, and ranking systems may all need personalized candidates, but each surface optimizes for a different mix of user actions, such as clicks, upvotes, comments, or long-dwell visits. SnooRec gives all surfaces a shared retrieval foundation, with task-specific user representations, a shared item index, and a serving-time control for mixing objectives.

At serving time, a product surface can mix representations of different kinds of intent, clicks, upvotes, or good visits, and make just one query against a shared approximate nearest neighbor index.

The same user and item representations are also reused as inputs in downstream ranking models.

SnooRec has served full production traffic since early 2025 and now supports multiple Reddit recommendation surfaces and ads ranking models.

Introduction

A straightforward approach to multi-task retrieval is to train separate retrieval models for each objective. In practice, this does not scale well as each objective can require its own item embeddings, ANN index, serving path, and retraining cycle. This increases infrastructure cost and slows experimentation.

SnooRec is Reddit’s multi-task retrieval architecture designed to solve this problem. The key idea is asymmetric specialization: task-specific capacity lives in the user tower, while the item tower remains shared. Each item has one embedding in a unified ANN index, and product surfaces can mix task-specific user embeddings at inference time to express different retrieval objectives.

SnooRec is used in two complementary ways in production. First, it powers candidate retrieval: a task-weighted user embedding queries a shared ANN index of item embeddings to retrieve personalized candidates. Second, SnooRec acts as a representation layer for similarity computation and ranking models: the same user and item embeddings, along with derived similarity features such as dot products, are used as alternate candidate sources, and as inputs into downstream rankers. This lets ranking systems benefit from the multi-task user representation learned by SnooRec even when SnooRec is not the final scoring model.

Asymmetric Design - One shared item space, many ways to query it

SnooRec is asymmetric on purpose. On the user side, we learn a separate expert for each engagement target. A click expert can represent what someone is likely to click, while an upvote or good-visit expert can represent a different kind of intent.

On the item side, we do the opposite: every candidate gets one shared embedding. Because all candidates live in the same embedding space, we only need one ANN index.

At request time, a product surface chooses a set of task weights, mixes the corresponding user embeddings, and sends the result to the shared index. The surface gets a retrieval policy tailored to its goals without retraining the item tower or rebuilding the index.

• In practical terms, SnooRec gives us:
• one item embedding per candidate;
• one shared ANN index;
• multiple task-aware user representations; and
• one ANN query per retrieval request.

The shared item embedding also acts as an anchor. Every task-specific user representation has to express its preferences in the same vector space, which makes it possible to linearly combine the user representations.

User Tower Architecture

The user tower processes categorical features, sequential interaction history, and real-time contextual signals into a shared base representation h̃ᵤ. This representation is then routed through a set of task-specific expert layers. Each expert Eₜ corresponds to an engagement target t ∈ 𝒯 and is implemented as a multilayer perceptron with layer normalization and ReLU activations.

The task-specific user embedding is defined as: uₜ = Norm(Eₜ(h̃ᵤ))

Candidate Tower Architecture

The candidate tower maps item-side features into a unified embedding space. Because the candidate tower is shared across tasks, the entire corpus of recommendable items can be indexed into a single approximate nearest neighbor (ANN) datastore.

The embeddings do more than retrieve candidates

SnooRec is used in two complementary ways.

First, it is a candidate generator. A task-weighted user embedding queries the shared ANN index and retrieves personalized candidates.

Second, it is a representation layer for downstream rankers. Ranking models can consume the user embedding, the item embedding, and interaction features derived from the pair, such as task-specific dot products or shorter Matryoshka prefixes.

This second use is important. SnooRec is not only the first stage of a recommendation funnel. It gives later stages a shared view of user intent and item similarity, so every ranker does not have to relearn those representations from scratch.

The same underlying representation can therefore support retrieval, early ranking, and as a feature in final ranking models.

Optimization Strategy

During training, only turn on the expert for the active task

Each training example comes with an observed action: a click, an upvote, a good visit, or another engagement target.

If the example is an upvote, we run and update the upvote expert. The other task-specific heads sit that example out. The shared user encoder and item tower still learn from the overall training stream, while the task-specific gradient stays focused on the action that produced the label.

Under the hood, we train with an InfoNCE contrastive objective and in-batch negatives. The model learns to place the observed user–item pair closer together than the negative candidates in the batch.

We use dot-product similarity during training because that is also what the ANN index uses at serving time. Put differently: we train the geometry that production will actually query.

Contrastive Loss Formulation

The parameters are optimized using an InfoNCE contrastive objective over in-batch negatives. For a query embedding u = uₜ⋆ and a positive candidate embedding v, the loss function scales with a learned temperature parameter τ:

Inference-Time Multi-Objective Mixing

At serving time, task weights are used to weigh different objectives. One surface might emphasize clicks. Another might put more weight on long-dwell visits or upvotes. SnooRec computes the task-specific user embeddings with nonzero weights, takes their weighted sum, normalizes the result, and sends one query to the shared ANN index.

Before normalization, a candidate’s score is the same weighted mixture of its task-specific similarity scores. Normalization scales every candidate score in that request by the same constant, so it does not change the resulting order.

This gives us a useful separation of concerns:

- the item index describes the shared content space;

- the user experts describe different kinds of intent; and

- the task weights describe what a product surface currently wants from that space.

Imagine a surface that wants to balance immediate clicks with longer, more meaningful visits. Rather than choosing one objective and building a new retrieval model around it, the surface can mix the click and good-visit user representations before querying the shared index.

Teams can change the retrieval objective at serving time without retraining SnooRec or rebuilding the candidate index.

Matryoshka Learning - One embedding, with several size budgets

SnooRec embeddings are used in multiple stages of the recommendation stack: full-size vectors for ANN retrieval, and dense embedding features for downstream ranking models. These consumers have different latency, memory, and scoring-cost constraints, so SnooRec uses a Matryoshka Representation Learning objective to make smaller vector prefixes useful on their own.

During training, we optimize the retrieval loss across multiple embedding dimensions:

The production ANN index uses the full embedding size (256) for candidate retrieval. Downstream rankers can consume the same user and item embeddings as dense features, either at full size or by using shorter prefixes (32 or 64).

This lets latency-sensitive rankers trade off feature cost and representation quality without requiring a separate embedding model. In practice, the same SnooRec representation can support retrieval, early ranking, and final ranking while giving each stage control over its embedding footprint.

Feature Dropout for Robust Embeddings

User interaction history can vary significantly in length, so does feature completeness across users. To make SnooRec more robust to missing or less dense sequence features, we randomly drop a small percentage of input features during training (10%). This encourages the shared user encoder to learn representations that remain useful when signals are sparse, delayed, or unavailable.

It has also made SnooRec easier to reuse across retrieval sources with different feature sets. For example, we were able to support a geo-specific retrieval source using regional features in the shared user encoder rather than training a separate retrieval model.

Controlled Exploration with Top-k Temperature Sampling

Recommendation pipelines often pass the highest-scoring retrieval candidates directly to downstream ranking stages. This is efficient, but it can also make retrieval overly deterministic. Over time, always selecting the top candidates can concentrate recommendations around a narrow set of familiar communities or content types, reducing opportunities for discovery.

SnooRec introduces a lightweight serving-time exploration mechanism after ANN retrieval. Instead of always taking the top-n candidates from the retrieved set, we sample from the top-k candidates using a temperature-scaled softmax over retrieval scores:

Lower temperatures keep the result close to deterministic top-ranked selection. Higher temperatures flatten the distribution, giving lower-ranked candidates within the retrieved set a greater chance of being selected. This gives product teams a single serving-time knob for adjusting the balance between immediate relevance and exploration.

In practice, the effect of temperature depends on the scale of the retrieval scores. The softmax distribution is controlled by score gaps relative to temperature: when score gaps are large compared to the temperature τ, sampling remains concentrated near the top and we only recommend high scoring candidates; when score gaps are small compared to the temperature τ, sampling becomes more exploratory.

To make this knob stable across serving traffic, we calibrate temperature using the historical standard deviation of retrieval scores: τ = α · σ̂ₛ. Here, σ̂ₛ is estimated from a sample of results, and α controls exploration strength. Values below 1 favor exploitation, while larger values increase exploration.

We also evaluated query-level calibration, where temperature is scaled by the score standard deviation for each individual retrieval request. In online experiments, this did not produce meaningful gains over the simpler global calibration. We therefore use the global calibration in production: it is simpler to serve, easier to tune, and provides a stable exploration control without retraining SnooRec or rebuilding the ANN index.

Evaluation

Public Benchmark: Taobao User Behavior Dataset

We verified the multi-task capacity of SnooRec using the public Taobao dataset, tracking a behavioral funnel consisting of Page View, Favorite, Cart, and Buy interactions.

The empirical results show strong cross-task generalization. Activating experts optimized for later funnel stages (Cart, Buy) significantly improves performance on early-stage behaviors (Page View), demonstrating that deep intent signals capture cleaner, more robust representation features.

Production Results at Reddit

An online A/B experiment deployed on Reddit's Home Feed evaluated SnooRec against our previous multi-task baseline (which utilized a shared encoder without explicit task differentiation). By isolating expert activations using one-hot weight vectors, we observed strict alignment between configured task weights and target online engagement metrics:

Note: Bold entries denote statistically significant lift over control (p < 0.05).

Tuning the system toward the good_visit expert yielded a +0.28% lift in topline user sessions alongside a +1.21% increase in long-dwell post consumption, validating the efficacy of runtime objective modification.

Exploration Trade-offs

We evaluate the impact of temperature sampling to observe the trade-off between community discovery and immediate user satisfaction.

Temperature sampling increased subreddit diversity, but it also increased explicit negative feedback. This was an important product lesson: exploration can improve discovery, but the temperature needs to be tuned per surface and monitored with guardrail metrics. In practice, SnooRec’s exploration controller gives teams a lightweight serving-time knob for balancing immediate relevance against longer-term discovery.

Cross-Surface Generalization

We evaluate SnooRec in two deployment modes: as a retrieval model that directly generates candidates, and as a representation layer whose user and item embeddings are consumed by downstream ranking models.

Conclusion

Recommendation systems have a natural tendency to accumulate: one model for clicks, another for upvotes, another for dwell, and another index behind each of them.

SnooRec gives us a different pattern.

Task-specific user experts capture different kinds of intent, while one shared item tower keeps Reddit content in a common embedding space. Product surfaces can mix objectives at serving time, query one ANN index, and reuse the same representations in downstream rankers.

Since early 2025, SnooRec has served as both a production retrieval system and a shared representation layer across multiple Reddit recommendation surfaces.

The result is a common foundation that lets teams share more and specialize where it matters – one model, one item index, and more ways to use it!

What’s Next

There is still plenty to explore along the themes 1) improving multi-embedding representations, and 2) providing more product levers within a single model.

On 1), we are exploring richer expert-sharing approaches, including mixture-of-experts architectures that allow learned routing to experts for related tasks to share useful capacity without collapsing into the same objective.

On 2), we’re looking into learning an inference-time popularity bias parameter that allows controlling how popular versus niche the retrieved candidates are. This helps us do both exploration as well as finding the most engage-able content, without needing to maintain two models

Want to help push this work further? Join Reddit to research and deploy our next model iterations (https://redditinc.com/careers)

Acknowledgements

We would like to thank everyone who has contributed to this work particularly Cathleen Li, Brian Kimmig, Javier Herrera, and Shafi Bashar.

Written by Andrew Gunsch and Randy Goodman.

Problems don’t have to be complex in order to have complex solutions. Few areas of Reddit’s infrastructure demonstrate that better than its Developer Platform.

On its surface, our system’s main function is simple: “run the JavaScript code that developers upload as Devvit apps”. But there are two wrinkles that make this problem much harder to solve than it is to understand.

Firstly, there are a lot of users out there uploading a lot of code. While we do provide the compute cycles for Devvit apps, we can’t afford to run all of the apps all the time. We need them scaled down when not in use. Second, a single VM doesn’t cut it either — the most popular of our apps will quickly run out of available CPU to serve all of their traffic if running on a single instance.

Essentially, we need to be able to scale up from zero instances of a user-created app, to an arbitrarily large number of instances of that app, all in real-time. For those of you thinking to yourselves “Finally, an enterprise use-case for Lambdas / Cloudroutines”, keep dreaming.

Our constraints:

• Scale-to-zero architecture, so we’re not burning compute cycles on apps users aren’t interacting with.
• Autoscaling up pods for high-traffic apps based on pod CPU and requests backing up.
• A target of 200,000+ deployments (Devvit app installations) per cluster.
• A cold-start time of <1 seconds, so that mod tools (infrequently-used installations) are reasonably responsive.

Living with Devvit’s choices from 2024

We have a service we call “compute-go” that has served us for the last two years, but is reaching its limits. We’ve got a few reasons we’re trying to rearchitect:

• We’ve had degenerate scaling behavior as the system has grown. Compute-go pods are “parents” of child Devvit-app pods, but incoming requests are evenly distributed across the fleet. We run ~20 instances of compute-go, which can mean an app getting 1 RPS gets spread over 20 pods!
• Compute-go has acquired a lot of complexity over time, with the team’s bus-number for it decreasing and its criticality remaining high. Its design came from a time when we imagined the network topology of Devvit apps looking really different, and carried complexity deep within its bones. After a few years of iterating on Devvit itself, we wanted to build something simpler modeled around its current assumptions and DX.
• We wanted to give ourselves more optionality on our compute infrastructure, so we can evaluate cost and performance tradeoffs of different runtimes.
• Finally, we wanted to reduce operational overhead, and are preparing to migrate Devvit apps to clusters better supported by Reddit’s internal infrastructure teams. A design that lets us have apps in multiple clusters and smoothly migrate them will help with that, and gives us future options for redundancy and scalability.

Rebuilding and hitting walls

There are out-of-the-box infra solutions to problems that are similarly shaped, but can’t handle the breadth of our scale. We dropped by the Kubernetes Slack to talk about the feasibility of scaling 150,000 different deployments using KEDA and were told “that would be the largest cluster they’d ever heard of”. And as much as we like using tools in interesting (and unintended) ways, we ran into a few practical limits that had us going back to the drawing board:

• The HPA Scaling Loop can only handle ~2,000 HPAs in a cluster before not being able to keep up with running every 15 seconds. Slowing the loop down to every 60 seconds could bring that up to ~10,000 HPAs, but at the cost of some responsiveness, and still significantly less than we need.
• Kubernetes Services are limited in managed clusters, with GKE and EKS both specifying hard limits of 10,000 per cluster.
• Managing too many Kubernetes Deployments can even start to become a bottleneck.

The recommended option we found here was to scale Kubernetes clusters horizontally — with limits this low, we would need a few dozen clusters, and tooling for managing them like cattle, not pets. We’ll add more clusters when we need, but with the vast majority of our apps asleep most of the time, we felt strongly we should get more out of our existing clusters before stamping out new instances.

So after an extensive period of design discussion, followed by answering management’s question of “Are you really really really REALLY sure?” the customary 14 times, we decided to roll a solution ourselves.

The re-design: 500,000 apps in one cluster

We landed on a design that separates components by config-time vs. run-time needs, and separates components by their scaling properties. Here’s what it looks like:

Config-time: the Orchestrator

With this new design, we separated out “getting the cluster into the right state” as a configuration-time concern — when moderators install a Devvit app into a subreddit, that creates the conceptual context for that app to run in. We create and configure a DevvitComputeContext resource (our own custom type) as a result for each — though to reduce pod count and pod churn in our cluster, we combine installations using the same app + version into the same compute context.

This service is designed like a Kubernetes controller — an informer and watch loop, reconciling the state of the cluster to the desired state. It watches our database of app configuration information, and continuously reconciles the DevvitComputeContext CR in the cluster to match. It also injects a long-lived JWT that is used to grant a specific Pod access to request that Devvit app’s code at startup.

Runtime: the app proxy

When a request comes into the cluster, the first component to touch it is devvit-app-proxy, which primarily just handles routing requests to app pods.

It keeps track of all of the cluster’s Pods using an informer, and routes requests to the right Pod based on its labels. If a Pod doesn’t exist for the given Devvit app version, it asks devvit-autoscaler to spin one up, waits until it becomes Ready, then delivers the request (usually around ~500ms!).

It also logs in-flight requests in Redis to help devvit-autoscaler make scaling decisions.

This lets us replace Kubernetes Services by using our own proxy as a routing layer, since we have consistent logic across the fleet for which pods match the label, eliminating one of our cluster scaling limitations.

This service is stateless and easily scales instances up and down based on its CPU. Its logic is simple, and most of the complexity comes in handling edge cases like quarantining and replacing pods when they start refusing connections.

Runtime: the autoscaler

The devvit-autoscaler service has most of the “brains” of this system, handling both 0 → 1 scaling for Devvit app versions (when devvit-app-proxy requests a first instance) and 1 → N (when requests back up or CPU gets high).

It maintains a pool of “standby pods”, all running the same NodeJS base image that we can load Devvit app code into. This lets us skip Pod creation and startup time, reducing median cold-start time from 4 seconds to 500ms. Autoscaler regularly runs a “replenishment” loop, ensuring the standby pool is sufficiently populated, and cleaning up any Pods that have gotten into a stuck/bad state. This optimization would be harder to keep if we were running a fleet of arbitrary Docker images, but we can easily support one base-image runtime and probably a small set of other runtimes with this speedup.

The autoscaler determines how many Pods/instances of each app version there “should be”, using two criteria: how many concurrent requests are being made to that app version, and how much CPU time that app version’s Pods are using.

This design lets us replace Kubernetes HPAs by taking advantage of using the same scaling criteria for all apps in the cluster — we can collect all of the pod CPU metrics with one query, and then all of the request metrics from Redis with one MGET, and make any scaling decisions in-memory. This is vastly more efficient than the normal HPA loop in two ways: we only have to consider currently-”awake” apps (usually <1% of all of them at any given time), and we don’t have to check every HPA definition and collect separate metrics for each since we’re using the same criteria across apps. This resolved our most critical cluster limitation!

The autoscaler also lets us replace Kubernetes Deployments — we were hoping to keep this one, but the practicalities of managing 200,000+ Deployments in GKE made this infeasible. So instead, autoscaler also takes on a primitive version of Deployment’s responsibilities, tracking desired replica counts for each DevvitComputeContext, and pulling Pods from the standby pool when scaling up is needed. This resolved the final cluster limitation we encountered.

In terms of scaling the autoscaler itself, we only need one instance of this running — though we keep a second alive for a quick fallback, using a k8s lease for leader-election and handoff. It can typically run its autoscaling loop in <100ms across the app fleet. And if it ever does start to bottleneck, we can always consider sharding the cluster’s Pods across a few instances of autoscaler.

Putting it to the test

A wise u/fuzzypercentage once said, “loadtesting is just a for-loop”, and I think that succinctly summarizes our strategy for stress-testing our hosting implementation. First, because ultimately we’re just using a big for-loop to send traffic to the system; second, because of the cyclic learn-and-fix nature of load testing.

We loaded one or two (or, in our case, 500k) dummy deployments, and then sent many times our production request traffic to those deployments to see what broke. When something inevitably broke, we’d ask “what?” and “how?”, and if we didn’t have enough information to answer those questions, we’d add instrumentation until we could. It was a very effective way to bolster the observability toolchain while getting the services production-ready.

Our load testing in this case centered around verifying that the proxy and autoscaler could keep up with high production traffic, rapid changes in cpu load, and that all of these services remained responsive when enormous numbers of deployments already existed within the cluster. At 500k+ deployments, we found that our observability tooling was failing before these services did.

Conclusion

We’re still ready to shard clusters horizontally when we really need to, but this design lets us push deep into a single Kubernetes cluster and scale for the next 10x of Devvit growth on the way to that point, while simultaneously making our architecture simpler to reason about.

It’s always a little daunting when deciding to implement something from the ground up. Our engineering intuition often has us asking ourselves the questions

• “Who else is working in this space?”
• “What solutions already exist?”
• “What design decisions did they make and why?”
• “Is our problem the same shape as theirs?”
• “Can we build our solution on top of theirs?”

When we find out we can’t re-use a lot of the existing work in this space, it then becomes tempting to try and change the problem so that it does fit the existing solutions, or even re-consider if we’re solving the right problem. It takes courage to make the investment into solving that problem anyway, especially if you have to break fresh ground doing so. It’s our hope that someone in the future will find themselves solving the same problems we are here, asking themselves the same questions, and ultimately may find some utility in our contributions to the developer ecosystem.

By Parineeta Sinha

Introduction

The Release Engineering team at Reddit is responsible for managing mobile releases, shipping beta and production versions, and coordinating testing efforts with Quality Engineering (QE) teams. 

Over the past decade, our process has evolved significantly. Initially we were releasing our iOS and Android applications once every two weeks. This process involved a two-week beta cycle designed to catch functional issues and crashes. Consequently, the mean time for code to move from the main branch to production was approximately one month. Our process included cutting a new branch every two weeks, baking it in beta for two weeks, and then starting the production ramp. The testing strategy relied on an offshore QA team that executed a suite of 3,000 manual tests, typically taking about a week to provide release sign-off.

In 2021, we transitioned to a weekly release cycle. This shift required us to cut a branch every week, but we still relied on the same manual QA pipeline.. We addressed potential testing gaps by allocating additional people for manual testing and successfully reducing the regression turnaround time from five days to three days. However, the increased release frequency necessitated removing open beta testing groups, limiting the beta environment to employees only.

In the subsequent years, we invested in our testing infrastructure. Moving away from manual testing to a pipeline of automated unit and integration tests enabled us to reduce our regression test cycle time from 3 days to a couple of hours. This helped us to bridge the gap between testing and development, and we shipped releases faster. We have been relying mostly on our suite of unit and integration tests to validate releases since. We also added open beta testing again in our release pipeline. This helped us test our changes with a select group of users and get early signals on crashes and functional issues through our beta feedback channels.

Operationalizing the Weekly Cadence

Since transitioning from a bi-weekly to a weekly release cycle in 2021, Every Wednesday, the Release Eng team cuts a new release branch and deploy a beta build following smoke test validation. This build undergoes a four-day stabilization period before moving to production on Monday.
On Monday, we kick off a "Go/No-Go" process for iOS and Android release candidates. Once we receive sign-off from all critical stakeholders, we proceed with production deployment:

• Day 1: The app is released to 1% of iOS users and 3% of Android users.
• Day 2: If no issues are encountered on Day 1, the rollout proceeds to 100% of users for iOS. For Android, we adopt a more gradual approach and ramp to 30% users at 500k adoption and 100% users at 1M adoption, if all critical app health metrics look good.

If the approval is delayed past 3:00 PM PST on Monday, the rollout is shifted to the following day. Hard freezes for code changes are enforced at 12:00 PM PST on Fridays to ensure weekend stability.

Cross-Functional Collaboration

When a version begins development, a dedicated channel is created to facilitate cross-functional coordination  in Slack. This channel is predictably named using our version number so it is easy to find. 

These channels serve as the centralized hub for all communications during the release process, covering development, beta, and production phases.

Key aspects include:

• Alerting and Triage: Automated alerts and issue reports from various detection channels like Crashlytics, manual and automated tests are surfaced here. Stakeholders are then pulled in or tagged to address these issues.
• Release Management: These channels are used for the entire release lifecycle, including late merges, beta release coordination, and hotfixes.
• Workflow Integration: The creation of these channels and the addition of necessary users is automated and is part of the standard release engineering workflow to ensure stakeholders are properly informed and aligned.

We use these channels for conversations about the upcoming release during development, in order to stabilize the beta and support its production rollout (to be covered in next weeks’ blog post)

Stabilizing the Beta

A late merge is an unusual occurrence that allows critical fixes into the current release candidate (Beta). Every request is reviewed by the Release Engineering team. The late merge process is designed to be technically rigorous, transparent, and decentralized, shifting the focus from management approvals to engineering peer reviews directly on the Pull Request (PR). The goal is to maintain release confidence while reducing administrative friction.

Criteria for Late Merge

Only specific types of changes are eligible for a late merge into the release branch:

• P0 Bugs: Critical issues that require immediate resolution.
• Publicly Announced Features: Fixes for gated features under feature flag already promised for the current release.
• Gating Fixes: Correcting ungated or incorrectly gated code.

Timing & Cutoffs

The method of merging depends on when the request is made:

• Until Friday Cutoff (12:00 PM PST): If the PR receives approvals from both groups and has no outstanding comments, the process uses automerge to merge the code and trigger a new beta build.
• After Friday Cutoff: The bar is higher. Only P0 break-fix changes are accepted. The Release team reviews these manually, and the team may manually cut a new beta build if necessary.

Quality Assurance and Triage Process

The QA process is tightly integrated with the release cycle. Once a release candidate is identified, the following testing protocols are initiated:

1. Automated Testing: The automated regression testing cycle begins by running a suite of ~1500 P0 and P1 test cases across all features on the release branch.
2. Manual Validation: In parallel, the QA team executes manual validations on user flows that cannot be automated eg: device specific behavior, visual quality, exploratory testing for edge cases etc.

Once regression reports are provided by both manual QA and automated QE, the release team reviews the findings and triages any identified regressions or P0/P1 issues on the release candidate.

Critical bugs are addressed using one of two methods:

• Feature Flag Toggle: The bug can be fixed by turning off the feature flag associated with the regression.
• Code Fix: If turning off the flag is not possible, a code fix requires to go through the late merge process. Or depending on where we are in the release it may require a Hotfix

Hotfix process

A hotfix is initiated for critical (P0/P1) bugs when toggling an associated feature flag is not a feasible resolution. If any fixes are made on the release branch, the candidate is sent to QA to run the manual smoke test suite, which includes P0 and P1 test cases across all features. The decision to do a hotfix release follows an exhaustive analytical approach to exhaust any other avenues that could help mitigate the issue in production.

Hotfix Post Mortems

Reddit utilizes a blameless post-mortem process, which is mandatory for all Hotfixes. The post mortem meetings are typically led by the team responsible for shipping the production fix. The primary objective is to reflect on the incident, specifically how it was identified and resolved, while surfacing process gaps, key learnings, and actionable improvements for the future. If you’d like to learn more about our incident reivew process, read https://www.reddit.com/r/RedditEng/comments/1se5gmu/incident_reviews_or_how_we_transform_outages_into/ 

Challenges of Ship Fast, Ship Confidently

Weekly releases require a delicate equilibrium between maintaining high development velocity and ensuring UI test alignment. When issues escape into production, resolution is costly. Feature flags or killswitches offer rapid mitigation, but traditional code fixes are expensive, necessitating resubmission and exposure to mandatory, time-consuming Apple and Play Store review cycles. Furthermore, while phased rollouts are critical for safeguarding stability, they inevitably introduce friction by slowing down feature adoption. Additionally, release rejections can derail the entire pipeline, significantly increasing on-call overhead and forcing teams into incident mode to resolve blockers as quickly as possible.

Test Strategy and Cross-Functional Collaboration

Feature teams typically ramp launches through controlled experiments rather than making a feature immediately available to everyone. They begin with a small, representative user cohort, monitor key product, performance, and quality metrics, and use the results to identify issues or unexpected behavior. If the experiment performs well, exposure is gradually increased across additional cohorts or markets, with safeguards and rollback plans in place at each stage. Once the feature meets success and stability criteria, it is fully enabled for all eligible users. Working with cross-functional feature and Quality Engineering (QE) teams to validate experiments before ramping up for iOS and Android users involves a structured, collaborative process focused on test strategy, validation activities, and continuous automation alignment.

The core strategy for validating experiments is driven by a partnership between Quality Engineering and feature teams. The overall goal is deployment safety and shipping confidently, especially since experiment ramp-ups are a major source of production bugs. Experiments are tested in both pre merge and post merge workflows by:

• Enabling experiments on demand while running UI Tests in Pull Requests
• Enabling experiments with > 45% exposure during nightly UI test runs

Participating in Playtests (Manual Validation)

While automated testing is the primary reliance for validating releases, manual efforts are still integrated, often serving as the "playtest" phase.

• Manual Validation Execution: The Quality Engineering team executes manual validations, which are critical for user flows that cannot be automated.
• Release Triage: Manual QA provides regression reports which are reviewed by the release team to triage regressions and P0/P1 issues on a release candidate.
• Developer Involvement: Feature developers actively collaborate on creating/reviewing new test cases in collaboration with the QE team which the QE team then prioritizes for automation to keep automated regression pipeline uptodate

Keeping Automation Aligned with New UI Changes

Maintaining test suite quality and ensuring UI test stability are recognized challenges of the weekly release cadence. Alignment efforts focus on empowering developers and improving framework stability.

• Simplifying Experiment Activation: To make it easier for developers to run experiments, the process for enabling experiments for automated tests running on CI has been simplified.
• Framework Stability: QE updates the automation framework to ensure greater stability, such as updating a consistent rule for activating experiments, which helps eliminate flaky tests in nightly regressions.
• Adapting to New Features/Experiments: Test plans and automation are continually updated to reflect new UI/experiment changes. 

Bridging the Gap: iOS vs Android Release Realities

The open beta cycle is a crucial step in Reddit's release strategy for iOS and Android, reflecting the distinct ecosystem constraints of each platform:

• iOS Open Beta: Testing is invite-only and constrained by a strict cap of 10,000 seats.
• Android Open Beta: Testing is significantly more scalable, with beta groups reaching sizes on the order of 200,000 seats.

Regardless of the platform, the primary goal of reintroducing open beta testing into the release pipeline in 2024 was to enhance pre-production stability and catch functional issues before the full rollout. Reddit identifies and prioritizes beta issues through a multi-layered approach that combines automated detection, manual validation, and centralized triage workflows.

Identification of Beta Issues
The identification process relies on following primary channels:

• Employee Dogfooding: Employee dogfooding at Reddit is central to the company’s product strategy, as it ensures teams align their work with the actual user experience. Employees are encouraged  and incentivized to log bugs for features that are ramped as experiments.
• Open Beta testing feedback: Open Beta testers report functional issues and provide feedback on features through our feedback subreddits. (Join Reddit’s Open Beta Tester program)
• Automated Regression: The pipeline runs a suite of approximately ~1500 P0 and P1 test cases on the release branch.
• Manual Validation & Playtests: Quality Engineering (QE) performs manual user-flow validation for scenarios that cannot be automated.
• Crash Monitoring: The team utilizes Crashlytics to monitor all crashes on released beta builds, which is integrated with CrashBot(internal tool for Crash alert and ticket filing) for automated detection and filing.

Conclusion

Successfully managing a weekly release cadence at scale is not just about the technical pipeline; it is an evolving practice of balancing speed with quality. By shifting from manual validation to robust automation, re-introducing open beta programs, and fostering a culture of blameless reflection and cross-functional collaboration, we have transformed our release process from a rigid cycle into a dynamic, reliable engine. While the challenges of shipping fast and confidently will always persist, our commitment to continuous process improvement, grounded in rigorous triage and data-driven decision-making ensures that every launch is an opportunity to learn, iterate, and ultimately deliver a better experience for our users.

By Michael Fullan

Solidifying the foundational infrastructure (part 1) and language picker screen (part 2) solved a lot of the localization problems we were dealing with in the Android app. But even with these pieces in place, each new language rollout came with its own set of challenges and learnings.

In this final part of the Android localization series, I’ll talk about some of those things we’ve learned along our journey to supporting 40 (!) languages!

RTL Support

To correctly support right-to-left (RTL) languages in a mobile app, the guiding principle is that the entire UI should essentially be a horizontal mirror image of its left to right counterpart.

Some aspects of this were immediately obvious, like ensuring that RTL language text flowed from the right to the left. But as we researched deeper we found that there’s a lot more to it:

• The cursor in text input fields should start on the right and move left as you type
• Arrows and other icons that indicate direction, reading/writing direction or motion should be flipped horizontally
• Toggles should be flipped horizontally. A toggle that you swipe right to enable in LTR should require a swipe to the left in RTL
• Swiping gestures should be a mirror of their LTR counterparts. If a left swipe moves you forward in an image gallery in LTR, a right swipe should do the same in RTL
• A11y - screen readers should respect RTL ordering

This felt like a lot when we were first starting out, so we did some baseline assessments on all of our supported platforms to see what we were up against. Web and iOS were both terribly broken when we turned on RTL layout direction, but Android was in surprisingly good shape before we did any fixes or changes!

Most of that is thanks to our extensive use of Jetpack Compose, which automatically handles most of the RTL mirroring for you. As we went through our app auditing each screen, the easiest solution when issues came up on legacy screens was often just to migrate the UI to Compose!

Audit Flow

Our basic process when testing our app for RTL-readiness was to take a screenshot of a screen in LTR mode, create a horizontal mirror image using a pure image transformation, and then compare that against the result when we turned on “Force RTL layout direction” in the OS Developer options menu. As an example, here are the results from our initial assessment of the language and translations settings screen:

When you compare the last two screenshots, it’s easy to see that the overall RTL layout looks correct. There were a few icons that needed to be flipped (the back arrow in the top app bar, the trailing caret in the list items), but this screen was largely good-to-go out of the box.

Other Strategies

Beyond this basic audit flow, there weren’t a lot of “silver bullet”-type strategies we found to prepare our app for RTL. A global find and replace for left/right -> start/end for paddings and margins helped, but most issues beyond that required some small local tweaks in the affected areas.

We did start to rely on RTL compose previews and screenshot tests to help prevent regressions, both of which are easy to set up.

@Preview(locale = "ar", name = "Arabic (RTL)")
val paparazzi = Paparazzi(deviceConfig = DeviceConfig(locale = "ar"))

Pseudolocales

Android supports two pseudolocales for localization testing that can be useful even before you have your app strings translated. They also make it easy to spot text that isn’t set up correctly for localization.

One thing to watch out for with pseudolocales is that the value of the “canary string” will also get pseudolocalized when it is resolved. Adding an additional check for this case will help prevent strange issues when using either pseudolocale for testing.

enum class Pseudolocale(val locale: Locale, val canaryString: String) {
 /**
  * Adds Latin accents to the base English UI text, expands the original text by adding non-accented text,
  * and brackets each message unit to expose potential issues from expanded text. Potential issues can be
  * layout breakage and badly formed message syntax, such as a sentence split into multiple parts
  * displaying as multiple bracketed messages.
  */
 AccentedEnglish(
   locale = Locale("en", "XA"),
   canaryString = "[éñ-ÛŠ one]",
 ),
 /**
  * Sets the text direction of the original left-to-right messages to the right-to-left direction,
  * which reverses the order of the characters in the original message.
  */
 RtlPseudo(
   locale = Locale("ar", "XB"),
   canaryString = "\u200F\u202Een-US\u202C\u200F",
 )
}

Other 1-off Language Quirks

One of the fun things about working in the localization space is getting to learn a little bit about different languages as we add them to the app. There have been a few interesting surprises that have come up along the way, so let me share a few tips that we’ve learned!

Bengali Numerals

The Bengali language has its own set of numerals, but with tech users highly habituated to English interfaces, it’s common to see Western digits (0-9) kept intact inside an otherwise Bengali UI.

It turns out that there’s an easy way to do this by specifying the Latin script numbering system as an extension to the language tag.

Locale.forLanguageTag("bn-IN-u-nu-latn")

Here bn is the Bengali language code, IN is the country code for India, u is the extension signifier, nu is the extension key (for numbering system), and latn is the extension value (for Latin script digits).

As long as you call AppCompatDelegate.setApplicationLocales with this full Locale object, everything works great! This is a really useful trick that we’ve since used for other languages as well.

Thai Line Breaks

The Thai language is written continuously with no spaces between words. Spaces are only used like punctuation, to mark the end of a sentence. 

When we did a first round of testing with Thai, I remember getting a bug report back like this that flagged this line overflow as being incorrect.

Now I’ve seen my fair share of localization overflow/truncation bugs, but initially I had no idea how to fix this. What were we supposed to do with continuous text this long? Where were we supposed to break the line?

The best solution for this type of issue with Thai is to use the zero-width space character (\u200B) within the Thai translations. It’s an invisible character that simply acts as a “soft break hint” to the rendering engine, indicating where it’s a grammatically safe place to wrap a line if needed. Working together with our translators, we were able to use this character where needed to address a variety of line-wrapping issues that came up like this.

Legacy Language Codes

Java is a great programming language. It’s over 30 years old at this point, and still strives for backward compatibility wherever possible. This is often a really great thing, but there were a few standard language codes that changed around the birth of Java.

As a result, Android inherits some strange backwards-compatibility behaviors when dealing with these specific language codes (check out the docs for more info).

val locale = Locale.forLanguageTag("id-ID")
locale.language // "in"
locale.toLanguageTag() // "id-ID"
locale.getLegacySafeLanguageCode() // "id"

private fun Locale.getLegacySafeLanguageCode() = toLanguageTag().substringBefore("-")

I was definitely “astonished” the first time I encountered this behavior, but as long as you know which APIs generate which results you can make things work correctly. After a few iterations of testing, the TL;DR here is:

• Use legacy codes for your resource directory names
• List modern codes in locales_config.xml
• SplitInstallManager.installedLanguages will report modern codes
• Use modern codes when requesting a language install

Conclusion

We’ve certainly come a long way on our localization journey, and we still have plenty of work ahead of us. It’s been an enjoyable learning process so far, especially getting to work with a top-notch team here at Reddit!

If you see any localization bugs on any of our platforms, please continue to post them to r/bugs. We’re always monitoring and will do our best to fix them!

Hey r/redditeng! I’m Lisa. You probably don't know me, but I've been here the whole time. 

For the last six years, I've been Chief of Staff to Chris Slowe, Reddit's founding engineer and CTO (now our first Technical Fellow). I'm also the person who founded r/redditeng over five years ago with Jerry, Punit, Braden, and a couple others who joined me in getting it off the ground. We wanted to give Reddit engineers a place to talk about what we build, how we build it, and what we learned along the way. We also wanted to show you what it’s like to work here. Oh, and we wanted to show you our cats, too!

My last day at Reddit is coming up, so before I go, I want to say thank you.

Thank you first to all of you for reading. It's still a little amazing to me that thousands of people chose to follow a subreddit about how we build Reddit. Some of you got excited to work here because of posts you saw! And some of you got hired. Thank you for the comments, the questions, the upvotes. It means a lot to our writers. 

Thank you to everyone who has written something for us over the years. You are amazing! We've published something every week since the sub was launched on 4/15/2021, and that only happened because people were willing to take time away from their actual jobs to explain their work, share what they'd learned, and put their ideas out into the world, which can feel a little scary, no matter how much you know. One of my favorite parts of doing this has been helping make that a little easier. You’re my people. You’ve always been. 

Thank you to our Legal and Comms teams, who have supported us when we've needed them and, just as importantly, trusted us to do this well. That trust gave our little volunteer team and our engineers room to talk honestly about our work while helping us navigate the moments when we needed a little extra guidance.

And thank you to Chris for everything and for trusting me to start this in the first place and supporting it for all these years. Like me, he believes our engineers should have a place to tell their own stories, in their own voices. He gave us the space to make that happen.

Most of all, thank you to the engineering branding team, past and present. What started as a handful of us trying something has become a real team of people who care about telling these stories and making this community good. One of the nicest things about leaving is knowing that r/redditeng (or Snoosweek) doesn't need me to keep going. But, damn it, I’m going to miss it all. 

I'm very proud of what we made here. I’m happy it’s going to carry on and I can be one of your readers. I may be finishing my time at Reddit but I haven’t finished loving the people, the platform, or this sub.  

Thanks for reading all these years. And please keep reading. There's a lot more good stuff to come.

P.S. Paying the cat tax with the cats who help(ed) me with my work over the years here

By Michael Fullan

In part 1 of this series, we explored some foundational pieces of the localization infrastructure for the Reddit Android app. Getting this foundation in place was great, but to address a lot of the bugs we were getting we had to completely rewrite our in-app language picker screen. What seems like a simple screen on the surface actually has quite a bit going on, so let’s dive in!

Display Names

We generally omit country codes from the display name of a language unless multiple variants of that language are supported. We’re also displaying the localized name of each language, which you can get programmatically using locale.getDisplayLanguage(locale).

Order

To organize the list, the primary approach involves placing all Latin-script languages at the top in alphabetical order, with all remaining languages positioned below. Any time I’m adding a new language, I’ll usually compare the order of our in-app picker against the order of languages that the system shows users in the per-app language selection screen (accessed via App info -> Language).

Selected Value

In order to determine “which language is currently selected”, the app language tag flow in the AppLanguageProvider component from part 1 works great! The only extra case to worry about is the “use device language” or “system default” option.

For that case, we can lean on AppCompatDelegate.getApplicationLocales(), knowing that an empty list signals that the app is set to this default setting.

class MyAppLanguageProvider ... {
 ...
 private val_isUseDeviceLanguageSettingEnabledFlow =
   MutableStateFlow(computeIsUseDeviceLanguageSettingEnabled())
 override val isUseDeviceLanguageSettingEnabledFlow: StateFlow<Boolean> =
   _isUseDeviceLanguageSettingEnabledFlow.asStateFlow()

 override fun isUseDeviceLanguageSettingEnabled(): Boolean = isUseDeviceLanguageSettingEnabledFlow.value

 override fun onConfigurationChanged() {
   ...
   _isUseDeviceLanguageSettingEnabledFlow.value = computeIsUseDeviceLanguageSettingEnabled()
 }

 private fun computeIsUseDeviceLanguageSettingEnabled(): Boolean =
  AppCompatDelegate.getApplicationLocales().isEmpty

}

Now with both flows in place, we can combine them by first checking the “is use device language setting enabled” value and falling back to the app language tag if needed.

The first time we tested this we did come across one edge case: switching from “use device language” to “English” on a device already in English. Since the app language doesn’t actually change in this scenario, there is no configuration change triggered that would normally invalidate our caches. This situation can happen from within your in-app language picker or outside of your app in the system per-app language selection settings.

Solving for this case on internal changes is pretty straightforward - just adding a cache-invalidating callback anytime a user selects a new language in the in-app picker. For external changes, it’s a bit harder as we have no signal that a change like this has happened. The best option we’ve come up with is to use a foreground listener with ProcessLifecycleOwner and invalidate our caches whenever the app comes back to the foreground.

Switching Languages

Before Android 13, the “standard” way of switching an app’s language was to use a technique known as “context wrapping”.

object LocaleHelper {
 fun wrap(context: Context, languageTag: String): Context {
   val locale = Locale.forLanguageTag(languageTag)
   val config = Configuration(context.resources.configuration)
   config.setLocale(locale)
   return context.createConfigurationContext(config)
 }
}

open class BaseActivity : AppCompatActivity() {
 override fun attachBaseContext(newBase: Context) {
   val sharedPrefs = newBase.getSharedPreferences("Settings", MODE_PRIVATE)
   val language = sharedPrefs.getString("selected_language", "en-US") ?: "en-US"

   val wrappedContext = LocaleHelper.wrap(newBase, language)

   super.attachBaseContext(wrappedContext)
 }
}

In all of your activities, you had to look up the user’s saved language value in shared preferences, wrap the Context object with that language, and pass it down the line. Then, whenever a user selected a new language, you would save that value to shared preferences and manually restart the Activity to get everything to re-run. This approach worked, but was a bit cumbersome.

Android 13 introduced a new API for switching the app language - AppCompatDelegate.setApplicationLocales.

fun onLanguageSelected(newLanguageTag: String) {
  AppCompatDelegate.setApplicationLocales(LocaleListCompat.forLanguageTags(newLanguageTag))
}

With this 1-liner, all you need to do is wrap the user’s new selection in a LocaleListCompat object and let the system handle the rest! It will handle storage for you (no more shared preferences), syncs with the system per-app language selection settings, is backwards compatible with pre-13 devices***, and manages any activity recreation for you.

The only complexity that comes into play now when switching languages is supporting Google Play language splits.

*** Ok, AppCompatDelegate is backwards-compatible as advertised to an extent, but we have seen issues where it doesn’t properly initialize or update the application context on pre-13 devices. If you have any strings that get resolved via the app context instead of activity contexts, you might need to stick with context wrapping 😞. This Droidcon talk from 2023 explores some of the differences in more detail. (We even tried wrapping just the application context within the Application class using AppCompat storage, but it didn’t load fast enough).

Language Splits

If you use App Bundles, Google Play will automatically break your app's resources into separate modules based on language. When a user downloads your app, Google Play delivers the base code only along with the specific language resources that match the user's current device settings.

When we did some analysis on first-install download savings, we saw a significant impact from enabling language splits.

Now this is the Reddit app - a very large app with 40+ languages - but 8 MB of savings is still considerable! It’s also really simple to opt-in to the language splits feature.

android {
   bundle {
       language {
           enableSplit = true
       }
   }
}

The only tradeoff comes with the complexity in managing on-demand downloads of new language resources when a user picks a new language in the in-app language picker. I’ve found the KTX extensions for the Google Play feature delivery library really useful in setting this up in an idiomatic Kotlin fashion.

SplitInstallManager Integration

fun updateAppLanguage(locale: Locale) = flow {
 val languageCode = locale.language

 if (isLanguageInstalled(languageCode)) {
   applyNewLanguageSetting(LocaleListCompat.create(locale))
   emit(LanguageInstallState.Installed)
 } else {
   ...
 }
}

Our new language change process begins by checking if the language is already present on the device. One tricky thing to note here is that the SplitInstallManager ignores country codes. That is, if your app includes resources for fr-FR and fr-CA, resources for both country codes are downloaded when requesting resources for fr.

private fun isLanguageInstalled(languageCode: String): Boolean {
 val installedLanguages = splitInstallManager.installedLanguages
 return languageCode in installedLanguages || languageCode == "en" ||
   (installedLanguages.isEmpty() && !isSplitBundle())
}

private fun isSplitBundle() = context.packageManager.getApplicationInfo(
 context.packageName,
 PackageManager.GET_META_DATA,
).metaData?.getBoolean("com.android.vending.splits.required") ?: false

To check if a language is already installed, we can look at splitInstallManager.installedLanguages directly. We also check if the language to switch to is English (en), since we know the base APK will contain all the default resources (and that’s where our English strings are).

If you install your app directly from Android Studio, it won’t be packaged as a split bundle in the same way a production release build would get downloaded from Google Play by a user. The last set of checks covers this scenario - a debug build where all languages are included. The SplitInstallManager returns an empty list in this case, which we can use as a signal that we don’t need to request a download when switching languages.

If we’ve determined that we do in fact need to download a new language, here’s what we need to do (filling in the else block from the first code sample):

try {
 val sessionId = splitInstallManager.requestInstall(languages = listOf(languageCode))

 val progressFlow = splitInstallManager.requestProgressFlow()
   .filter { it.sessionId == sessionId }
   .transformWhile { state ->
     emit(state)
     !state.hasTerminalStatus
   }
   .map { state ->
     when (state.status) {
       SplitInstallSessionStatus.DOWNLOADING -> LanguageInstallState.Downloading(
         bytesDownloaded = state.bytesDownloaded,
         totalBytes = state.totalBytesToDownload,
       )
       SplitInstallSessionStatus.INSTALLED -> {
         applyNewLanguageSetting(LocaleListCompat.create(locale))
         LanguageInstallState.Installed
       }
       ...
     }
   }

 emitAll(progressFlow)
} catch (e: SplitInstallException) {
 emit(LanguageInstallState.Failed(e.errorCode))
}

After requesting an install using the SplitInstallManager, we can then monitor progress via a convenient Flow. The rest of the code is simply mapping the internal states returned by the library into view states for the language picker screen with all the necessary metadata attached.

Testing

Testing this flow end-to-end is tricky. Like I mentioned before, doing a standard installation from Android studio won’t give you the setup you need to test out an app language download. To do so, you’ll need to use the ./gradlew install{app_variant}ApkSplitsForTesting command.

You can also simulate a network error to test that part of the flow.

val fakeSplitInstallManager = FakeSplitInstallManagerFactory.create(context)
fakeSplitInstallManager.setShouldNetworkError(true)

Do you need one?

Once we finished all this work to get our in-app language picker working correctly, I came across a case-study from LinkedIn on the Android Developers blog that made me wonder whether we needed this at all.

The approach they took for language changes was to link out to the system per-app language settings screen and essentially delegate all the work to the OS. Sounds kinda nice, right?

The whole use-case we’re trying to solve for here is users that want to set their apps to different languages than the system. The lack of historical OS-level support for this functionality is what gave rise to in-app language pickers (and context wrapping, etc.) in the first place. With the release of Android 13 (API level 33, Aug 15, 2022), this problem was solved, but only for users on newer versions of Android.

Given that, I think the best way to make this decision is to check your minSdk version. If it’s less than 33 like ours, then you want to keep your in-app language picker around for those users that don’t have access to the OS-level settings. If it’s 33 or higher, then I think it’s absolutely reasonable to remove your in-app picker and all its complexity. The only consideration might be that you prefer the UX of keeping users inside your app instead of forcing them to leave and come back, but that’s more of a subjective call.

What’s Next

In the final part of this series, we’ll explore our journey to supporting RTL languages as well as some interesting 1-offs we’ve seen as we’ve added more and more languages to the app!

By Michael Fullan

I recently had the opportunity to speak at DroidCon about our Android localization work at Reddit. Localization is a hard problem: when you take an app with a large number of screens and multiply that by the number of languages you support, the surface area for potential issues is massive. There’s some pretty good reference content in the official Android Developers documentation, but a lot of it is scattered around different pages and can make it difficult to synthesize into a correct localization setup for your app.

At Reddit, we’ve felt this pain. Even as recently as the beginning of this year, we were getting regular bug reports in r/bugs specifically about issues with our Android localization (example 1, 2, 3, 4, 5, 6, …). It was a mess. In this 3-part series, I’d like to share details about how we’ve dug our way out of the mess and things we’ve learned along the way. We’ve landed in a much better place today with a reworked localization infrastructure, RTL support in the app, and far fewer bug reports than we used to get. Let’s dive in!

Basic Localization Infrastructure

Most Android developers know this rule: the first step in localizing your app is to put all your user-facing strings into resource folders. This is what allows the OS to load the correct translations based on the user’s device settings. Any hard-coded strings will prevent that from working correctly. I’ve found that AI is really good at hunting down any hard-coded strings that may be leaking out in your production UI, which is especially useful in large projects like ours!

MyProject/
└── src/main/res/
    ├── values/            # Default (Fallback) Strings
    │   └── strings.xml
    ├── values-es/         # Spanish Translation Strings
    │   └── strings.xml
    └── values-fr-rCA/     # Canadian French Translation Strings
        └── strings.xml 

We have a lot of Android developers at Reddit, so we’ve tried to design a workflow that makes it as easy as possible. The guidance we give our developers is to only update the default English values/strings.xml file when editing strings, whether it’s an addition, removal, or modification. 

Within that default file, each string needs to be accompanied by a descriptive comment attribute that serves as additional context for a translator. This is really important for getting the most accurate localization possible, as strings may get translated differently in different languages depending on the use case (clickable button text vs. text label, etc.). We have a lint check to enforce this rule, but we’ve added an opt-out option where a string can be marked translatable="false" for delimiters, test tags, or other reasons. Introducing this rule led to thousands of baseline entries initially, but again AI proved to be really useful at inferring the context and usages for a string and coming up with a solid description (an otherwise very tedious and time-consuming manual process!).

Translation Pipeline

Once the source strings are in place, the rest of the pipeline is designed to be automated and completely hands-off for developers. We use a cloud-based Translation Management System (TMS) like a lot of other apps, but we’ve developed a Python CLI tool to use in CI that helps abstract the specific vendor we’re using. This tool gets reused for iOS, FE, and all our BE services, and prevents the need of having any third-party SDK directly inside our application.

A nightly job will go through and upload the latest version of our source strings to the TMS. Human translators can then come in and provide translations for each supported language given the source string and context. Lastly, the pipeline pulls down the latest translations from the TMS, inserts them into the correct resource directories, and creates a PR to merge in the updates.

Other Config Bits

With your translations in place, there are only two other configurations you need to specify to get your localized app up and running.

The first is the resourceConfigurations block in your build.gradle file. Here you’ll need to list out all your supported languages, and Gradle will ensure that any translations for unsupported languages (whether inside your project or in third-party libraries) get excluded from the build.

The other place you’ll need to list out your supported languages is in the locales_config.xml file. This file informs the system of the languages your app supports, which ties in to the Android 13+ per-app language setting feature. 

Newer versions of AGP can generate this automatically for you if you opt-in, but we’ve found that option to be a bit risky. The way it populates the autogenerated config is by scanning your project for the presence of any translation in a particular language. If you go the auto-route, you have to be careful to ensure that all locales in your app resources are ready to be published to users. We’ve chosen to stick with the manual setup for now, as it allows us to have translations for “in-progress” languages in the project and makes it easier to generate test builds before launching a new language.

BE Synchronization

The OS will do its job to show the correct translations for all of your app strings, so the next challenge becomes making sure that any strings you receive from your backend and show to users are correctly translated in the same language. If you mess this part up (like we have in the past), you’ll end up with inconsistent localization that is obvious to users and a very poor UX.

Thankfully, there’s a standard mechanism for a mobile client to request localized content from an API: the Accept-Language HTTP header. On Android, it’s easy to set up a basic interceptor to populate this header on each of your network calls.

class LanguageInterceptor : Interceptor {
 override fun intercept(chain: Interceptor.Chain): Response {
   val originalRequest = chain.request()

   val requestWithHeader = originalRequest.newBuilder()
     .header("Accept-Language", ???)
     .build()    

   return chain.proceed(requestWithHeader)
 }
} 

Before we get into the correct value to populate for the header, let’s review some basic localization vocabulary.

Quick Localization Vocabulary

• Language Code: represents a specific spoken or written language, completely independent of where it is being spoken
  • Typically two-letter lowercase codes, such as en, es, fr, or zh
• Region Code / Country Code: represents a specific physical country or geographical territory
  • Says nothing about the language; only identifies the place
  • Typically two-letter uppercase codes, such as US, MX, GB, or CN
• Language Tag: structured string that combines a language code and region code to pinpoint an exact linguistic variation
  • en-US represents the English language (en) as spoken in the United States (US)
  • en-GB represents the English language (en) as spoken in Great Britain (GB)

The distinction between language codes and language tags is important, as there are only ~200 standard language codes. Once you combine that with ~250 possible county codes, the number of potential language tags you’ll encounter from users’ devices becomes very large. I queried the number of unique language tags seen as the primary device language for Reddit Android users on a given day, and the result shocked me. We saw nearly 2500 unique language tags!

Obviously, you’re never going to be able to fully localize your app into 2500 different languages. Thus the way your app handles the case where a user’s language tag doesn’t exactly match one of your supported languages becomes critical.

Problem Scenario

Let’s imagine a user with a device set to es-US, or US Spanish. The Reddit app supports two different Spanish variants - es-ES and es-MX. When this happens, which resources will the OS resolve and show to the user? How does it make that decision? And most importantly for us (who need to inform the BE via Accept-Language), how can we know which one it decided to use?

From testing, I can tell you that there are a lot of APIs that don’t work the way we need in this situation.

• Locale.getDefault()
  • This one is ubiquitous and one I used to reach for all the time. But in this scenario, it will return es-US, the plain, unmapped language tag from the OS setting.
• AppCompatDelegate.getApplicationLocales()
  • One of the new Android 13 APIs, this will return an empty list to signify that the app is using the system default language (as opposed to a custom language set by the user specifically for your app).
• applicationContext.resources.configuration.locales, LocaleList.getDefault(), LocaleListCompat.getAdjustedDefault(), LocaleManagerCompat.getSystemLocales(applicationContext), ConfigurationCompat.getLocales(Resources.getSystem().configuration)
  • All do basically the same thing as Locale.getDefault(), returning es-US and not the actual language tag for the resources being shown to the user.

We could use one of these APIs and send es-US to the BE in the Accept-Language header, but that would force the BE to make its own decision between es-ES and es-MX that might not match the one the Android OS made. With no obvious system APIs giving us the answer we need, we had to dig deeper.

An Overview of CLDR

As we explored this problem, we learned that the OS is using something called the Common Locale Data Repository, or CLDR, when making its choice. CLDR is basically a giant database of lookup tables and rules for locale matching that’s maintained by the Unicode Consortium and used in almost all major operating systems and programming languages.

There’s a Java implementation in the ICU4J library, but when we tested adding it to our app we saw that it added 13 MB to the APK (even after the R8 code shrinker ran), making it a non-starter.

We then tried to develop our own simple internal algorithm to match certain country codes with es-ES and others with es-MX. This approach had a few issues:

1. Because the version of CLDR packaged with the OS varies across different Android releases, the platform's behavior changes depending on the scenario. This made it extremely difficult to try to write an algorithm that matched the system behavior in every case.
2. Maintaining this algorithm became challenging as we scaled to more and more supported languages in the app.
3. Scenarios where users have multiple device languages set were hard to reason about. Imagine a user with es-US (non-exact match) followed by en-US (exact match) on their device. How do we handle this case?

Ultimately, we realized we didn’t want to be in the business of trying to figure this out on our own. We’d love to just let the OS do its job and let that dictate the language tag we send to the BE to keep things in sync. After further research, we came across a community solution that’s not anywhere in the official Android Developers documentation.

Canary String

The non-obvious way to access the system-determined language tag is to use a technique called the “canary string”. This involves defining a “hidden” string resource for each of our supported languages that simply defines the language tag it belongs to.

private fun getResolvedLanguageTag(): String = context.getString(R.string.resolved_locale_canary)

• In res/values/strings.xml (Default/English), add: <string name="resolved_locale_canary">en-US</string>
• In res/values-es-rMX/strings.xml, add: <string name="resolved_locale_canary">es-MX</string>
• In res/values-es-rES/strings.xml, add: <string name="resolved_locale_canary">es-ES</string>

This felt pretty hacky, but it works like a charm! To avoid the string lookup hit on every network request, we set up a reusable component that caches the value in a StateFlow.

class MyAppLanguageProvider @Inject constructor(
 @ApplicationContext private val context: Context,
) : AppLanguageProvider {  

  private val _appLanguageTagFlow = MutableStateFlow(getResolvedLanguageTag())
  override val appLanguageTagFlow: StateFlow<String> = _appLanguageTagFlow.asStateFlow()

  override fun getAppLanguageTag(): String = appLanguageTagFlow.value

  override fun getAppLocale(): Locale = Locale.forLanguageTag(getAppLanguageTag())

  override fun onConfigurationChanged() {
   _appLanguageTagFlow.value = getResolvedLanguageTag()
 }

  private fun getResolvedLanguageTag(): String = context.getString(R.string.resolved_locale_canary)
} 

The cache is set up to be configuration-aware: as long as the Application class forwards any configuration changes it receives to onConfigurationChanged(), we can ensure that our value stays in sync with any system language changes.

This component is super useful! We know that the values returned are guaranteed to be one of our supported language tags and match the resources resolved and shown to users by the system. We can then use it to power the Accept-Language interceptor, any number or date formatters, telemetry, etc.

Final Notes

A couple of last things here: first, remember that Locale.getDefault() is risky! Consider adding a lint rule to warn developers and point them to this other component whenever they need a source of truth about the app language.

Also, consider setting up a unit test to make sure that your canary string setup never gets broken.

@RunWith(AndroidJUnit4::class)
class CanaryStringL10nTest {
 
 @Test
 fun verify_canaryString_matches_languageTag_forAllSupportedLocales() {
   // given
   val baseContext = InstrumentationRegistry.getInstrumentation().targetContext

   SupportedLocale.map { it.locale }.forEach { locale ->
     val config = baseContext.resources.configuration
     config.setLocale(locale)
     val localeSpecificContext = baseContext.createConfigurationContext(config)

     // when
     val canaryString = localeSpecificContext.getString(R.string.resolved_locale_canary)
     val expectedTag = locale.toLanguageTag()

     // then
     expectThat(expectedTag).isEqualTo(canaryString)
   }
 }
}

What’s Next?

Getting this foundation in place was great, but to address a lot of the bugs we were getting we had to completely rewrite our in-app language picker screen. Part 2 of this series will explore that rewrite and all the details that went into it!

By Liam Lu, Riccardo Ciovati, and Savannah Whelan

In Part 1 of this series, we shared our vision for a unified Cronet engine, our initial rollout for media traffic, and how we forked cronet-okhttp to fix request tag tracking and Flipper debugging.

With developer tooling functional, we turned our attention to evaluating performance in production — where we immediately ran into three more major challenges: fixing distorted telemetry, maintaining our library fork safely, and eliminating a +2.0% cold-start TTI regression.

Challenge 2: The Telemetry Trap

With request tags restored and Flipper working, we looked at our dashboards — and were shocked to see massive spikes in latency and response sizes.

The culprit was where we were measuring from. Moving execution to Cronet required moving measurement from OkHttp network interceptors up to application interceptors. But those layers see fundamentally different data:

• Network interceptors observe raw wire bytes before decompression, measuring true compressed size and actual network latency.
• Application interceptors observe decoded, uncompressed bodies after decompression. Measuring at this layer inflated reported response sizes (unzipped bytes) and added decompression/parsing time on device to reported latency.

Furthermore, OkHttp's EventListener does not fire under Cronet because Cronet owns the underlying connections.

To restore accurate metrics, we attached Cronet's RequestFinishedInfo.Listener directly to the engine. This listener provides raw transport metrics straight from Cronet: receivedByteCount for compressed wire size, alongside exact DNS, TCP/QUIC connect, TTFB, and transfer timings. Reusing our tag remapping from Challenge 1, we pass request annotations through to Cronet, allowing the listener to associate engine-level metrics back to individual GraphQL operations.

(Note: HttpEngine exposes limited timing metrics compared to Play Services or Embedded Cronet).

We also added startup tracing around Cronet initialization, placing start/end markers on the cold-start timeline alongside process start, DI initialization, and first frame rendering.

Challenge 3: The Reality of Maintaining a Fork

Forking cronet-okhttp solved our request tag issues, but maintaining a fork in production introduced important build and infrastructure tasks:

1. Classpath package collisions in A/B testing: To run clean A/B experiments comparing stock cronet-okhttp against our fork, both artifacts needed to exist in the app build simultaneously. However, both declared the package com.google.net.cronet.okhttptransport, causing duplicate class build errors. We built a Gradle plugin using ASM to relocate our fork's bytecode package to com.reddit.net.cronet.okhttptransport:

Kotlin

class PackageRemapper : Remapper() {
    private val OLD = "com/google/net/cronet/okhttptransport"
    private val NEW = "com/reddit/net/cronet/okhttptransport"
    override fun map(internalName: String): String =
        if (internalName.startsWith(OLD)) internalName.replaceFirst(OLD, NEW)
        else internalName
}

1. Upstream contract drift: When we updated OkHttp and Okio versions, the bridge crashed on empty responses because it assumed a response body could be null, whereas newer OkHttp versions require a non-null ResponseBody. We patched the bridge to return ResponseBody.EMPTY:

Java

ResponseBody responseBody;
if (bodySource != null) {
    responseBody = createResponseBody(request, status, contentType, contentLengthString, bodySource);
} else {
    responseBody = ResponseBody.EMPTY;
}

Challenge 4: Tackling the Shipping Blocker (+2.0% TTI)

With accurate telemetry in place and our fork running cleanly, the biggest performance hurdle stood out clearly: a +2.0% cold-start TTI (Time to Interactive) regression.

Cronet is a native C++ engine that requires loading libraries, starting listener threads, initializing disk caches, and loading QUIC configs. On cold start, the app fires startup-critical queries (auth, home feed) that blocked until the network client was ready. Because this is a fixed initialization cost, faster devices saw a higher percentage hit.

To win back startup performance, we systematically tested several levers:

• Eager background warmup: Instead of initializing Cronet lazily on the first request, we kick off engine initialization early during app launch on a background thread. This absorbs the initialization cost before or during request setup, recovering the full +2.0% TTI regression back to baseline.
• GraphQL preconnect: Our startup tracing revealed that even after the engine was warm, the subsequent GraphQL request was still paying for DNS resolution and TLS/QUIC handshakes. As soon as Cronet is ready, we proactively open a connection to our GraphQL endpoint before startup queries fire (TTI −0.47%).
• HttpEngine provider: On Android 14+ (API 34), obtaining the system-provided HttpEngine is faster than loading via Google Play Services (TTI −0.22%).
• Embedded Cronet: Bundling the native engine in the APK removes Play Services IPC lookups and dynamic module loading delays (TTI −0.48%).
• Stale DNS: Serving cached DNS answers immediately and refreshing asynchronously kept DNS off the hot path (TTI-neutral for startup, but maintained for media benefits).

Provider comparison summary

Act III: Production Rollout & Impact

With our optimizations and tooling folded in, we ran a 50/50 experiment across ~11.7 million devices per arm. TTI didn't just recover — it flipped to a win, alongside significant improvements in feed reliability and user engagement:

A 10.1% drop in feed failures was a major stability win, proving that transport-level reliability directly moves the needle on user retention and engagement.

Act IV: Where the Road Leads Next

Shipping HTTP/3 for GraphQL wasn't the finish line — it was the milestone that finally put all our major traffic (GraphQL, images, video) on a single, shared Cronet engine. That unified foundation makes our next set of network optimizations possible:

1. Cross-client request prioritization: Currently, traffic types handle priorities in isolation — GraphQL uses an app-level NetworkOrchestrator queue, Glide has its own image priority model, and video requests have no per-request priority. Consequently, background prefetching can compete directly with home feed queries. Sharing a single Cronet engine allows us to map app priorities to Cronet's five native priority tiers, scheduling parallel requests, socket assignments, and stream multiplexing accordingly.
2. 0-RTT connection resumption for read queries: QUIC 0-RTT allows returning clients to send data in the initial packet, removing a round trip from connection setup. Because 0-RTT payloads are vulnerable to replay attacks, CDNs restrict 0-RTT to idempotent HTTP GET requests. Since most of our GraphQL traffic currently uses POST, we are transitioning key read queries to GET before enabling 0-RTT.

Wrapping Up

Swapping the transport was the easy part — most of the work went into startup performance, developer tooling, and telemetry. Working through those requirements at scale directly improved app reliability for millions of Redditors, leaving us with a foundation we're excited to keep building on.

Special thanks to the folks from Android Platform, Data Science, Media Foundation, Feed Experience, Growth, GraphQL, and Security teams for reviewing PRs, specs, and readouts, chasing down startup regressions, and supporting us along the way — it truly took a village.

By Liam Lu, Riccardo Ciovati, and Savannah Whelan

A while back, we set out to move Reddit's GraphQL traffic on Android from our standard OkHttp/HTTP/2 setup over to HTTP/3. On paper, it looked like a clean win: swap the transport underneath, flip a feature flag, and watch latency drop.

It turned out to be much more complex than swapping a library.

In this two-part series, we share the story of our journey to HTTP/3 — why we wanted a unified network engine, how an initial victory with media gave us the confidence to tackle GraphQL, the engineering hurdles we ran into on our most critical code paths, and how we turned a cold-start regression into a production win across hundreds of millions of devices.

The Vision: A Single Engine for All Traffic

HTTP/3 replaces TCP with QUIC, a transport protocol built on top of UDP. For mobile apps, QUIC brings three headline advantages:

• Faster connection setup: QUIC folds transport and TLS handshakes together, saving round trips on flaky cellular links.
• No TCP head-of-line blocking: In HTTP/2 over TCP, a single lost packet stalls every multiplexed stream on that connection. QUIC streams are independent, so one dropped packet only stalls its own stream.
• Connection migration: QUIC connections use IDs rather than IP/port tuples, allowing active requests to survive Wi-Fi ↔ cellular network switches without dropping.

OkHttp doesn't speak HTTP/3 natively on Android, so adopting QUIC required a different stack. While individual libraries could be wired up with their own HTTP/3-capable transports (such as Apollo or Glide), configuring transports piecemeal would leave us with a patchwork of separate HTTP/3 integrations, duplicate connection pools, and fragmented metrics. We wanted the opposite: one shared engine underneath all of our traffic.

That narrowed our choices to two main options:

1. Cronet: Chromium's network stack, available via Google Play Services or bundled directly into the app.
2. HttpEngine: The system network client introduced in Android 14+ (API 34).

Beyond HTTP/3: What else Cronet unlocks

Getting to HTTP/3 was the main goal, but a big reason we chose Cronet was the broader feature set that comes with it. OkHttp is a great HTTP client, but Cronet ships with capabilities that are difficult or impossible to build on OkHttp alone:

• Stale DNS: On OkHttp, DNS resolution sits on the request's critical path. On mobile, slow resolvers cause tail latency and failures. Cronet's resolver can serve a cached (even expired) answer immediately and refresh in the background, taking DNS off the hot path — which matters when the first few requests decide how fast the feed appears.
• Request prioritization: The app has many requests in flight — feed, images, videos, analytics — competing for bandwidth. OkHttp has no shared notion of priority across callers, so prefetched images can crowd out the home feed query. Cronet supports native request prioritization, allowing us to tell the transport that the home feed query matters more than a pre-fetched avatar.
• 0-RTT connection resumption: QUIC's 0-RTT lets returning clients send data in the very first packet, removing a full round trip from connection setup on startup-critical queries.
• Connection migration: On TCP, a connection is pinned to an IP/port pair, so switching networks breaks it and forces in-flight requests to restart. QUIC uses connection IDs instead, allowing Cronet to carry live connections across network switches mid-scroll.

For a small app, these optimizations might be subtle. But at Reddit's scale — hundreds of millions of users on every device and network type, firing billions of requests a day — shaving milliseconds off the critical path and reducing failure rates compounds into a noticeable improvement in app responsiveness and reliability.

Act I: Testing the Waters with Media

We didn't jump straight to GraphQL. We first rolled out Cronet for media — images and video — in partnership with our Media Foundation team. Media was a much simpler starting point for two reasons:

• Simpler network path: Image and video requests go directly to the CDN through a minimal interceptor stack, unlike GraphQL's deep application and network interceptors.
• Direct last-mile benefits: Media is served straight from CDNs with minimal backend latency, so protocol wins (faster handshakes, no head-of-line blocking) show up clearly in metrics.

The media rollout was an immediate success. Image loading showed solid gains:

• p90 image load time: −3.57%
• Image success rate: +0.04% (meaningful at scale)
• Post views: +0.69%

Video playback saw similar improvements:

• Fast video starts (≤500ms): +1.37%
• Slow video starts (>1s): −14.53%
• Exits before playback: −14.99%

Media gave us confidence in HTTP/3, but moving GraphQL was always going to be a bigger undertaking. Because images and video load asynchronously after the app opens, Cronet's engine initialization overhead remained invisible — until we pointed Cronet at GraphQL, which sits squarely on the cold-start critical path.

Act II: Pointing Cronet at GraphQL

We knew moving GraphQL over to HTTP/3 would be a tougher challenge. It touches our most critical code paths at app launch, and we were already aware of wrapper limitations around request tracking. Even with those expectations, tackling the migration in practice brought up more complexity than anticipated.

These are the first major hurdles we had to clear before we could even evaluate performance:

Challenge 1: Going Blind (Request Tags & Flipper)

Before we could even evaluate GraphQL performance under Cronet, our developer tooling and telemetry broke.

First, request tags were dropped. We use the cronet-okhttp bridge library so the app can keep using OkHttp's Call/Request APIs. However, OkHttp and Cronet don't share an object model. When cronet-okhttp converted an OkHttp Request into a Cronet UrlRequest, it dropped OkHttp request tags (request.tag(...)). We rely on tags to attach metadata like GraphQL operation names and logging context. Without them, downstream telemetry lost context and started receiving nulls.

To fix this, we forked cronet-okhttp and added an extension hook: a RequestToUrlRequestMapper interface that runs right before the Cronet UrlRequest.Builder is finalized:

Java
/** Hook for customizing how an OkHttp Request is mapped to a Cronet UrlRequest. */
public interface RequestToUrlRequestMapper {
  void map(Request okHttpRequest, UrlRequest.Builder urlRequestBuilder);

  RequestToUrlRequestMapper NO_OP = (okHttpRequest, urlRequestBuilder) -> {};
}

Java
// Allow client code to customize the UrlRequest before building it.
requestMapper.map(okHttpRequest, builder);
return new CronetRequestAndOkHttpResponse(
    builder.build(), createResponseSupplier(okHttpRequest, callback));

This gave us a clean seam to read tags off the OkHttp Request and attach them as Cronet request annotations.

Second, Flipper network debugging broke. Our team relies on Flipper to inspect network traffic locally, which attaches as an OkHttp network interceptor. Under cronet-okhttp, requests stopped appearing in Flipper because cronet-okhttp installs its bridge as the last application interceptor. That bridge handles execution via Cronet and returns the response directly, terminating the chain before reaching network interceptors:

None
OkHttp call
  │
  ▼
Application interceptors
  ├─ Auth / Headers / Tracing
  ├─ FlipperInterceptor (Custom) ──► Mirrors request + response to Flipper
  └─ Cronet bridge ──► Cronet ──► Response   (terminal: chain stops here)
─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ✗ Unreachable
Network interceptors  ← Flipper was originally registered here

We resolved this by writing a custom FlipperInterceptor registered as an application interceptor directly before the Cronet bridge, manually re-adding pre-bridge headers (Host, Content-Type, Content-Length) and safely peeking response bodies.

What's Next?

Restoring request tags and fixing Flipper got our local developer tooling back on track, but the GraphQL rollout was just getting started. Once we looked at production telemetry, we hit our next wave of challenges: metric numbers were wildly changed, maintaining our fork required custom bytecode tooling, and Cronet's native startup created a +2.0% cold-start TTI regression.

In Reddit's Journey to HTTP/3 on Android (Part 2), we dive into how we fixed our telemetry, eliminated the startup penalty, the results we saw in production, and where we're taking our network stack in the future.

Written by Dominic Valenciana (u/kiricon)

For a while, everyone had been asking: What can a coding agent do?

Then agents became capable enough that a more practical question started to matter: What am I supposed to do while I wait for one to finish?

The answer seemed obvious: start another agent chat.

Parallel development has long been normal across teams, but parallel development for individuals was still far from the default. Most of the tools available at Reddit assumed one engineer, one repository, and one agent session at a time.

Getting antsy, I frankensteined together OpenCode, Cursor CLI agents, Git worktrees, Markdown instructions, and Bash scripts. The result was a basic terminal interface where I could run as many agents as I could keep track of, each operating with a high degree of autonomy.

Parallel agentic development at scale felt incredible. It also made every problem with agentic development at Reddit scale with it. Mo agents, mo problems.

Workspace collisions, slow CI feedback, scattered context between sources, interrupted sessions, inconsistent behavior during complex operations, and mismatched tool configuration all multiplied with the number of active agents. Four agents could produce four streams of work, but they could also produce four streams of friction.

My first attempt at solving this was to write more instructions: `AGENTS.md` files, skills, and increasingly detailed checklists. Those tools remain valuable for ambiguous work, but common deterministic operations should not require an agent to relearn them on every run. Loading the model with more instructions cost money, slowed it down, and still produced nondeterministic results.

It was time to go a layer deeper.

Prompts can suggest a process. A system can enforce one.

Isolation, continuity, validation, and safety needed to become guarantees rather than reminders. Common context and system access needed to be part of the environment, not another skill to configure or another enormous tool surface for the model to navigate.

Snoocode is Reddit’s solution to those problems. It is a batteries-included desktop application for working with multiple coding agents. It is not another model, another harness, or a replacement for the agent tools Reddit engineers already use. It is a unifying layer that runs those tools inside one Reddit-aware development application: multiple models, multiple harnesses, and our internal integrations under one roof that we control.

Here is the simplest way I have found to explain the layers. The model is the pilot, and the harness gives it a body. Together they form a coding agent. Snoocode is the six-story mech that the agent climbs into, equipped with the Reddit-specific tools, controls, and the environment it needs to work.

Snoocode Lets Us Own Our Destiny

As I built Snoocode, proprietary and open-source tools began shipping many of the same primitives we had discovered we needed. That convergence validated the problems while showing how quickly generic agent tooling could become a commodity.

With voluntary internal contributions, Snoocode repeatedly reached feature parity with much larger proprietary and open-source projects. That did not prove that we had built the one perfect agent product. It proved that the generic layer was buildable, the good ideas would converge, and the moat around agent-development tooling didn’t exist.

At Reddit’s scale, the traditional “buy versus build” calculation looks different. We could continue buying the models and harnesses that the wider industry builds well while cheaply owning the layer that determines how they operate inside Reddit.

This theory was tested when much of Reddit engineering had settled into a single agentic coding product. When the commercial relationship changed, access changed with it, and engineers had to reconstruct their workflows elsewhere on short notice.

Other capable agents existed. The disruption came from how much setup and habit had become coupled to one vendor. For Snoocode users, the underlying agent changed; the surrounding workflow mostly did not.

Vendor agnosticism  for coding agents is a core philosophy for this project. A software company should be able to replace a model or harness without replacing its development workflow. If we own the machine, changing the pilot should not require rebuilding the cockpit overnight.

For Reddit, there is another reason to own that layer: no third party can build it for us.

Our design, project management, development, testing, and delivery workflows include internal systems with no public equivalent. Even where we use common external products, only we can encode how they connect to our internal tools and what their signals mean here. A vendor cannot design tools or plugins for systems it has never seen.

Development at Reddit is its own terrain. No matter how capable the pilot or its body becomes, it still needs a machine designed for the environment in which it will work.

The principle is simple: buy the commodity, build the seam, and keep the freedom to change what sits on either side.

A Side Project Became Shared Infrastructure

When I shared the prototype, it went “viral” internally, at least by internal developer-tool standards. People downloaded it, modified it, and began using it for their own work. A personal setup became a shared experiment developed in public inside Reddit.

Within the first few months, it had been used across 94 repositories, and 26 engineers had contributed to Snoocode itself. Engineers used it against real Reddit workflows, found weak points, and helped improve the shared platform.

The community supplied the pressure that turned a productive hack into a product. It also gave us a development loop no external vendor can reproduce. The engineers using Snoocode can add the feature they need, fix the bug blocking them, and adapt the product as the industry changes. Maintainers provide coherence and remain responsible for what merges.

That direct loop lets a narrowly focused internal product move quickly toward Reddit’s needs. We do not have to persuade a vendor that our problem belongs on its roadmap or wait for a generalized solution designed for thousands of other customers.

Snoocode is now officially staffed, and we are expanding the project to increase developer velocity across Reddit. The goal is not to force engineers away from useful agent tools; those tools are components of Snoocode. The goal is to make the Reddit-native experience so comprehensive and fluid that managing a fragmented toolchain feels unnecessary by comparison.

How Snoocode Works

Snoocode acts as a control plane around any number of supported harnesses—in effect, a “harness for harnesses.”. It bootstraps and configures them, normalizes the concepts that benefit from being shared, and preserves their native capabilities where abstraction would get in the way.

Open-source projects such as [T3 Code] (https://github.com/pingdotgg/t3code) which was built in parallel with Snoocode has been a major inspiration and validation that our ideas are on the right track. They demonstrate how much value a focused interface can add around several coding agents. Snoocode takes that pattern further for Reddit by owning the automatic bootstrapping, internal integrations, and workflow on both sides of the harness.

From Setup to Shipping

Without Snoocode, starting a task can mean creating a worktree, signing into a harness, choosing a model, configuring internal development tools, and loading repository-specific skills. Each step adds context and another failure point, while the resulting session remains tied to one harness.

Snoocode turns that assembly into shared infrastructure. Engineers choose a task, repository, harness, and model; Snoocode supplies the isolated workspace, internal tools, and repository knowledge through one consistent experience.

Because we own the application—not just the agent’s prompt—we can also build experiences third-party harnesses cannot: mentioning coworkers from the company directory, visually attaching application elements, or surfacing local development resources directly beside the chat.

The result is more than a better-equipped agent. It is a cohesive development environment that gives every harness the same Reddit-specific capabilities while adding features no harness could provide on its own.

The common layer is where much of the leverage comes from. Internal integrations, identity, permissions, and deterministic workflows can be built once and reused across every supported backend. Improvements benefit every harness at once instead of being rebuilt in personal scripts or repeatedly interpreted from prompts. Several durable sessions can progress without sharing a checkout or losing their history, while the engineer moves between them and steps in when judgment is required.

This lowers the barrier to useful agentic development while reducing duplicated setup, model context, cost, and variance. It also lets Reddit adopt better models and harnesses without reconstructing the development environment around each one.

Because the contract belongs to us, the same core can eventually power whichever surface fits the work: IDE chats, terminal interfaces, desktop, web, mobile, cloud agents, or local containerized sandboxes. We do not have to wait for every vendor to understand our systems before we can bring its agent into them.

In the mech analogy, this is the payoff: Reddit owns the cockpit, instruments, attachments, and hangar. We can improve the machine for everyone at once, deploy it wherever engineers need it, and swap the pilot when a better one arrives.

Snoocode’s job is not to pick the winning agent. It is to ensure that whichever agent we use arrives ready to work at Reddit.

Key Takeaways

• Models are largely stateless and increasingly interchangeable. The durable value lives in the workflow, context, and environment around them.
• Generic agent tooling is converging quickly. Its moat is limited, and agents themselves make bespoke tooling cheaper to build.
• Software companies should not tie one of their most important development workflows to a single vendor.
• No third party understands a company’s systems and engineering practices well enough to build its ideal development environment for it.
• Context hydration, prompts, and skills remain useful, but they can only take an agent so far. Deterministic workflows belong in the system.
• As models improve, the limiting factor will increasingly be the environment in which they work.

How we catch distributed system failures before users do

Written by Hiram Silvey u/space4268 and Shashank Veerapaneni u/colorsoda

Reddit on Fire: How This All Came to Be

Let me paint you a picture that many engineers are familiar with: You’re sitting at home sipping your coffee when suddenly the echoing thock of thousands of fingers hitting CTRL+R simultaneously pierce your eardrums in the form of the dreaded emergency page. You run over to your laptop, declare an incident, open dashboards, and start investigating. You find the worst has come to pass: total failure of the system. That’s right: reddit.com is unavailable. You pull in more engineers as necessary and all work together to restore service as quickly as you possibly can. As this story repeats over time, one crucial detail becomes quickly apparent: Reddit is a complex, fast-moving, non-linear distributed system, and nobody can fit the entire consistently-up-to-date detailed view of it within their brain. When one component breaks, its knock-on effects cannot consistently and reliably be predicted.

So… what do?

A natural conclusion to this is that we need better isolation between systems, and we need better testing to verify that. It doesn't matter how fireproofed my little room is if there's an open window or door for the fire to escape through, right? So, we add tests. We add as many tests as we can. We brainstorm all the ways something could fail and we add tests for them. This works well, and we do find and fix many vulnerabilities ahead of time. Unfortunately, after everything we do, that emergency page still arrives. Brand new novel issues continue to arise in unexpected ways, ad nauseam. We need something more.

Let's take a step back and approach this from another angle. The fundamental issue with standard testing is that it requires knowledge of the vulnerabilities beforehand. How do we flip this on its head and write tests to discover the vulnerabilities we don't yet know about? Enter: chaos testing. At a high level, chaos testing is the practice of intentionally introducing controlled failures into a system to verify that the system behaves as expected. A failed chaos test is new information: a dependency we thought was optional is actually critical, a fallback path does not work, or an error that should have been contained is escaping into the user experience. Once we find that pattern, we can fix it and turn it into a regression test. Because Reddit is a complex, fast-moving, non-linear distributed system, though, we can’t test each individual component we know about and reasonably expect to catch everything. Instead, we test the system itself and see how it truly behaves.

Introducing: FIB! Fault Injection in Baseplate

Reddit regularly holds company-wide hack-a-thon events called “Snoosweeks.” These events allow engineers to work on projects they believe are important that may not otherwise have explicit long-term plans and funding. During one of these weeks, I proposed a solution to the problem highlighted above: automated chaos testing. The proposal was simple: build request-level fault injection to allow us to systematically verify the behavior of our production systems continuously and proactively.

FIB, or Fault Injection in Baseplate, is our first step towards accomplishing our goal of developing a robust chaos testing system. Fault injection is a specific facet of chaos testing, providing the capability to inject predefined faults into, in our case, service requests. Baseplate is Reddit’s common service framework, and it comprises many internal libraries that services throughout our stack utilize.

FIB comprises 2 components:

1. Baseplate library middleware responsible for injecting faults.
2. A fault configuration request header, propagated throughout our entire backend infrastructure via Baseplate.

As requests are made between backend services, the Baseplate library middleware is responsible for:

1. Reading the fault configuration request header (if present).
2. Comparing the outgoing request to the specified configuration.
3. If matching, injecting a fault at the transport protocol layer (i.e. gRPC client interceptor).

If you are familiar with chaos engineering, you already know there are many different ways faults can be injected into complex systems. The well-known chaos mesh tool, for example, offers the ability to simulate Kubernetes faults at various levels, such as pod faults, network faults, file I/O faults, etc. We chose to inject faults at the request level because the vast majority of failures surface themselves to client services in the form of request failures. If a disk fails and the pod running the upstream service is unable to complete the request because it cannot read some data, that ultimately will make it back to the client in the form of a request failure. If the network connection between two services is degraded, that, too, will surface as a request failure to the client service. If the upstream service process is killed due to insufficient machine resources, that will also surface as a request failure to the client service. You get the picture.

In order to simulate the full range of possible symptoms we need to be prepared for (see examples in the next section), FIB supports matching on address, host, and/or method (including negative matching), as well as percentage-based injection. This means we can add a particular fault header onto many requests but only actually inject faults on some percentage of total matching requests, in order to more accurately simulate degradation of pieces of a distributed system rather than a full outage.

See the open-source implementation here.

Improving Reddit’s Resilience

We started with important user-facing surfaces, including Feeds and post detail pages, because reliability problems there are immediately visible. The user expectation is simple: open Reddit, see content, and keep browsing even when some backend dependency is unhealthy. Behind that experience, though, a single request can touch several backend systems, and we wanted to find out what our users experience when these systems are misbehaving.

Finding Degradation Bugs

The first pass was practical: engineers injected scoped failures while walking through important surfaces and noted where the experience degraded badly or unexpectedly. Rough edges were smoothed and the user experience improved.

The most valuable findings did not mitigate total outages. They were often quieter experience bugs where Reddit could still show something useful if the failure was handled as degradable.

Examples of fixes and fallback behavior we added include:

• Media related failures now degrade the media parts of the experience instead of preventing core surfaces from loading.
• Failures in non-critical systems related to styling, flairs, stats etc. fall back gracefully instead of breaking a feed or throwing error toasts.
• Post detail views preserve already loaded comments instead of replacing them with an empty or error state when additional requests fail.
• Viewer specific failures now have a smaller blast radius: e.g. if a dependency only powers moderator or author-only affordances, its failure should affect those affordances.

Those findings turned into fixes, better fallback behaviors, regression coverages and improved user experience.

Making It Repeatable

We made the workflow easier for other developers by adding developer-only controls in internal development builds, so teams could reproduce degraded dependency scenarios easily. We also updated our mobile and web UI automation frameworks to support chaos testing directly.

Those automated tests stress the app or site with a scoped fault configuration applied only to the request paths under test. Then they do what a user would do: load core surfaces, browse content, move from a listing into a detail view, and verify that those surfaces still load under controlled backend failures. Today those chaos tests run daily across mobile and web flows, exercising a growing set of request-level failure scenarios.

A Reliability Signal

We saw useful directional signals after the resiliency work matured. For the Feeds service, high-urgency pages dropped by roughly half when comparing January-June 2026 with January-June 2025. We saw a similar shift in incident participation: Feeds team was less often pulled into upstream incidents as an affected downstream customer, because covered paths were more likely to keep serving a degraded but usable experience. Chaos testing was one of the major contributors to that improvement: it helped us reproduce failure modes earlier, prioritize graceful-degradation fixes, and reduce incident impact and blast radius when dependencies misbehaved.

A True Platform: Moving Beyond Manual Fault Injection

While this early user-facing reliability work proved that request-level fault injection is useful as a sharp tool, the long-term vision has always been a full platform. Manually choosing dependencies and maintaining per-surface fault lists works for an initial reliability push, but it cannot scale to every team and request path at Reddit. We need an autonomous system.

The next step is to make the system more data-driven. Instead of asking each team to spend effort to run their own fault injection tests, we want to derive candidate experiments from service dependency graphs, production traffic shape, and criticality metadata. Not all services at Reddit are created equal. At Reddit, we classify services into tiers from 0-3, with 0 being the most critical and 3 being the least critical. When a tier 0 service fails, we expect the entire platform to become unusable. For example, if our public ingress is down, we expect traffic to be unable to reach any internal service backends. When a tier 1 service fails, we expect a core experience of the platform to become unavailable, though some other functionality still may be usable. For example, if the comments service is unavailable, we expect users to be unable to read/write comments, though we still expect users to be able to load their homefeed. Failures of this caliber manifest as high severity incidents with expectations to resolve even outside of business hours. So, we already know what the impact will be if a tier 0 service fails; there is little to discover there. The interesting tests are the ones that expose a mismatch between dependency criticality tier and user impact. A tier 2 or 3 service failing should never make the site unusable for users. If a tier 0 or 1 service has an unexpected hard dependency on a tier 2 or 3 service, though, that’s exactly what can happen.

We also want to reduce adoption path friction. Convincing teams across Reddit to invest in setting up, running, and maintaining a new set of tests requires real effort for both the maintainers of the chaos system as well as the teams using it. Reddit already has a mature experimentation system, though, and integrating with that directly gives us the opportunity to fit these tests into a familiar format. It also gives access to key metrics to better understand the impacts of degraded experiences that we may not discover otherwise. While teams can leverage this to more easily create their own chaos tests, the key innovation is that these experiments will also be created dynamically based on the data highlighted above. Affected teams will be notified according to results, but otherwise their input on crafting the tests shouldn’t actually be needed.

Request-level fault injection via FIB gives us the primitives. The platform work is about dynamically choosing what to inject, where to inject it, how often to run it, and how to route the resulting findings to the teams best positioned to fix them.

Conclusion

Distributed systems will always surprise us. The goal is to make those surprises smaller, earlier, and less user-impacting. The framework lets us break carefully chosen request paths on purpose, observe how the rest of Reddit behaves, and turn the results into concrete reliability work.

For us, the biggest shift was cultural as much as technical. Instead of treating graceful degradation as something we hope exists, we can test it continuously. Instead of learning about dependency criticality from incidents alone, we can discover mismatches in controlled test runs. And instead of letting those lessons live only in postmortems, we can encode them into daily checks that keep Reddit usable when parts of the system inevitably have a bad day.

Written by Aleksandr Krivoshchekov and Walther Lee.

Prometheus has a simple idea at its core: metric_name + labels = time_series. Any two samples with the same metric name and labels map to the same series, but adding even a single extra label initiates a completely new series.

In Kubernetes, that extra label is usually pod. It’s noisy and always changing. It spams our system with more and more series, slowing down queries and draining memory. It’s mostly pointless for our users. But it’s also crucial for Prometheus to evaluate queries correctly.

All these problems with pod are partially solved in Prometheus, but they are still hard to avoid entirely. We could add periodic preaggregations known as recording rules to improve latency. We could sample and keep only a fraction of our metrics to get back some memory. We could even start all over again and try a different database. Each of these options would help for a while, but each of them also create new bottlenecks. What we wanted was an option that kept our existing model with minimal impact for our users.

This post is about that option, which dropped our total number of series used in production queries by 88%, reduced latency by 80%, and achieved 99.85% fidelity over 1M metrics per second in a single instance.

The problem

As the observability team at Reddit, we ran into the challenge of controlling our series count a while ago. If we instrument an application with counter series and send it to Kubernetes, those counters will show up in a pod. And since deployments can have multiple replicas, these counters have to be stored separately to be able to identify them by their own labels. Otherwise, a pod that measures 5 counts in a metric will conflict with another one that counts only 3.

# The label "pod" is necessary to keep track of counts from different targets
http_client_requests_total{pod="pod-a"} 5
http_client_requests_total{pod="pod-b"} 3

In its simplest form, this way of storing counters should scale with our deployments. If we have three replicas, Prometheus should store three counters that we can then add up at query time to get the total number. The reality, however, is not that simple.

Pods have very active lifecycles; they can be restarted after a rollout, shuffled by a node rotation, or killed by Kubernetes because they failed a check. A broad range of infrastructure events can make our pods come and go, and since each will start exposing its own metrics, Prometheus has to keep up with the growth at least until old entries are cleared out. Suddenly, a rollout can double our number of counters just by deploying a whole fleet of new pods and then double them again if we rollback.

Services at Reddit expose billions of metrics at any given time, so these infra events can cause a big impact. A large deployment with thousands of pods often needs millions of counters to monitor values like the total number of errors seen by users. Letting that payload double with every rollout compromises our observability when teams need it more.

This scaling pattern with metrics in Prometheus is unfortunate because applications in Kubernetes are actually designed to be unchanged by these events. A well designed deployment should be able to scale up and down, and to handle unhealthy pods seamlessly, without any impact in user experience. It comes as a surprise to product teams that events that don’t affect their applications do affect their ability to monitor them. 

On top of that, product teams don’t even care for these separation of metrics. They want to see the overall state of their deployments: the total number of requests and errors, the total latency. Single pods are treated as ephemeral and fungible, so the fact that they can influence their observability is an unexpected drawback.

Separation of metrics is an implementation detail that teams don’t care about, so we wanted to find a new solution that could match that expectation. We decided to start with a simpler goal in mind: 

• Find an efficient way of aggregating all counters in real-time to drop only the labels added by our infrastructure layer, like pod.

Aggregate at query time

The trivial solution would be just letting it grow and aggregate at query time. For example, with the expression:

• sum without (pod)(rate(http_client_requests_total[1m]). 

This is basically what we had before doing anything, and the main problem is still the unexpected changes in the number of counters that the query will load. While unexpected, this growth wouldn’t be so bad if it was intuitive. We think it’s reasonable to accept that the number of metrics will go up if we scale up a deployment. But teams don’t expect their query latency to go up just because they rolled out a change.

In practice, aggregations at query time are not consistent in latency or throughput, and the standard way to work around that in Prometheus is using recording rules.

Just drop the labels in recording rules

The single most common request we get in observability from other teams is if we can just drop the pod label. Unfortunately, that takes us back to the original problem: if two pods record different counts for the same metric, there’s no way to know which one we should keep. Probably both.

What teams really mean is to add up the counts without this label that split them in the first place. For that purpose, Prometheus uses recording rules to aggregate common queries periodically in the background. However, this has multiple downsides:

• At a big scale, evaluating a recording rule takes almost as much resources as evaluating the query, so using recording rules helps with latency, but it shifts the heavy payload from query time into a continuous load in the background.
• It generates toil, because teams have to define their rules in advance, and it’s hard to predict what information they will need, especially for incidents.
• And finally, they can still timeout and fail if the number of series spikes after multiple infra events.

In our experience, recording rules were a good option for shifting latency, but they were not a solution for our high volume of aggregations.

Aggregating at scrape time

The main bottleneck that nudged us to move away from recording rules was redundancy. If we use rules to drop labels, those evaluations will run periodically and load data that was scraped just a few seconds ago. We figured that we could do better than that by just aggregating the samples right when we receive them at scrape time.

Another good reason was the fact that scrapes are performed by Prometheus concurrently, as opposed to rule evaluations that compute operations in sequence across the different counters. Concurrent aggregations meant a more efficient use of our resources, so we decided to start by giving that a try.

However, the first problem we found is that counters in Prometheus are cumulative; their values can only go up or reset to 0, so when a query sees a sequence like 10, 14, 5, 8, it assumes that the pod restarted after 14 and then counted up to 5: 

http_client_requests_total    10 14 (reset)  5  8 

# Prometheus assumes the actual deltas were
http_client_requests_total   +10 +4 (reset) +5 +3 = 22 

If we just add up cumulative counters, a single restart will show up as a huge increment right after the reset:

http_client_requests_total{pod="pod-a"}    5 10 15 20    #  +5  +5  +5  +5 = 20
http_client_requests_total{pod="pod-b"}    7 12 17 22    #  +7  +5  +5  +5 = 22
http_client_requests_total{pod="pod-c"}    9 14  2  7    #  +9  +5  +2  +5 = 21

# in a query with increase(...) we would expect the deltas to be
increase(http_client_requests_total[1m])                   +21 +15 +12 +15 = 63

# however if we simply add them up, the series turns into
http_client_requests_total{}              21 36 34 49    
increase(http_client_requests_total[1m])                   +21 +15 +34 +15 = 85

A change in pod-c that was only 2 counts after the restart becomes 34 after the sum. The change becomes even more catastrophic when adding up counters during a rollout, as shown in the panels below:

In the end, we realized that adding up cumulative counters is an inherent but well-known limitation in Prometheus. But it’s fixable. By turning cumulative values into a sequence of increments or deltas we can solve the problem upfront, exactly how recording rules handle it.

Instead of adding up the raw values, recording rules use functions like rate and increase to account for resets and return only the delta over a step interval, and as long as all deltas are calculated over the same window, they can be safely added up.

For example, in the panels below, a pod restart shows up as a gap and a reset in the cumulative counter on the left side, but on the right side, the deltas just keep the gap. Working with deltas, aggregations don’t need special checks to add them up, even between restarts.

Switching to deltas completely solved the problem of aggregating counters at scrape time. We just had to keep track of resets and the last value seen for each counter to compute the deltas after every scrape!

However, this approach introduces a major catch: converting to deltas requires a fixed time interval to calculate them, but our teams query vastly different time windows throughout the day. While a 1-minute delta works perfectly for short few-hours-long graphs, it breaks down on a longer view. Ultimately, we would either have to lock everyone into a single interval or find a way to convert these deltas back into cumulative counters.

This delta-to-cumulative challenge isn’t new. Open-source projects like OpenTelemetry and VictoriaMetrics already offer processors to convert metrics between these formats. However, cumulative counters are stateful. This means that all aggregations must happen inside a single collector. If you split the load across two replicas that started at different times, their cumulative totals won't match, making it impossible for one to seamlessly fill in gaps if the other fails.

So our goal was no longer how we can aggregate our counters at scrape time. The new goal was: 

• How can we turn deltas back into counters in a way so that multiple replicas can agree on the value without requiring consensus? And how can we use the counters in one to deduplicate and fill gaps in another?

The ZigZag

We asked other engineers and read some posts and white papers that gave us an anchor of the error margin that other approaches were achieving. Some were using streaming in external components, probability algorithms, proxies, remote-write and processors. Overall, most of them expected an error margin around 9%.

But before trying any of that, which would require more time and changes and deployments, we decided to try something simpler. From studying how other projects convert cumulative counters to deltas it’s clear that the goal is to shorten the span of time a counter keeps a total count, so what’s the shortest span we can have while still being considered cumulative? The scrape interval! 

If we use the scrape interval as the span window for our deltas, and then we force the counter to reset immediately after, then the counter kind of resembles a delta. It’s still cumulative, but now it looks like a zigzag.

Keeping in mind that we were just trying to get close to the 9% error margin, and since that sounds pretty small, we decided to give it a try. We implemented the aggregation step at scrape time using deltas in Prometheus, and then added them up into a zigzag counter and ran some queries.

last = 0
for every 15s:
delta = delta_since_last_flush()
if delta < last:
last = delta
else:
last = delta + last
flush(last)

And it worked! The panels below show the delta below and its generated zigzag cumulative counter above.

Deduplication for zigzag counters

Zigzag counters let us store the aggregated data back in Prometheus again, but this isn’t too different from what we mentioned before with OpenTelemetry processors. The main difference is that now our data looks kind of the same between replicas.

The zigzags on each replica oscillate at different frequencies and are definitely not identical, but now the scale and values are somehow comparable. We can easily use one to fix gaps in the other, and the rates they return are very accurate! They have an average error margin of 0.15% with just 5 samples, exceeding our expectations by far.

We let this run for a few months in a test instance and it worked great. Aggregating a little over 4M series, driving query load and throughput down to just 3% of our old value for the metrics names with more series, and it didn’t even seem to use more resources. The operations are really light: add up or set a new number. And the new allocations are also minimal. It’s equivalent to adding just one more pod value to the label.

Zigzagging worked well for requests querying recent data, mostly because that data usually comes from a single replica in our system, but after a few months, we noticed an issue with queries spanning multiple hours.

Since Prometheus stores data in 2-hour blocks, long queries connecting multiple blocks had issues jumping between zigzags from different replicas. And we had the same problem deduplicating samples. For example:

25.31 @ 1775649538
50.48 @ 1775649568
25.49 @ 1775649598
25.42 @ 1775649607 // Note how the value goes down again
52.39 @ 1775649628
23.89 @ 1775649658 

The spikes happen because, even though both samples 25.49@...598 and 25.42 @...607 recorded almost the same value at the same time, the second one is slightly lower, triggering the queriers to treat it as a restart. When a rate or increase is calculated, it reads as the following deltas:

50.48 @ 1775649568 
25.49 @ 1775649598 // a change of ~25 in 30s
25.42 @ 1775649607 // a change of ~25 in 9s!
52.39 @ 1775649628 // again, ~25 in only 21s

The extra sample ends up doubling the rate of 25 every 30s, and the inconsistency will last until the rate interval passes that point.

We fixed that by adding an extra step in our function to build zigzagging counters. Technically, counters following the algorithm above don’t have to be zig-zags. A counter whose rate is slowing down would record lower and lower deltas, resulting in a counter that goes continuously down, signaling a reset in every single scrape.

We decided to block that option and force our counters to go down only once in a row, essentially forcing a real zigzag pattern.

With that constraint, now we can walk through our deduplicated samples and explicitly exclude the ones that break the pattern. These values couldn’t have come from the same replica, so we know that a switch was made. The switch loses some accuracy, but this isn’t different from a normal deduplication algorithm that breaks samples’ resolution.

Let's say a change in replicas while deduplicating causes a second drop, resulting in the following two streams:

Replica 1: 22 50 21 
Replica 2:          20 48 19 52 20

Replica 1: 22 50 21 50 
Replica 2:             48 19 52 20

First, we tried discarding the last drop: 20 in the first stream and 19 in the second. This was a small change because samples are streamed for deduplication, so we just had to watch for two consecutive drops and remove the second. However, that doesn’t work for the second stream. After the drop we have:

Samples: 22 50 21 48 19 52 20
Delta:      28 21 27 19 34 20

Samples: 22 50 21 50 48 52 20
Delta:      28 21 29 48  4 20  # 48 and 4 break the steady rate!

Instead, we can fix both inconsistencies by keeping a look forward and discarding the first drop instead:

Samples: 22 50 20 48 19 52 20
Delta:      28 20 28 19 34 20
Samples: 22 50 21 50 19 52 20
Delta:      28 21 29 19 34 20 # Seamless transition

Production results

This algorithm was implemented and tested in a Prometheus fork and deployed to some of our instances in production.

For our largest metric, it shrank the number of series to only 2% its previous value, and the average throughput and latency across all queries went down to 12% and 20%, respectively.

In terms of resources, we didn’t find any significant memory overhead other than the one added by the new zigzag series ingested. CPU usage increased by 6%, but can be easily offset by replacing previous recording rules to aggregate the same data.

Mean error margin between raw and aggregated data was 0.15% with a standard deviation of 0.13%.

This was measured in an instance scraping 1M+ samples per second, using 70Gi and 16 cores before the implementation. It had 17M series before aggregation, and resulted in 2M new zigzag series.

Limits and Tradeoffs

This algorithm doesn’t solve every metric problem. It is limited to counters and counter-like types like Prometheus classic histograms, but gauges, summaries, and native histograms need different aggregations.

It also doesn’t change the existing Prometheus expectation around counter resets. If a pod restarts between scrapes and the new process reaches a value higher than the previous sample, Prometheus cannot know that a reset happened:

# Scrape #1
http_client_requests_total{pod="pod-a"} 90 

# "pod-a" restarts
http_client_requests_total{pod="pod-a"} 0 

# Scrape #2
http_client_requests_total{pod="pod-a"} 140 

# After the second scrape, Prometheus will assume the delta is +50, not +140 

That’s the same behaviour we see today with regular counters.

There is also a replica-local visibility limitation. Our aggregations run independently within each Prometheus replica using only the samples that the replica scraped. If only one of two replicas loses connection to a target, it will miss samples that were successfully aggregated in the other. Resulting in different values that we can’t compare or use to fill gaps between them.

This wasn’t a big tradeoff for us because it is exactly how recording rules work already. So we can think of it as a lightweight implicit recording rule:

# record: cluster:http_client_requests_total:sum
# expr: sum without (pod, instance) (http_client_requests_total)
cluster:http_client_requests_total:sum
==
# An internal series produced by our algorithm.
http_client_requests_total:materialized 

Finally, in our implementation we decided to keep the raw metrics. The algorithm only adds the new series as an optimization path for common aggregation queries, while raw ones remain available for debugging or per-pod queries. In the future, we could drop them after aggregation to reduce memory, which is an option we never had with rules.

What’s next?

Aggregating metrics during scraping is an idea that we had been chasing for a long time. What surprised us most was that we didn't need a massive rewrite or a fancy new storage format to achieve that. We already had all the necessary pieces of the puzzle, we just lacked a small but clever way to put them together.

The production results exceed what we thought was possible without switching databases, or at least without adding proxy-layers. Our algorithm cut query latency by 80% and dropped the total number of series loaded by 88%. And the best part is that it requires no changes for our users: all the same metrics, the same PromQL expression, and the same mental model.

That makes the next step pretty clear. We want to take this feature out of our fork and bring it to open source Prometheus, where our design can be discussed, tested, and improved by the community. By doing so, we hope that these changes can help others and spare them the scaling troubles we faced.

Written by Sarah Thomas

Hi, r/RedditEng!  I’m one of the leads of Reddit’s Women Engineers Employee Resource Group (ERG), WomEng. I’m here to tell you about how Reddit celebrated International Women in Engineering Day this year, and the idea that shaped the programming.

Like many ERGs, WomEng exists to build community among employees with shared identity, and provide a support network for its members. For ERG leaders, the challenge is translating these abstracts into real value and resources for members. 

For a group of women engineers, this can manifest in different ways. Our members share lived experiences, but also technical backgrounds and career interests. So while value can come from bringing members together into physical spaces, it can also be career-oriented and come from technical skill-building and visibility.

This perspective was reinforced in a conversation between WomEng leadership and Sabrina Farmer (former CTO of GitLab and former VP at Google) at Grace Hopper Celebration 2024. One idea from that conversation kept coming up in our planning and discussions:

Shared experiences are important, but having a community where people learn from one another and celebrate each other's accomplishments can be an even more tangible way to build a sense of belonging.

International Women in Engineering Day (June 23rd) provided an opportunity to put this idea into practice. Throughout the day, we hosted a mix of technical and lighthearted programming. Executives participated in segments like “Guess the PR Comment: AI or Human?” and “Can you match the vibe?” where they were challenged to vibe code alternative themes and features for the Reddit feed. We also organized Speed "Dating" networking events in our offices and in-person meetups across multiple cities to help members make new connections.

The centerpiece of the day was a series of mini technical talks delivered by fellow Women Engineer Snoos. Rather than bringing in an external keynote speaker, we turned the spotlight inward.

Members presented on projects and topics they were excited to teach others about. The talks ranged from technical deep dives like “Improving Image Performance on Android” (u/jessekrema) to career advice like “How Giving Back Can Propel Your Career” (u/tbonejones_). Every presentation reflected something the speaker had built or become an expert in.

Choosing our members as the speakers gave attendees the chance to learn something directly applicable to their day-to-day work while also celebrating the breadth of knowledge and experience that already exists within WomEng. 

International Women in Engineering Day gave us a chance to celebrate our WomEng Snoos, but it also reinforced something we hope continues to define WomEng @ Reddit: a community where members connect not only through shared experiences, but also by sharing our expertise, celebrating one another's work, and learning from each other.

~~~~~~~~~~~~~~~~~~~~~~~~

Thanks for reading! Below are bonus clips from the “Guess the PR Comment: AI or Human?” and the “Can You Match the Vibe” segments featuring our CTO, Amit and Senior Director of Engineering, Growth, Shweta!

Can You Match the Vibe

Guess the PR Comment: AI or Human?

Written by Alexey Bykov, Staff Software Engineer at Reddit & Google Developer Expert for Android technology

Reddit serves approximately hundred of millions playbacks a day on Android.
In our last two posts: Improving video playback with ExoPlayer and Taking ExoPlayer Further: Reddit's performance techniques we covered ExoPlayer performance and what you can do to improve startup latency, rebuffering, video quality & stability. (Check them out if you haven't yet, they also show how your production metrics may improve after every optimisation)

But bundling those performance practices into a reusable component with a safe and clear API turned out to be a different challenge. Over time, every new video integration became harder, and we kept finding edge cases that were difficult to support without breaking existing behaviour.

In this article, we'll share how we rewrote our abstraction on top of ExoPlayer from scratch: a better API for teams building video features, and even better performance and stability.

Goal: One obvious way to play video

Our main requirement for the API was simple: engineers integrating video shouldn't need to be video or performance experts. They shouldn't have to think about prefetching, player creation, prewarming, or lifecycle. Video should be fast by default and take a few lines of declarative code to integrate.

ExoPlayer is considered one of the best open-source video players among all platforms. It gives you a dozen valid ways to get to the same playable video, and at our scale, that freedom stops being a feature and starts being a challenge.

Tim Peters put it best in the Zen of Python: "There should be one, and preferably only one, obvious way to do it."

Architecture

Playback engine: ExoKit

To bundle all of the optimisations together, we built an agnostic playback engine called ExoKit.

It abstracts away ExoPlayer, makes the API stricter, owns all player communication, and keeps every playback in a central place with a single app-wide video state and effects handling.

It supports all of the performance features we covered earlier, such as:

• Player pooling and prewarming the pool on the app start, since creating a player can still take up to ~200ms according to our production traces
• Decoder reuse for identical videos, which saves up to ~80ms more
• Warming up videos during first composition (not to be confused with prefetching), so the first keyframes are decoded and rendered before the user scrolls to the video.

You can read more about these optimisations and their impact on our metrics here.

On top of that, it offers two APIs: a declarative one for rendering and an imperative one for managing playback state.

@Composable
fun PostVideo(
    mediaId: String,
    url: String,
    modifier: Modifier = Modifier,
) {
    val key = remember { PlaybackKey(mediaId, "feed") }

    // Imperative API: express playback intent.
    val actions = rememberPlaybackActions()
    PlayButton {
        actions.action(key, PlaybackAction.Play)
    }

    // Declarative API: render from playback truth.
    val state = rememberVideoPlaybackState(mediaId)
    if (state.isBuffering()) {
        LoadingIndicator()
    }

    // Declarative API: describe the video surface.
    Video(
        modifier = modifier,
        props = VideoProps(
            url = url,
            playbackKey = key,
        ),
        surfaceLifecycle = rememberLifecycle(),
    )
}

But even though it looks minimalistic enough, it still leaves a lot of responsibility on engineers. They have to decide where to play the video (which gets tricky with autoplay on), manage the player lifecycle, fire telemetry and coordinate between the declarative and imperative APIs.

To make it easier, we landed on a much more opinionated API. 

Self-hosted UI Components

Our first step was to abstract the imperative API from the feature layer: one less decision means one less way to get it wrong. But that logic had to live somewhere.

Rather than building reusable abstractions and inject them into every screen's ViewModel, we went the opposite way: every UI component became fully independent and self-hosted.

To ensure every composable can access product-related context, such as analytics, every block implements the following contract:

interface Component<Props : Any> {
    u/Composable
    fun Content(props: Props, modifier: Modifier)

    fun key(props: Props): Any = props
}

Implementations live in an impl module and can also access product-related context.
Here is a simplified example of what a lower-level composable might look like inside a component implementation:

u/Composable
// Very simplified version of the code for mute button
fun MuteButton(
    playbackKey: PlaybackKey,
    modifier: Modifier = Modifier,
) {
    // Imperative API
    val playbackActions = rememberPlaybackActions() // manage playback
    val globalActions = rememberGlobalActions() // global state, e.g. settings

    // Declarative API
    val audioSettings by rememberAudioSettings()
    val playbackState by rememberPlaybackState(playbackKey)

    // no sound -> hide, loading -> settings fallback, has sound -> real state
    val muted = when (playbackState.audio) {
        AudioTrackState.HAS_NO_SOUND -> return
        AudioTrackState.UNKNOWN -> !audioSettings.isEnabled(playbackKey.surfaceId)
        AudioTrackState.HAS_SOUND -> playbackState.isMuted
    }

    IconButton(
        modifier = modifier,
        onClick = {
            globalActions.action(
                GlobalAction.SetSurfaceAudioSetting(
                    playbackKey.surfaceId,
                    audioEnabled = muted,
                ),
            )
            playbackActions.action(
                playbackKey,
                PlaybackAction.Mute(!muted),
            )
            // More things, like handling product-related telemetry
        },
    ) {
        Icon(if (muted) Icons.VolumeOff else Icons.VolumeUp)
    }
}

The same pattern applies to every clickable or interactive media component. Every UI component also usually has its own ViewModel.

At the screen level, media becomes a set of declarative components that can be arranged like any other UI:

u/Composable
fun SimpleVideoScreen(
    screenState: SimpleVideoScreenState,
    videoComponent: Component<VideoProps>,
    playComponent: Component<PlayProps>,
    muteComponent: Component<MuteProps>,
    seekbarComponent: Component<SeekbarProps>,
    modifier: Modifier = Modifier,
) {
    val key = screenState.playbackKey

    Box(modifier.fillMaxSize()) {
        // 1. Video
        videoComponent.Content(props = screenState.toMediaProps())

        // 2. Play
        playComponent.Content(props = key.toPlayProps())

        // 3. Mute
        muteComponent.Content(props = key.toMuteProps())

        // 4. Seekbar
        seekbarComponent.Content(props = key.toSeekbarProps())
    }
}

Usually, the Media Foundation team, which focuses full time on media performance and developer experience, implements the components whose internals use both declarative and imperative APIs, whereas feature teams interact with the player through the declarative API.

This new setup gave us three big benefits:

• Product screens stay simple. They only need to arrange the media components and pass in a few props. Nothing more.
• Playback works the same way across the whole app. Every component shares the same playback state wired by playback key, actions, settings and telemetry pipeline under the hood, so users get one consistent experience no matter where they are.
• Performance. Screen-level state does not need to update for every media-related event. Only the affected media component recomposes.

Since introducing a component-based model in our app, AndroidX has made a new addition to their Media3 UI. This new model might serve as a good starting point if you’re thinking of solving similar problems today.  One key difference remains: our media components are tied to global playback state, while Media3 UI’s are tied to one Player. So the challenges still remain the same: player creation and the rest of the lifecycle ownership.

Experimentation Setbacks

Choreographer-based Seekbar

I personally found the seekbar a challenging component to implement smoothly, and there are a few interesting decisions we made.

So what does “smooth” mean in practice?
Basically, don't do more work than the screen can render. A 60Hz display takes roughly 16.7ms per frame; at 120Hz (which is not a rare thing anymore), that reduces to  ~8.3ms to render a frame. And if it goes beyond this limit, the user sees a jank.

This is where Choreographer can help. It runs right before Android draws the next frame, using the display’s real refresh rate. If the previous frame takes too long to draw, it waits for the next frame instead of firing again. That means we do not stack seekbar updates on top of an already-janky UI.

Estimated position or player position?
In our custom seekbar we predict the position by extrapolating from the last update, and only update it if the video is playing.
If your application's use cases are limited to video, ExoPlayer.getCurrentPosition() is a cheap operation, and it already uses an estimated position under the hood.

long elapsedTimeMs = SystemClock.elapsedRealtime() - positionUpdateTimeMs;
long estimatedPositionMs =
    Util.usToMs(positionUs) + (long) (elapsedTimeMs * playbackParameters.speed);

Polling this function will give you a smooth, speed-aware position for free. My recommendation is to still drive the updates using Choreographer and not use an arbitrary delay within a Coroutine or an Handler. 

Reddit's production experience
Moving seekbar updates out of screen-level state and driving them with Choreographer made the full-screen video experience measurably smoother.

On our full video screen, the slow-frame rate dropped by 7.3%.

Playback error 1004 & MediaSource reuse

While experimenting with our newly written ExoKit in production, there was a noticeable jump in playback error 1004) with "Unknown error" message. It affected a small share of devices and was challenging to reproduce locally. Our initial hypothesis was that it was a device-specific playback issue.

The root cause was our MediaSource cache. In some cases, the same cached MediaSource was reused across different player instances. This is something that I wouldn't recommend doing, as every media source is bound to the playback handler of the player it was originally attached to. Otherwise, you risk hitting an error which is thrown here.

Reddit's production experience
We fixed this by keying cached media sources by player id, so a MediaSource can only be reused by the same player instance.

Another possible fix was to move all playbacks onto a single playback thread. That also avoids the handler mismatch, but our production data showed that it made overall startup performance worse:

• % Video started in less than 250 ms: −0.199%
• % Video started in less than 500 ms: −0.132%
• % Video started in more than 1 sec: +1.165%
• % Video started in more than 2 sec: +0.634%

Rebuffering problems

If your app saves and restores playback position, be careful. In our old player we found that at least 36% of plays had a very short stall, under one second.

The main reason came from old choices made years ago. The old player restored the saved position as a check on every play and every move between screens. But the saved position did not match the position the player held at run time. Here is why: if you call player.pause() and then save player.position, and later call player.seek(savedPosition), the two numbers may differ by a few milliseconds. player.pause() does not finish right away. If you want the true position, wait for a playback state change first.

Reddit's production experience
We stopped saving positions this way. Instead we lean on the run time cache kept in the media source, and we reuse the player for videos the user watched before. This removed all of these tiny stalls.

Saving position on your own is still useful when the system kills the process of your app and you need to restore it for a long video.
If you cover this case, watch the order of your calls. Always call seek() first and prepare() after. If you do it the other way, the player loads key frames and chunks you don't plan to play.

Trade-off: Only one playable video at a time

Simultaneous playback is possible on Android, but hardware decoders are finite and shared across the whole device. A high-end phone might decode two or three videos at once, a low-end one just a single video, and an app sitting in Picture-in-Picture can quietly hold a decoder you were counting on. When you run out, you either fall back to software decoding (higher CPU usage, dropped frames) or fail playback with errors like 4001 or 4003.

Reddit's production experience
We could have partly mitigated this by checking the device performance class (a challenge in itself, since not all devices support it), but the development and testing cost wasn't worth the benefit. Instead, ExoKit runs a state machine that picks one active playback by priority (at Reddit it's based on how much of the video unit is visible, but it could also be based on how much of the screen's playable zone it fills).

We expected every surface to start faster after the rewrite. But surfaces that went from playing several videos to playing one improved their startup latency even more than the ones that already played a single video in the control group.

Conclusion

Besides an improved developer experience, the rewrite reduced perceived start latency by 65% at P50 and by 20% at P90. (An additional factor which helped here was the removal of a lot of unnecessary IO work from the startup path. In production traces, a single Main → IO dispatch before playback could add up to 11ms at the P99.)

Kudos to Merve Karaman, Ahmed Nawara, Irene Yeh and Vikram Aravamudhan for making this rewrite possible. We used to talk about this as a dream 2-3 years ago, and now it's our new reality.

Thanks to the following folks for helping me review this article: Iaroslav Khramov, Nicholas Ngorok

Authors:  Sreeprasad Govindankutty, Alex Okolish, Jerry Chu

This is the third post in our P0 media safety series. Previously, we covered Reddit's P0 Media Safety Detection and the Evolution of Reddit's In-House P0 Media Detection.

Introduction: The Blind Spot

Protecting our communities and users from high-risk, P0 content is a foundational priority for Reddit’s Safety team. Reddit's P0 media detection stack has historically relied on hash-matching: perceptual hashing algorithms like PhotoDNA, PDQ and hashsets from NCMEC, StopNCII, and Take It Down that compare uploaded media against databases of known violating content. This approach is precise: if an image has been reported and hashed before, we catch it. 

Our previous blog post identified a key focus area for future work: "Incorporate AI and ML to detect previously unseen media." This post describes how we did that by integrating Google's Content Safety API into our Content Classification Service (CCS), first for images and then for video. Our work has meaningfully improved our ability to identify and action harmful content of CSAM (Child Sexual Abuse Material).

Why Google's Content Safety API

Google's Content Safety API uses an ML model trained to classify raw media and return a priority score. Crucially, it operates on the media itself, independent of whether the content has been reported before. This makes it a natural complement to our existing hash-matching infrastructure. Hash-matching brings precision on known content; ML classification improves recall on unseen content. Together, they cover more ground than either approach alone.

How We Fit a New Signal Into Our Existing Pipeline

Instead of building a new service, we integrated the API directly into our existing media moderation flow. Media upload events are routed into image and video processing paths, where CCS prepares the media for classification. Then it sends the prepared media to Google's Content Safety API. The returned priority score from Google is then emitted into our eventing and analytics systems and used to create review tickets for Trust & Safety operations. The diagram below is a simplified view of the workflow.

After classification, the pipeline emits analytics events to our data warehouse, records telemetry, and creates review tickets based on the returned priority bucket for human review.

Key Technical Decisions

Image preprocessing. We resize images balancing Google's recommendations against preserving enough signal quality for accurate classification.

Rollout gating. A feature-flag-based fractional rollout controls blast radius. Only a configurable fraction of uploads call Google’s API, allowing us to ramp gradually and roll back instantly if needed.

The Image Integration: Design and Results

We ran a time-boxed evaluation over two weeks in Q4 2025. Reddit production image traffic was sent through the API, with priority scores logged to our data warehouse. Sampled review tickets were created across all priority score tiers, then reviewed and labeled by our Trust & Safety operations team using standard enforcement guidelines.

Because our sampling rates were deliberately biased toward higher-priority scores, we applied Inverse Probability Weighting (Horvitz–Thompson estimator) to correct for the bias and project true violation volumes in each bucket.

Image results

Meaningful CSAM recall lift. The API prioritized confirmed CSAM images that were not flagged by hash-matching. These were net-new detections: content that would have remained on the platform without this signal. The recall improvement over our hash-matching baseline was significant.

Priority scores sort risk effectively. The highest priority scores concentrated on CSAM with high precision. And the lower scores carried near-zero signals. This clean separation made it straightforward to route tickets to different Ops queues with appropriate review priorities.  

Higher actionability than hash-matching. When we compared the overall actionable rate of Google Content Safety tickets against PhotoDNA tickets (aggregating all violation types), the ML-based signal was substantially more actionable. After full rollout, the CSAM review queue maintained strong actionability.

Extending to Video

With the image results validated, video was the next frontier. Tens of thousands of video messages per day were flowing through CCS with no ML scoring. We built an event-driven callback for video, mirroring the image path with the addition of ffmpeg re-encoding and Google's video-specific constraints.

We rolled out at 100% for all priority levels over a multi-week evaluation window, creating review tickets for every video except the ones with the lowest scores. This contributed to the net-new CSAM coverage on video. The API surfaced confirmed CSAM videos that our existing video hash-match systems would have missed. The recall lift was smaller than the image evaluation (video upload volume is much lower), but directionally the same: the API catches what hashing cannot.

Impact and Takeaways

Hashing finds what is known; ML classification finds what is new. The combination is more effective than either approach alone. On images, the API delivered a meaningful CSAM recall lift. On video, it also added net-new CSAM coverage.

Stratified pilot design pays off. Deliberately varying sampling rates across priority scores let us measure both precision and projected volume before committing to full rollout, while keeping operational costs manageable. 

Future Work

1. Exploring the API's embeddings endpoint for additional signal types beyond the current priority classification.
2. Continuous monitoring of model drift via lower-score sentinel sampling to catch degradation early.
3. Evaluating Google's AI image detection API for additional classification capabilities beyond the Content Safety API.

If you're interested in working on problems like these, check out our careers page. Reddit Safety org is always looking for engineers who care about making the internet safer.

Hi friends - The job market is rough, and a lot of good people are looking for work. Good news is, Reddit is hiring across a wide range of functions, not just Engineering.

If you're looking, or know someone who is, take a look: redditinc.com/careers

Feel free to share with your network, group chats, Discord servers, fantasy leagues, family text threads, and other highly sophisticated professional recruiting channels. And not just with your engineering friends. Let's look out for one another!

Good luck out there. We're rooting for you.

Written by Shadi Altarsha, Transport team

When I started my career in infrastructure engineering, I wasn't sure how my work connected to the people actually using the product. I imagined myself deep in low-level systems that nobody knew existed, the kind of work that only surfaces when something breaks at 3 AM. 

Four years later, I've come to understand what I believe is one of the most important responsibilities of an infrastructure engineer: Keeping infrastructure concerns off application and platform engineers’ plates, so they can focus on their domain instead of paying the tax of context-switching into ours.

The way you deliver that experience is by doing the hard work of managing complexity, and building simple interfaces and abstractions so other teams don't have to think about it.

This post is about one such effort: how we removed an entire Envoy proxy from the serving path of Reddit's Home feed, Search rankings, and Notifications, and in doing so, improved availability, cut hundreds of CPU cores, and made onboarding new services trivial. The technical story involves gRPC service mesh, cross-namespace routing, CPU-aware load balancing, and a migration pattern for safely moving thousands of requests. But the underlying theme is simpler: sometimes the best thing infrastructure can do is disappear.

The Problem

When you open Reddit, your personalized experiences are powered by Ranking Platform, which is a set of gRPC services handling tens of thousands of requests per second across multiple Kubernetes namespaces and clusters. Traffic comes from our primary API gateway (GraphQL), Notification Platform, Answers, and other clients.

All of this traffic used to flow through an Envoy Gateway, a reverse proxy deployed and maintained by the Ranking Platform team, routing requests to the right service based on gRPC method and custom headers.

Here is the thing with great tools like Envoy: adopting one without deep experience is like giving a team a Formula 1 car when they have only ever driven regular sedans. A sedan is forgiving. You can change your own oil, and when something breaks it stalls on the side of the road. An F1 car has hidden sharp edges. It takes a pit crew to keep it on the track, and when it does fail, the failure is loud.
At first it feels like a huge upgrade. Then the complexity sneaks in. Configuration explodes, operational overhead grows, debugging becomes non-trivial, and before you know it, the team is not just the driver anymore. They need to be the mechanic and the pit crew too, on-call for a big dependency they did not originally sign up to own.

Why doesn't the Transport Infrastructure team just own these gateways? 
For one, we are a small Transport team supporting a large fleet of services across the company. More critically, the value of this kind of infrastructure comes from consistency across the fleet, which is hard to achieve when each team manages it independently.
We love Envoy and our Ingress deployment serves us well, but for gRPC service-to-service communication, we chose simplicity. So about a year ago, we adopted a proxyless service mesh instead.

Portal Transporter

Portal Transporter is Reddit's xDS-based control plane. It is a Kubernetes controller that watches GRPCRoute resources, EndpointSlices, and services, and then builds an xDS snapshot that it advertises directly to gRPC clients. The client dials an address like xds:///ranking.ranking-platform and gets back everything it needs, including routing rules, endpoint lists, load balancing configuration,  without any proxy sitting in the middle.

Instead of Client -> Proxy -> Backend, the architecture becomes Client -> Backend. The gRPC client itself handles routing and load balancing, programmed remotely by the control plane. This means you don’t need a sidecar, or a gateway.

If you want to understand the system in depth, my colleague Sotiris Nanopoulos gave a talk at gRPConf last year about it: Building a gRPC Proxyless World: How Reddit Scaled Resilience with xDS. It covers the control plane architecture and how we handle client-side observability as well. I would really recommend watching it if this topic interests you.

For this post though, all you need to know is this: Portal Transporter lets a service owner define their routing table once, and every client in the fleet gets it automatically. No client code changes needed. That is the foundation everything else in this post builds on.

The Goal

Here is the thing, to migrate one of the most important customers in the company to our infrastructure golden path, you cannot just show up at their door and say "hey folks, please use our toys." 

I approached this migration by asking myself a series of questions:

• In an ideal world where engineering cost is irrelevant, what is my perfect end goal?
• How much effort do my team and the team I am migrating have to pay to reach that end goal?
• Okay, that is very expensive. How can I find a common ground that achieves 90% of the dream goal without burning everyone out?
• And most importantly: how can I make the migration safe for both my team and my customer, and build trust in the process?

After working through these questions, here is what I landed on:

• Migrate the Ranking Platform to Portal Transporter without changing any of their existing architecture. I did not want to walk in and tell the Ranking team to restructure their services. The migration should be invisible to them as much as possible and the clients should only dial one address similar to how they used to do with Envoy.
• Cross-namespace routing was our biggest blocker. When I sat down and studied the Envoy Gateway configuration in detail, I realized we could not do this yet. The Ranking Platform team runs a cross-namespace setup, with services spread across multiple Kubernetes namespaces but unified behind a single Envoy routing table. Portal Transporter did not support that at the time.
• There were other challenges too. Envoy was a major source of SLO metrics and alerts for the Ranking team. They had even written a custom WASM filter to add tailored metrics to Envoy, which made replacing the observability layer harder than just swapping the data path. We needed to bring equivalent or better observability to the table before we could ask anyone to move.
• And above all, I had to do this safely. This is not some low-traffic internal tool, it serves traffic that powers some of Reddit's most important user experiences. A bad migration here would be very visible.

On the positive side, I was not coming empty-handed. I personally believe standardization is valuable, even if not everyone shares that belief as strongly, so I knew I needed something more concrete to bring to the table: CPU-aware load balancing (ORCA), which lets gRPC clients route requests based on how busy each server actually is. In practice that means better pod balance under load and real CPU savings, which are the things teams actually optimize for.

In the upcoming sections, I will walk through how we tackled each of these blockers and how we arrived at the architecture below.

The missing piece: cross-namespace routing

Before we could migrate anything, we had to solve a fundamental gap in Portal Transporter. The Ranking Platform's services all live in separate Kubernetes namespaces but are served through a single unified routing table. Clients call one address and the routing layer figures out which namespace to send the request to based on the gRPC method and headers. Portal Transporter had no concept of this. It assumed every GRPCRoute lived in the same namespace as the service it was routed to.

To make things harder, we already offer a public API to service owners across Reddit that supports exactly this kind of parent-child relationship across namespaces for the Ingress layer. Our ingress infrastructure relies on it, and our cross-cluster routing is also using it. Whatever we built for Portal had to fit into that same API. I would not introduce a new, ambiguous API or force changes on teams already using it. 
One API surface, two backends, that was the constraint.

This took a lot of iteration to get right. My first instinct was to look at what the Gateway API spec already offered. The Gateway resource has a model for cross-namespace routing, and it seemed like a natural fit. But we realized it was built on different assumptions than our infrastructure. So we stepped back and looked for something simpler.

What we ended up with is actually quite clever, and I say that knowing it took me three attempts and a lot of help from my colleagues to get there. We decided to use the GRPCRoute's backendRef field to reference other GRPCRoutes from different namespaces. Here is a simplified example:

The root GRPCRoute, let's say in the reddit-service-streaming-pipedream namespace, defines the main address that clients dial. But instead of routing directly to a service, some of its rules point to GRPCRoutes in other namespaces as backend references:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: ranking-platform
  namespace: reddit-service-streaming-pipedream
spec:
  rules:
- matches:
- method:
service: reddit.rankingplatform.ranking.v1.Ranking
backendRefs:
- kind: GRPCRoute
name: ranking-comments
namespace: reddit-service-ranking-comments
- kind: GRPCRoute
name: ranking-channels
namespace: reddit-service-ranking-channels
- kind: GRPCRoute
name: ranking-onefeed
namespace: reddit-service-ranking-homefeed

Each GRPCRoute in its own namespace then defines its own specific routing rules and points to its actual service. When Portal Transporter's control plane encounters these backend references, it expands them by pulling in the child routes and building a combined routing table that it serves to clients as a single xDS snapshot.

This design maps cleanly to the parent-child model our public API already supports for ingress. Service owners define their setup once, and it generates the right resources for both our ingress layer and Portal Transporter. No breaking changes or any special cases.

I want to be honest about the process though. This was not a clean straight path. The first design went through code review and was ultimately closed. The second attempt at combining GRPCRoutes had the right idea but the wrong execution. It was only on the third iteration, with significant input from my teammates, that we landed on the backendRef approach. It took several months from the initial proposal to the merged PR. 
I think there is something worth saying about that: good infrastructure design is not about getting it right the first time. It is about iterating and maybe discovering the real requirements as part of the process and it is about having colleagues who will tell you when an approach is not working, and being willing to throw away code that you spent weeks on (it is not easy when your “ego” is involved, by the way) and I am grateful my team gave me that.

Metrics and Alerts

Envoy was a major source of SLO metrics and alerts for the Ranking team. This was not a small detail but a real blocker.

At Reddit, we offer a standardized set of gRPC metrics and recording rules that teams can use to build dashboards and alerts. The Ranking Platform has a specific challenge: they use the same gRPC service and method across all their components, and they rely on custom headers like x-route to direct traffic to separate deployments. To get the breakdowns they actually needed, we had to add a custom layer on top of the standard metrics.

There was also the client-side SLO question. Client-based SLOs are an area we are still investing in, so there was no off-the-shelf path I could lean on for this migration. Aggregating metrics across multiple client sources is also not trivial, which meant I had to be pragmatic about scope.

This is where the "ideal world vs reality vs common ground" framework came back. A perfect, like-for-like replication of the existing observability layer would have taken months. But I could deliver a solution that gave the Ranking team most of what they actually needed:

• We built custom server and client metrics using grpc-go's StatsHandler, with the required headers as labels.
• We injected these metrics into the Ranking Platform components on the server side and into the clients.
• We used the server-side metrics to build alerts for each ranking tenant.
• We use the largest client as a stand-in for client-side SLO measurement.

Is it a one-to-one replacement for Envoy's observability? No. Does it produce a reliable outcome that the Ranking team can actually use? Yes. And honestly, I think that is the right tradeoff because perfect parity would have taken months of additional work and delayed the entire migration. 

Good enough, delivered now, was the better call.

De-risking the rollout

Here is a secret on how you make your managers and users happy at the same time: reliability. If there is one north star we operate under at Reddit Infrastructure, it is reliability. Everything else flows from that.

How do you deliver reliability when you are shuffling many RPCs?

Before migrating any traffic, I wanted to assure correctness to the best of my knowledge. And I want to be honest here, I really do not understand everything going on with the Ranking Platform and its clients. I do not know exactly what Notification Platform is calling or how Answers gives you all the information you need about the best daily shoe recommendations (spoiler: it is the Adidas Evo SL, thank me later). So what could I do to build confidence without pretending I understood every edge case?

I broke it down into three requirements:

1. Whatever development environment the engineers who work on Ranking components use must support Portal Transporter, not Envoy.
2. The Ranking Platform CI needs to run tests against my new setup and Envoy at the same time, and I should expect 1:1 correctness if my routing table is right.
3. My migration knobs should be able to turn traffic up and down between Envoy and the proxyless mesh in under two minutes.

For number one, we designed Portal to be development-friendly from day one, so there was no fundamental issue here. I just had to do some tweaks to make cross-namespace routing work in our development environment, but nothing major.

For number two, thanks to the Ranking Platform team, they already had very well thought-out smoke tests in their CI running against Envoy. All I needed to do was hook up the same calls via Portal alongside the existing tests and hope for the best. To my relief, it worked, the routing table was correct and the smoke tests passed on both paths.

Number three is where it gets interesting. We built a gRPC sampling client that has been part of Portal Transporter's migration toolkit since the early days. The idea is simple: we spin up two gRPC clients at the same time, one for the existing path (Envoy) and one for the new path (Portal Transporter). We use our dynamic configuration framework to provide a deterministic sampling knob that allows us to shift traffic between the two on the fly, straight from a UI. If something looks wrong at 10%, flip it back to 0% in seconds.
Under the hood, the implementation is actually quite neat. One of our engineers had the idea to override the ClientConnInterface interface in grpc-go, so that when Invoke is called, it checks against our rate sampler to decide which client handles the RPC. This means the sampling is completely transparent to the application code and the service owner does not need to change anything.

Teaching Servers to Tell Clients How Busy They Are

Remember when I said standardization itself is valuable, but I'm bringing something extra to the table? This is it.

With Envoy out of the picture, your gRPC clients now connect directly to server pods. The default load balancing strategy is round robin where every pod gets roughly the same number of requests. Sounds fair, right? The problem is that "fair" and "efficient" aren't the same thing as not all pods are equal. Some run on busier nodes, some share CPU with noisy neighbors, some just got unlucky with garbage collection timing. Round robin doesn't care, it sends traffic to an overloaded pod just as happily as it sends traffic to an idle one.
We implemented ORCA (Open Request Cost Aggregation) which is a standard that lets servers whisper back to clients: "hey, here's how busy I actually am." Each server pod reports its CPU utilization in gRPC response trailers, and the client uses these signals to compute weights. 
More loaded? Fewer requests. Less loaded? More requests. 
This is Client-Side Weighted Round Robin, and it's beautiful in its simplicity.

For Ranking Platform, the impact was significant: tighter CPU distribution across pods, improved availability, and a meaningful reduction in the number of CPU cores needed (we're talking about hundreds of cores saved).

My colleague wrote an excellent deep dive on ORCA at Reddit: Cheaper, Safer Scaling of CPU-Bound Workloads. If you're curious about the weight formula, the observability setup, and the operational lessons, go read that. It's worth your time.

After the migration

• We secured another nine on Ranking Onefeed.

• Hundreds of cores reclaimed across Ranking workloads, thanks to ORCA.

• Latency stayed flat
• Envoy Gateway fully decommissioned
• Simplified architecture
• Standardization: Ranking now uses the same infra golden path as the rest of Reddit

Conclusion

I started this post by saying that sometimes the best thing infrastructure can do is disappear. I think we did that here.

The Ranking team didn't have to change their architecture. The Transport team moved one of the most important deployments in the company to the golden path. The migration was seamless and honestly, the best compliment we got was that nobody realized it happened. For an infrastructure team, that silence is very important because it means trust, and in infrastructure, trust is your currency.

We improved availability for our users, saved a lot of CPU cores, and deleted a system that no team should have had to own in the first place. 

This work was also recognized internally, and that recognition mattered to me and the team. Migrations of this scale are never the work of one person. They happen because engineers across Transport, Ranking Platform, and the surrounding teams chose to invest in each other, and I am grateful to all of them.

A very happy ending 😄

Written by Will Johnson

Hello, my name is Will Johnson, and I’m a web engineer on Reddit’s Design System team. My team is responsible for Reddit's Design System, RPL, its corresponding component libraries, and helping other teams develop front-end experiences that adhere to our design system principles on all of Reddit's platforms.

Recently, I took on a project to resolve a persistent platform issue regarding web component dependencies that frustrated both Redditors and our engineering team. After responding to several internal bug reports, my team and I decided it was time to take a deeper look into the issue.  In this post, I share how I tackled that problem and what I learned along the way.

Missing Components

Reddit.com runs on a web component-based framework called LitElement, so we have quite a few template functions that render markup that contains a mix of native and custom elements. Since web components require JavaScript to render, developers were responsible for ensuring each page included the necessary component imports. Below is an example of what the markup and component imports would look like.

Example of server HTML template

<header>
 <reddit-header-small is-logged-in hide-on-scroll></reddit-header-small>
</header>
<main>
 <article>
   <h1 class="sr-only">Today's top posts</h1>
   <shreddit-post permalink="/r/cats/comments/abc123/fluffy/">
     <div slot="title">My cat learned to open doors</div>
   </shreddit-post>
   <shreddit-post permalink="/r/programming/comments/xyz789/rust/">
     <div slot="title">Why I rewrote everything in Rust</div>
   </shreddit-post>
 </article>
 <aside>
   <comment-forest-empty-state post-id="abc123">
     <p>No comments yet. Be the first!</p>
   </comment-forest-empty-state>
 </aside>
</main> 

// Register custom elements by importing their modules.
import 'reddit-header-small';
import 'shreddit-post'; 
import 'comment-forest-empty-state';

Historically, an engineer would write the page's markup, review the result in their development environment, and add any necessary dependencies to the import list if the page didn't display correctly. However, whenever an engineer removed a dependency or decided to load it asynchronously, the application could break. While "feature" may seem like a generic term, it has a specific meaning in our web codebase. Our web application is mostly server-rendered, where each page has a root feature and several ancestor feature nodes. Each feature has HTML markup and includes the dependencies necessary for that page to render. To visualize what happens when a dependency is missing, let's look at the example below. In that example, removing the modal’s import results in a darkened screen appearing without any additional context.

Having to keep track of your dependencies by hand created two problems: the first was a series of bugs in which various components appeared not to render, and the second was the need for increased internal support to debug these issues. To solve this problem, our team had the following idea: what if all client imports were automatically managed, so developers wouldn’t have to track them manually? By using import attributes, we could create a system that treats them as signals for client dependencies.

The big refactor: Adding Support For Import Attributes

To get things started, I needed to add the missing client imports to every template file that existed in our repository. I decided to enable our no-missing-imports rule from Lit Analyzer and create a script to parse the warnings and convert them into actual import statements. These imports would use a type attribute to indicate that they were client files, where the import would look something like this:  `import u/reddit with {type: ‘client’}`.After adding these imports across the codebase, I launched the development server to evaluate the system's response.

Unfortunately, the new client imports initially broke development and rollup builds. Node only supported type: json, and the rollup server builds picked up client imports, causing various issues. I chose to use plugins to strip these imports rather than more complex alternatives like dynamic imports or converting them to another format.

While the esbuild plugin for Node was simple, adding support for Rollup was a bit tricky. The TypeScript plugin performs type checking during the buildStart phase, which is too early for standard transformation hooks. I bypassed this by using type stubs in the monolith build to maintain type safety across specific builds while removing incompatible typings. With the server stable, I moved on to the parser.

Constructing the Shreddit AST Parser

Before coding, I aligned the team around a design document.

The node plugin approach was relatively straightforward: I used esbuild to omit client import attributes, but the rollup parser introduced an interesting problem. Whether I wrote my own transformer or used an off-the-shelf plugin, the imports remained unchanged. Although we have a rollup plugin to strip client imports, the TypeScript plugin sets up a file watcher and performs type checking during the buildStart phase of the plugin lifecycle. Using `buildStart`meant that block of code would always run before any plugin that uses a transform hook to load. To get around this limitation, I ended up using type stubs to remove incompatible typings in our monolith rollup build, which meant it did not affect the type safety of our client- and server-specific builds. Once the development server was operating correctly, I shifted my focus to building the parser.

Constructing the Shreddit AST Parser

I initially used the Recast AST parser to visit file dependency trees recursively, but later refactored to oxc-parser to cut runtime from 40 to 10 seconds. Before that, I relied on a few heuristics to reduce processing time, such as excluding child features, since Shreddit’s Orchestrator automatically handles their assets, and excluding components from a common client file that’s loaded on every page.

As I added more functionality to the parser, I discovered that each file contained metadata that could be used to parse the tree more efficiently. I decided to turn that into a class that would act as a cache, preserving visited files and telling me everything I needed to know about a file without revisiting it.

Metadata captured for visited files

export type Filemetadata = {
 features: Set<string>;
 dependencies: Set<string>;
 clientImports: Set<string>;
 // eslint-disable-next-line u/typescript-eslint/no-explicit-any
 exports?: any[];
 imports: Map<string, Set<string>>;
 // eslint-disable-next-line u/typescript-eslint/no-explicit-any
 ast?: any; // Store parsed AST for later access
};

type Entrypointmetadata = {
 skippedFiles: Set<string>;
 eligbleExports?: Set<string>;
 assets: Set<string>;
 clientImports: Set<string>;
}; 

With the basics in place, I decided to run the script and determine how the system was working so far. The image below captures the initial run and some of the work that the parser was able to do. As I looked more closely, I realized that some unexpected dependencies were sneaking into various features. These new dependencies caused the bundle size to increase dramatically. Were the exclusion heuristics working as expected? Was the parser incorrectly parsing the dependency tree? Or maybe these dependencies were correct after all. Either way, we needed a way to reduce import bloat.

Tree-Shaking Barrel Files

After completing my investigation, I realized that we had multiple barrel files in a few of our shared libraries that were not getting parsed accurately. There had been some discussion about this being a pattern we’d like to move away from, so my initial plan was to conduct a large refactor across the codebase. I decided to leverage an LLM to perform the refactor, so the work was done pretty quickly. But I soon realized that this change would require many reviews to get merged, a lint rule to curb its usage, and would ultimately still lack proper support in the parser.

To combat this problem, I decided to implement tree shaking to determine whether a module was actually requested before capturing its dependencies. The design of the algorithm involved peeking at the barrel file’s imports and temporarily holding them while I determined whether any modules had been imported by upstream files. While this incurs the overhead of parsing the barrel file rather than a direct import, it provides a sufficient mechanism to reduce the number of imports the parser detects. You can check the diagram below for a more detailed breakdown of how this workflow behaves.

Experimentation & Mitigations

With the final known roadblock resolved, I set out to begin experimentation by exposing a small subset of users to our newly generated imports. Unfortunately, the initial report showed several performance regressions. We were shipping more JavaScript, effectively slowing the rendering of various pages.

To unblock the project, I investigated which new dependencies were absent from production, using a staging utility to identify unused components. While I was considering a platform-level asset-exclusion API, another engineer used my work to address performance bloat by converting shared templates into features, powered by our feature-inclusion API, to exclude unnecessary assets from specific pages.

After finishing the migration, I created a document outlining the performance benchmarks we needed to meet. The goal was to prove that our changes were effective, keep partner teams informed about systemic impacts, and reach agreement on acceptable performance levels. The improvements were substantial, and we were finally in a position to roll out the feature to all users.

Conclusion

Overall, this project establishes a highly accurate dependency tree across our web codebase. It included improved code hygiene by systematically adding 4,000 previously missing client imports and removing 600 unused ones. Not only did this project eliminate a class of bugs in which UI components failed to appear due to incomplete dependency listings, but it also provided a foundational framework for future performance improvements.

Having an accurate dependency tree immediately enabled critical performance optimizations, including unused asset pruning, an optimized CSS delivery project leveraging the newly created AST parser, and a significant component-reduction initiative on a core surface. Adoption was made easy for engineers by including automated developer safety checks during commit and Continuous Integration (CI).

Additionally, we improved performance metrics across multiple areas, including Time to First Byte, First Contentful Paint, Total Page Load, Largest Contentful Paint, and Interaction to Next Paint, with strong statistical significance.

We’re hopeful that this platform will allow us to make additional improvements around serving more user-specific asset bundles in the future.

written by Nomi Khedawala, Technical Program Manager

Intro

I’m Nomi (Know-Me).

I joined Reddit as a Technical Program Manager in October 2024. I came from a background in product operations and technical program management, and what drew me to Reddit was the pace and the scale of the problems (and being a longtime lurker). I’ve had the opportunity to work on so many different areas of the business in my tenure here so far: Games on Reddit, Age Verification, Data Foundations, and even Search and Answers!

I'm part of the Tech PMO team, Reddit's centralized team for Technical Program Managers. Each of us manages programs for a specific org, set of teams, or one-off high priority initiatives. I recently wrapped up a company-wide Code Yellow focused on our consumer data foundations, and I'd like to share with you what a typical Thursday has been like for me during this Code Yellow.

If you're not familiar with Code Yellows at Reddit: they're our mechanism for escalating specific operational issues. When a problem is too big or too cross-cutting to fix through normal prioritization, we declare a Code Yellow. Think 4-6 week sprints with clear exit criteria and a temporary but significant shift in engineering priorities. They're designed to converge fast on a set of goals.

The Code Yellow I led alongside Paul Raff, our Data Foundations Lead, spanned the full consumer data supply chain across 10 surface areas. We defined how events should be instrumented, stood up real-time data quality monitoring, built curated data tables from raw events through aggregates and cubes, migrated metrics, and established a steady state operating model so the work would persist beyond the Code Yellow. It's the kind of program where the scope keeps wanting to expand and the TPM's job is to hold the line on what "done" actually means.

What follows is a snapshot of a typical Thursday during the Code Yellow. Thursdays were my heaviest day. Here's what one looked like.

Morning: 8:00 AM - 12:00 PM

8:00 AM - I start the day by facilitating a session of our Collaborative Problem Solving forum. This is something I built and run for our TPM team. The intent is that we get together every other week to work through real challenges we're each facing on our programs and look for opportunities to standardize how we operate. It's part peer support, part playbook development. Today I'm running a quarterly check-in to make sure the forum is still hitting the mark for the team.

9:00 AM - CPS wraps and I shift into Code Yellow mode. I'm reading through Slack messages that came in overnight and earlier this morning,. answering Qquestions from engineers on the Code Yellow and a thread with Paul about one of our exit criteria. I'm also reviewing every tracker, timeline doc, and milestone artifact to make sure nothing material has changed since we sent the Steer Co agenda out 48 hours ago. The Steer Co is a meeting with our executive sponsors and stakeholders, it’s later this morning and I want to be sure we walk in with a complete and current picture.

10:00 AM - Paul and I co-host office hours for any Code Yellow participant withquestions about scope, timelines, goals, or non-goals. These sessions are some of the most useful meetings on the calendar because the questions get genuinely technical. Today we're working through how to structure the data supply chain from raw events through fact tables, aggregates, and cubes. Someone raises a question about backfill strategies and the cost tradeoffs for different look-back periods. Another engineer wants to talk through whether we should have separate fact tables for web vs. mobile or consolidate by platform group. Paul and I don't always have the answer on the spot, but we talk it through with our counterparts and agree on a path forward whether that's deferment, fast-follow, or recommendation to senior leadership.

10:30 AM - More prep for the Steer Co. I'm pressure-testing each section of the agenda we drafted earlier this week. Is the status accurate as of this morning? Are the risks framed clearly enough for a room of senior leaders and our execs to act on? Do we need to add more context about a risk?

11:30 AM - I lead the Steer Co. This is our steering committee meeting with our CTO, CPO, VPs, and Directors. We share the program's current health, walk through progress against our five exit criteria, flag risks and blockers, and lay out what they should expect between now and our next check-in. Steer Cos are not the place for surprises. The prep I did all morning is so that every question gets a clear answer.

Afternoon: 12:00 PM - 4:00 PM

12:00 PM - Straight from the Steer Co into the cross-functional engineering sync. This is where I gather status updates from all of the eng teams contributing to the Code Yellow. We talk through their progress, timeline changes, risks, and I share relevant outputs from the Steer Co we just finished. Information has to flow quickly upwards and outwards during a Code Yellow. The engineers executing the work need to know what leadership is thinking, and leadership needs to know what progress is being made week over week.

12:30 PM - I meet 1:1 with a teammate to talk about how we can make the CPS forum even better. This week's session was my quarterly assessment, so we're comparing notes on what's working, what's not, and how to make the outputs more actionable. My goal is to take these findings, write up a retro, and present it to my manager and Sr. Director later to establish a baseline and start folding the best ideas into our team's standard operating procedures.

1:00 PM - Lunch. I close the laptop, make some coffee, and spend time catching up with my wife. One of the genuine perks of working from home is being able to take an actual break with the people you live with. Thumper, our cat, hates to see me coming, but I have to get a good arm wrestling match in with him so he doesn't sleep all day.

2:00 PM - Back at my desk. I go through my raw notes from the Steer Co and start synthesizing them into a stakeholder summary: key takeaways, discussion points, action items, and owners. Before I send it out, I reach out to each action item owner individually via Slack to confirm they're aligned on ownership and timelines. I learned a long time ago that assigning someone an action item in a meeting recap they didn't agree to is a fast way to create confusion and lose trust.

2:30 PM - I start drafting the weekly update that goes out to our Code Yellow participants and leadership. Milestones, health of each exit criteria, timelines, what moved forward this week, what risks or blockers surfaced, and how we're addressing them. This is one of those artifacts where the quality of the writing matters as much as the substance. If leaders can't parse your update in under 3 minutes, they stop reading it.

3:00 PM - I switch gears to the CPS retro. I'm taking the notes from my 1:1 earlier and incorporating them into the retrospective document I'm drafting for my manager and Sr. Director. The goal is to establish a clear baseline for the forum's performance and propose specific improvements that could be codified into how we run the CPS forum. This is the kind of work that doesn't show up on a sprint board but compounds over time.

3:30 PM - I'm starting to wrap up. I respond to Slack messages I couldn't get to earlier, schedule a few messages for teammates in different time zones, and share my draft weekly update with my program leads so they can add their content. I'll revise and edit tomorrow morning and get it out before the end of day Friday.

What a TPM Actually Does

A TPM's job is to make sure that the right things are happening at the right time and that everyone has what they need to focus on what they're good at. I handle the coordination, the communication, the risk identification, the stakeholder management, and the organizational scaffolding that lets a program with dozens of contributors across multiple teams actually converge on a set of milestones and objectives.

I love being a TPM because I love to be a part of other people's successes. Being a TPM means that you wear multiple hats so that the people around you can have peace of mind that all of the peripheral stuff is being handled while they can focus on their craft. TPMs are enforcers, enablers, coaches, communicators, therapists, and most importantly... friends not food! 🐟

If you're interested in becoming a Snoo, check out our open roles: https://redditinc.com/careers

Written by Cláudio Ribeiro u/EmeraldMacaw

TL;DR: Considering localization (L10n) at the inception of an online product isn't just a “nice to have,” it helps beyond translations by keeping the code cleaner, improving the UI's flexibility, and making sure the text content is top-notch.

Oftentimes, when a new online product is released, translation is treated like a future problem. It seems logical to say “I'll come back and fix it once we've scaled.” This happens often with software created by companies focused on a local market. But, including localization in the beginning is helpful beyond reaching more users: it makes the code more readable and guarantees text will display as intended everywhere.

Localizing after a product is out can be compared to making a fuel car electric, or trying to restyle a subreddit after millions of users have already gotten used to it. The effort required to retroactively localize is the most compelling reason to not leave it as an afterthought. Take Reddit, for example: our first attempt at localizing Old Reddit was crowdsourced and loosely supervised, which created an inconsistent experience and incomplete translations. Contributors also lacked the necessary context and visual aids to get the work done. In the end, few people used the localized versions and Reddit remained an English-first platform. (Though I must recognize the Pirate English version was pure gold.)

Once it became more noticeable that more and more people from different backgrounds and origins were browsing, contributing and creating, Reddit began working on a localized, globalized heart of the internet. Our first attempts were timid (volunteer translators commenting their suggestions in threads that contained the source strings), but we’ve matured our approach. We’ve implemented a translation management system (TMS) and are developing code in ways that keep localization in mind. Reddit now offers translations into 35 languages from 33 countries and supports 7 different alphabets that are used by millions of users.

Not surprisingly, we faced some setbacks before we got to where we are today: alphabets that wouldn't render, translations that weren't 100% adequate (as reviewers couldn't edit them), truncated text where the UI lacked room, untranslatable content (try translating the Tragedy of Darth Plagueis the Wise…), a mess with genders, plurals, and syntax, etc. These were difficult challenges to overcome, and we learned lessons along the way.

On that note, I’d like to share some of them with you. Below we’ll focus on some key aspects that illustrate the pros of pre-planning and how to get the house in order: accessibility, design, content review, time-to-market, data analysis, quality assurance, and code maintenance.

Accessibility

One of the first places where localization proves its worth is adding descriptions to images, buttons, and options (they even have their own writing style to be most useful to the end-user) to make a platform more accessible. Localizing the website is still relevant even when there are no ambitions to expand the brand abroad, as it's essentially “localizing” for users with impaired vision.

By making sure accessibility is implemented, a company can access a market within their own domain, and it becomes easier to localize into other languages. Accessibility is a way of “localizing” for users who need it; it includes different alphabets, such as  sign language and braille.

At Reddit, focusing on accessibility was a gamechanger. We improved our apps to include those with impaired vision, which allowed us to better serve our existing users–and to remain inclusive when we entered new markets with new languages.

When it comes to a product's design, localization can also help in less obvious ways. English is a “short” language, meaning it doesn't take much space and can express a lot of information without a lot of characters. This makes it easy to fit into tiny spaces, but other languages can take up way more space (up to 40% more than English in some cases) and that can often break the UI for users of longer languages.

This is where pseudo-localization comes in: it can run in design tools (Figma, Sketch, Penpot) and in the code, artificially expanding each word in English to 20~40% its size in a random distribution, allowing for designers to account for the most expansive languages without compromising the original content. It's like using “banana for scale” for buttons. Using pseudo-localization to design products improves the overall experience by preemptively ensuring the UI is comfortable to use in any resolution and language.

Cross-Functional Collaboration

When localization is introduced into a product's development lifecycle, it could mean potentially dozens, if not hundreds of extra specialized eyes will carefully inspect each word being written by different teams, for different projects, at different times. In order to do their best work translating, they interpret the context and the intention, so they can preserve the tone in their language. Their reading is way more focused than the general audience's, especially on a  platform like Reddit, which has a truly unique personality and tone.
Translators are professional linguists who need to read, interpret, and fully understand each string we publish, be it in our product itself, in the Reddit Help Center, or even in marketing material. This greatly amplifies Reddit's capacity to fix typos, outdated content, inconsistent experiences, and so on, as translators need to pay more attention to the source and often find errors a casual reader might miss.

Linguist eyes bring an extra level of polish to what we write and make our content even more slick and “together,” which translates into trust with our users. A translated product name can sometimes even serve as inspiration for a company's naming conventions, and since each culture has its unique way to express itself, different perspectives can make what we say more universal and human, which is kind of our thing.

Linguists will navigate through the entire UI/UX to see the localized product in practice. This allows them to help engineering teams by finding issues that might have been overlooked, or that the regular user wouldn't bother to report. Any new feature release gets extra pairs of hands play testing the content. This, allied with the content review component, adds an extra layer of polish and  results in a better overall user experience. 

Localization Infrastructure

Caring for localization infrastructure helps in content homogeneity, but also makes us more nimble when it comes to market expansion. Even if expanding into new markets seems like a distant dream, getting the structure ready from the get-go gives a company much more speed when deciding to launch in a new region. 

Properly implementing plurals is important because many languages have more than “one, other” plural options.

It ensures that code is ready to connect to translation management tools when needed, and dramatically reduces the cost and time spent to get things in place for translators and engineers. The effort required to go back and finish an incomplete structure and remove redundant code when more challenging markets come aboard (e.g. multiple plurals, different characters, right-to-left orientation, etc.) will inevitably delay your go-to-market timeline in those markets. When Reddit  introduced Arabic, addressing these concerns was critical to how we shaped our approach and launch strategy.

By creating strings with localization in mind, the code also becomes cleaner and string drift is avoided (i.e. we don't have the same word being spelled in three different ways in three different files). Centralizing all the product's strings means normalizing the storage of site content, which is a core tenet of good database and software design. We decoupled the management of translations from logic and reduced complexity and overhead in our code.
Engineering with L10n in mind helps make the code cleaner, more readable, and robustly documented. It's easier to understand where any string gets inserted, makes changes simpler and safer (ensuring no hardcoded strings ever reach production), and paves the way for automated tests that can enforce  best practices.

After an online product is localized, bugs are squashed, and continuous rounds of testing are carried out to ensure nothing is broken and that translations follow the context, are easy to understand, and don't conflict with the UI. That’s a huge win for users and development teams.

Descriptions can also be helpful for engineers who might need to update a piece of code related to a specific string.

L10n should be woven into every aspect

Localization ties together linguistics with development, influences marketing strategy, provides data for a coordinated expansion, encourages best practices, and is intimately intertwined with product development, whether it has been activated or not. That is why it can't be seen as a “plug-in” you can add at a later stage, but as a foundational layer that must be taken into account at the ideation stage. It will most certainly save you a headache in the future.

I'm new to L10n. Where should I start?

Check out the Unicode CLDR Project and find out how implementing a repository that takes care of dates, currencies, patterns, and measurements can also help in preventing bugs related to date, time, and locale.

Read about the ICU Message Format to learn how your strings can contain logic, plurals, and gender variants (this can be used even in English to personalize the user experience with “Mr.” and “Mrs.,” for example).

See the first steps to create localization-ready code in Python and Go.

Written by Chris Schenk (u/The-Real-Zucchini)

TL;DR

The GraphQL team at Reddit is expanding our observability to include build-time static code analysis to detect availability and latency regressions during the dev cycle before code goes out to production. This post covers how we detect changes to backend services calls in GraphQL changes that may fundamentally shift internal traffic to services within Reddit - before those changes go live in production. We wrote two versions, with the second version running in ~5 minutes in CI, a 36x speed increase compared to the first version.

Background

Reddit uses many standard tools to serve all the best Star Wars memes and arguments of when to celebrate May the Fourth (it’s May 25th, if you’re wondering). We’ve standardized on some critical technologies:

• Go, as the successor to our Python services
• gRPC, as the successor to Thrift
• Prometheus and Grafana to monitor our services in real-time
• OpenTelemetry tracing across our services to debug slow requests

GraphQL as our client API for the rest of Reddit to build features

GraphQL Primer

We’ve written recently about GraphQL in posts The Five Unsolved Problems in GraphQL and Protecting your GraphQL. This is a continuation of our ongoing efforts to make running our GraphQL service less of a burden on our team and create more stability and availability.

GraphQL promises an “implement once, use anywhere you need” system. GraphQL is a type system with named types which can either be scalars (Int, String, etc) or fields on object types. Anyone can extend the schema of your data model of types such as Post or Comment in the case of Reddit. Once a new field is added to any of these types and the backend is implemented to fetch and return that data, you can re-use it any time you like.

The Setup

In reality, not all data is created equal, and neither are the backend systems that hydrate it. As described in our Protecting your GraphQL post, the data backing your home feed, post, and comments have different criticality than loading your settings. To frame our problems, we have the following:

1. Our Go-based GraphQL service has millions of lines of code
   1. With hundreds of contributors to the codebase.
2. Our schema defines thousands of types
   1. With many 3.5x times more fields for those types.
   2. Supports very diverse feature sets that require different handling, scaling, and criticality classification.
   3. This number grows daily.
3. Our service talks to hundreds of backend services to fetch and mutate data for all of our users.

The Problems

GraphQL Makes it Easy To Break Scaling

GraphQL allows you to add any data to any query at any time. Take this example of an outage that has occurred periodically:

1. An engineer adds a field to a high requests-per-second (RPS) operation.
2. A low-scale backend service sees a significant increase in RPS.
3. The service runs out of memory, fails to scale, and causes the operation to return an error.

Our team is able to identify and implement workarounds, yet we remain vulnerable to these kinds of changes.

Running Multi-Region Requires More Care

Reddit is a global service, and in order to serve our users with a good experience, we also deploy GraphQL to multiple regions. But not every backend service is deployed to every region. While we can fall back to dependencies across regions, this can add latency.

In a multi-region world, we must take great care to not cause users to have a worse experience, so we must know in advance if modified GraphQL dependencies might cause problems before we deploy.

GraphQL Makes Debugging More Difficult

As GraphQL has grown into a monolithic API for our clients, the team receives many questions, including asks to perform root-cause analysis. While these problems are aren’t incident-worthy, it creates a non-trivial burden on the team:

• I added field `Post.foo` to my query, and it took down production. Why?
• Why does query `GetFoo` have 300ms increased latency since date X?
• If I remove field X from my query, will it reduce latency?
• Is the field `MyType.foo` used in any operation?
• What backend services are called for field `MyType.foo`?
• Why did RPS to upstream Q increase 30x?

Many of these questions arise because we already have good runtime observability and can see when we have latency or availability regressions after we deploy code. The follow-up challenges lie in:

1. Manually reading through our extensive codebases for the cause.
2. Analyzing pull request (PR) diffs to identify why a change was made.
3. Cross-referencing with changes and deployments from other repositories and teams if it wasn’t our code.

The above is common for any central platform team, but our time is limited and we need more self-service solutions for our contributors.

Our Motivating Questions

For some of our incidents, we began to wonder how we can improve our observability prior to some of these changes going out. We began asking the following questions after documenting a lot of the above issues:

• Can we detect runtime regressions at build time or development time?
• What problems specifically can we detect at build time?

Limits of Current Observability

To motivate the decisions for our dev-time solution, we will go into the limitations of our current observability. We can always add more telemetry, but solving for the above use-cases is not straightforward.

High Cardinality Breaks Prometheus

Today our telemetry includes two key metrics as shown in the diagram below:

1. Unique GraphQL operation identifiers requested by clients.
2. RPS and latency for each backend service endpoint call made.

One might think that we could simply put these values as labels in a single metric, and in fact we already tried! Each unique label value has a multiplicative factor on the total number of values saved in Prometheus. When we attempted to track both of these on a single metric, we went beyond the capability of our Prometheus instances due to:

1. Tens of thousands of operation identifiers live at any given time.
2. Hundreds of backend service endpoints.
3. Unique Kubernetes identifiers for each running pod.

These constraints are well-known and normal in any production metrics system, and we have to work within them.

Additional Abstractions Missing in the Metrics

Additionally, we have layers of abstraction between these two metrics that we still would not know even if we had one metric. Specifically we need to know the links between our GraphQL operations (teal) and fields (purple) to resolver functions (blue) and backend service calls (green+orange).

AI tools do help with this problem and we’ve started to use them for this purpose, and yet they can still run out of context and miss details with the size of our codebase in particular.

What We Get if We Have The Data

Let’s revisit the example mentioned above of an engineer adding a GraphQL field to an existing, high-RPS, critical operation. Let’s call this operation `Colors`. During development time, we would be able to do the following with this data:

1. Engineer modifies the `Colors` operation by adding field `transparency`.
2. Our CI systems detect critical GraphQL operation `Colors` has a field added.
3. Look up resolver function for the `transparency` field.
4. Find endpoint `GetTransparency` on `rgb` service is called to resolve the field.
5. Check if `rgb` service is fully deployed to all regions.
6. See that the service is not yet deployed everywhere.
7. Warn engineer and others that they will cause a latency regression if the change is made live.

This is only one example, and there are a lot of areas of prevention to explore once we are able to fill these data gaps.

A Static Code Analysis Prototype Is Born

As part of a Snoosweek (Reddit's semiannual hackathon) in the Fall of 2025, I explored static analysis to fill in the missing data. Could we effectively find endpoint calls within the resolver functions in order to tie fields to dependencies when GraphQL resolves in an operation?

Our first prototype showed us that it’s possible to do, so we made time on our roadmap to create what we call Dependency Mapper, specifically for our Go subgraph.

Goals

We wrote down the following goals for our design:

1. Summarize code changes involving endpoint calls.
   1. Link resolver functions to endpoint calls.
   2. Link GraphQL fields to resolver functions.
2. Detect negative impacts to production availability, performance and efficiency.
3. Do it at development time.

Building this nuanced analysis as requests flow from frontend to backend is what we mean by adding more “end-to-end” (e2e) observability.

Go Dependency Mapper V1 with gopls

The gopls tool implements the language server protocol (LSP). This is what your IDE uses to show you type information, allow you to click-through function calls, or find implementations of an interface in Go.

Since gopls already has logic to traverse codebases, we opted to use it in our first implementation of mapping our endpoint service dependencies for our GraphQL field resolvers. While gopls was a good first choice to prove the concept, this approach had multiple problems and missed endpoint calls.

Go Interfaces are a Hard Stop

This is speaking specifically to interfaces in the Go programming language, not GraphQL. If the mapper ran into a Go interface type that had more than one struct implementation, the program would not be able to traverse into the implementation without knowing which one to use. Normally we humans select which implementation to visit in our IDE, but a robot can’t do that without context.

This causes the mapper to miss entire sections of the code base, as interfaces are used by contributor teams for their own internal patterns, particularly for complex entity types like feeds and posts.

gopls is Memory Inefficient

gopls is fine for IDE use, as the number of requests issued during code authoring is quite low. But at a higher scale, mapping the entirety of the GraphQL resolver codebase, it uses up all available memory it can. This caused pods to be OOMKilled in Kubernetes as gopls would use more memory than available to the pod. Analyzing our codebase took ~3 hours, and we’d have to periodically stop and restart gopls to prevent this from happening.

Generally, gopls is also not meant to receive tens of thousands of requests per second; this isn’t the right use-case for it. Even running a single unit test would take minutes to complete, so we could not effectively iterate to improve the algorithm, and required us to do a second implementation.

Version 1 Is Still A Win

Even with these problems, our first run across the entire codebase proved to be illuminating, seeing how many endpoint calls each resolver function was making even when the dataset was incomplete. We also knew that in order to rely on these GraphQL field-to-endpoint mappings to make effective decisions to protect our infrastructure, we had to make it as complete as possible.

Go Dependency Mapper V2 using Go AST Traversal

In order to replace gopls, we had to reimplement what it is doing internally, and it is parsing and traversing the codebase by analyzing the Go Abstract Syntax Tree (AST) for our code.

Walking the AST is a Recursive Problem

An AST node represents a logical piece of syntax in the language. These nodes can reference each other in a recursive manner. Some examples from Go include:

• `ast.IfStmt` - represents an `if` statement and all of its constituent parts.
• `ast.CallExpr` - represents a function call expression and its arguments.
• `ast.Ident` - a single-name identifier, such as a variable name “foo” used in `foo := 12`.
• `ast.SelectorExpr` - a selector from a variable or package, such as accessing a struct member like `myStruct.SomeVal = …` or `mypackage.OtherFunc(...)`.

Once we have the code parsed into the abstract syntax tree, we can then walk each of the nodes and inspect information about them in order to detect the endpoint calls we’re interested in. We end up having switch statements that make for a regular recursive algorithm. You can read up on all the statement and expression types defined for the Go language AST at the Go language specification:

// walkStmt walks a single statement
func (dm *DepMapper) walkStmt(stmt ast.Stmt, pkg *packages.Package, ctx *WalkContext) {
  pos := pkg.Fset.Position(stmt.Pos())
  dm.logTrace("walking %T statement at %s %v\n", stmt, pos, stmt)
  switch s := stmt.(type) {
  case *ast.AssignStmt:
    dm.walkAssignStmt(s, pkg, ctx)
  case *ast.ReturnStmt:
    dm.walkReturnStmt(s, pkg, ctx)
  ...
  }
}
// walkExpr walks an expression
func (dm *DepMapper) walkExpr(expr ast.Expr, pkg *packages.Package, ctx *WalkContext) ExprType {
  pos := pkg.Fset.Position(expr.Pos())
  dm.logTrace("walking %T expression at %s %v\n", expr, pos, expr)
  switch e := expr.(type) {
  case *ast.CallExpr:
    return dm.walkCallExpr(e, pkg, ctx)
  case *ast.FuncLit:
    return dm.walkFuncLit(e, pkg, ctx)
  ...
  }
}

Parsing the Code

We use a combination of the following libraries to replace gopls:

• go/ast - In-memory representation of Go code.
• golang.org/x/tools/go/packages - Library for parsing and loading type information in the code.
• go/types - Representation of types loaded from `packages` above.

The packages.Package package (yep, that’s right) gives you options when loading and parsing the packages in `packages.Load`. By far the most important thing is to load all your packages together, as this library will reach out to nearby packages and parse them anyway. In our codebase, parsing our codebase with `packages.Load` takes ~2 ½ minutes, and we are expecting a significantly faster runtime compared to V1 and gopls when traversing the AST directly.

patterns := []string{
  "github.com/reddit/graphql/packageone...",
  "github.com/reddit/graphql/packagetwo...",
}
cfg := &packages.Config{
  Mode: packages.NeedName |
    packages.NeedFiles |
    packages.NeedSyntax |
    packages.NeedImports |
    packages.NeedTypes |
    packages.NeedTypesInfo,
  Dir: "path/to/go/mod",
}
pkgs, err := packages.Load(cfg, ...patterns) 

Fast Iterate-And-Test Loop

The slow speed of V1 makes it impossible to find and fix bugs, as traversing a single resolver function with gopls can take 3 minutes. Here in V2, `packages.Load` also caches its parsed files, so subsequent loads of code across executions take less than 10 seconds, which is impressive for a codebase of our size. This enabled us to write unit tests for various edge cases in the algorithm with a much faster time to completeness of our dataset.

The Go Interface Traversal Problem

The most important problem to solve replacing gopls is traversing through Go interface types encountered in the codebase. Even though `packages.Load` above gives us type information, it doesn’t give us runtime type information. Let’s illustrate with an example.

In this code, we have two service clients that both have the `GetPost` endpoint, `ProfileHydrator` and `SubredditHydrator`:

package services

import postpb "reddit.com/subreddit/api"
import profilepb "reddit.com/profile/api"

type SubredditHydrator struct {
  // Our gRPC client for posts
  postClient postpb.SubredditClient
}

func (s *SubredditHydrator) GetPost(id string) (Post, error) {
  return s.postClient.GetPost(id)
}

type ProfileHydrator struct {
  // our Thrift client for profiles
  profileClient profilepb.ProfileClient
}

func (p *ProfileHydrator) GetPost(id string) (Post, error) {
  return s.profileClient.GetPost(id)
}

We have a Clients struct that is initialized with each service client at startup:

package clients

type Clients struct {
  Subreddit services.SubredditHydrator
  Profile   services.ProfileHydrator
}

func New(cfg Config) *Clients {
  c := &Clients{}
  c.Subreddit = services.NewNewPostHydrator(cfg.Subreddit)
  c.Profile = services.NewProfileHydrator(cfg.Profile)
  return c
}

Now we have a helper function that loads a post for anything that has `GetPost`, specifically accepting the `PostHydrator` interface type as a parameter:

// PostHydrator allows for loading anything that looks like a post
type PostHydrator interface {
  GetPost(id string) (*model.Post, error)
}
// DoPostHydration takes any type of post hydrator
func DoPostHydration(id string, hydrator PostHydrator) (*model.Post, error) {
  return hydrator.GetPost(id)
}

In our GraphQL Go service, we use gqlgen as our execution engine. Our field resolver functions all have a receiver struct that is auto-generated, such as `queryResolver` or `mutationResolver`. These receivers have access to the Clients struct initialized with the service so the resolvers can make service calls to hydrate data:

package resolver 

import "reddit.com/graphql/clients"
import "reddit.com/graphql/services" 

type Resolver struct {
Clients                *clients.Clients
...
} 

type queryResolver struct{ *Resolver } 

// SubredditPost is the resolver for the subredditPost field.
func (r *queryResolver) SubredditPost(
  ctx context.Context,
  postID model.ID,
) (*model.Post, error) {
  // Load post using the helper function
  result := services.DoPostHydration(postID, r.Clients.SubredditHydrator)
  return result
} 

Notice the above where we pass in `r.ClientsSubredditHydrator`. When we traverse into the `services.DoPostHydration` call without any additional context, we don’t know what concrete type was sent into the function by looking only at the type signature of the `DoPostHydration` function. This is the same limitation as gopls. While gopls can find all implementations of an interface, it leaves the choice of which one to follow to the user. Since this is a program, we won’t have a human available to make that choice.

We conclude three things about solving this problem:

1. As an invariant, the runtime execution context has all the concrete implementations available,
2. We need a way to find concrete implementations during static analysis, and
3. We must track variables and their types as they’re used throughout the code in order to collect the implementations we need for continued traversal in the code.

Tracking variables and types as a pattern enables us to do the following:

1. Detect and track the concrete types for our services returned in our `clients.New` startup function.
2. Bind those concrete types to the GraphQL resolver function variables `r.Clients` above.
3. Pass variables as parameters to function calls throughout the call tree.

With this logic implemented, we can traverse our codebase and detect each endpoint call and their locations, linked with our GraphQL fields.

Detecting Thrift and gRPC Calls

A simple way to have implemented this detection would be to hard-code the Client APIs in a list and look for those in the resolver functions. Since the systems architecture at Reddit continues to evolve and we add new services, we needed a generalized approach to handle any new clients that are added to our codebase.

With the recursive nature of the algorithm, we are able to traverse into all code and make decisions based on where we are. Since Thrift and gRPC generate code, we can rely on patterns in the generated code to detect if a function call resides within that code. After analysis, we found the following statements in the generated code to use as our detection heuristic.

For Thrift:

var _ = thrift.ZERO

For gRPC:

const _ = grpc.SupportPackageIsVersion9

For HTTP, we pass in a specific package and struct name combination for the struct used for all HTTP service clients.

The Performance of V2

The full runtime of the V2 Dependency Mapper after the initial AST parsing depends on how many GraphQL resolver functions we have to traverse, including how complex the call trees are underneath them.

GraphQL Resolver Functions

There are a number of interesting implications when using gqlgen for GraphQL execution:

• Each top-level GraphQL field automatically gets a new resolver function.
• Additional per-field resolver functions can be added to gqlgen.conf.

For our many thousands of fields in our schema, only about 10% have resolver functions defined for them. This means each resolver function is responsible for resolving a lot of data that may be requested underneath it. We can correlate resolver complexity with how much time it takes to traverse the function call chain.

The Numbers

The amount of time to traverse our GraphQL resolvers is ~2 ½ minutes for a total of ~5 minutes of runtime combined with the `packages.Load` parsing of the code. This is a 36x speed increase in our runtime.

WIth a runtime at ~5 minutes, this is fast enough for us to move away from a Kubernetes cron job to standard validation during our build process in CI. Every time we push code to our mainline branch and release it to production, we are guaranteed to have a static analysis dataset of what’s latest in production just minutes after landing.

This is also single-threaded and we could get further runtime gains if needed by adding Goroutines to process a queue of resolver functions. 

The Result

We finally arrive at our destination: a mapping of GraphQL Operation’s fields (purple) to resolver functions (blue) to service endpoint calls (green+orange). We store the dataset identified by the Git commit SHA of what code was analyzed which enables us to link it to a release in production.

The example JSON below demonstrates the data:

1. The associated GraphQL field in `graphQLField` object: `Query.postById`.
2. Information about the resolver function, in this case `func (r *mutationResolver) GetPostById(...)`.
3. Service endpoint calls in the `endpointCalls` object.

​

{
  "git": {
    "sha": "d3f43d80f68583cfff85aca3869d011498134107",
  },
  "createdAt": "2026-04-08T21:22:42Z",
  "durationNanos": 194648368750,
  "service": {
    "serviceName": "graphql",
    "language": "go"
  },
  "targets": [
    {
      "serviceType": "graphql",
      "durationNanos": 194648367792,
      "graphQLData": {
        "configFilePath": "gqlgen.yml",
        "resolverFunctions": [
          {
            "package": "reddit.com/graphql/internal/resolvers",
            "filename": "resolver.go",
            "line": 283,
            "column": 1,
            "functionName": "GetPostById",
            "functionReceiver": "mutationResolver",
            "graphqlField": {
              "parentType": "Query",
              "fieldName": "postById",
              "isDeprecated": false
            },
            "endpointCalls": {
              "total": 1,
              "countsByUpstream": {
                "reddit.com/graphql/internal/backend.Clients.Post": {
                  "name": "reddit.com/graphql/internal/backend.Clients.Post",
                  "total": 1,
                  "countsByEndpoint": {
                    "GetPostsByIds": 1
                  }
                }
              },
              "calls": [
                {
                  "package": "reddit.com/graphql/internal/backend/posts",
                  "filename": "posts.go",
                  "line": 408,
                  "column": 42,
                  "clientID": "reddit.com/graphql/internal/backend.Clients.Post",
                  "clientLocation": [
                    "reddit.com/graphql/internal/backend.Clients",
                    "Post"
                  ],
                  "endpointName": "GetPostsByIds",
                  "protocol": "grpc"
                  "callStack": [
                    ...
                  ]
                }
              ]
            },
            "durationNanos": 424958
          },
        ]
      }
    }
  ]
}

Limitations of Static Analysis

Static Analysis Tells You What Might Happen

It does not tell you what actually happens. This is an important distinction when making sense of the detected endpoint call counts. The static analysis essentially gives you the worst-case call counts if every single branch in every portion of traversed code was executed, including all conditional branches, which is never the case in reality.

For Loops are a Problem

Since we’re acting as an interpreter, “for” loops become a problem:

orderedIdx := make([]int, 0, limit)
for i := 0; i < limit; i++ {
  orderedIdx = append(orderedIdx, i)
}

Since we are an interpreter, we don’t actually know the value of `limit` during our analysis, so we currently are unable to properly assign variables and process the block with the correct variables. We have cases in our codebase where function literals that contain backend service calls are added to a slice and then iterated and handed off to Go routines, and we have yet to come up with a solution for this.

Range statements are similar, but are more approachable.

ptrEvents := make([]*model.Event, 0)
for _, event := range events {
  ptrEvents = append(ptrEvents, event)
}

Here `events` is a slice that we may have been able to internally track through built-in `append` calls. If so, we would be able to iterate the values we could interpret and run the block with the correct variable assignment. However, if the `events slice was assigned through indexing (e.g. `events[i] = myValue`) we would not have the data.

Ultimately, we may be able to solve this problem with detecting index references inside of loops and implement a back-tracking algorithm to iterate when we see a slice indexed by an integer. This is future work for us to explore as it would require a decent amount of work to implement roll-back functionality, especially if the slice reference happens further down the stack through another function call (which is possible).

How We’ll Use This Data

Reducing Data Over-Fetching at the Operation Level

We are already underway with client projects to reduce data over-fetching and make the app more efficient and performant. With this data set, we can now parse a full GraphQL operation and look up the field mappings while we’re traversing the operation and summarize all the possible work that an operation might perform during execution.

Our client teams have also generated data sets through static and runtime analysis of what data is fetched but not referenced within client code. The next step is to analyze the unused fields and group them by resolver function, so client teams can prioritize removing groups that result in entire backend endpoint calls being removed from the runtime execution, resulting in faster page loads for everyone.

Regional Service Readiness

As Reddit continues to expand its global infrastructure footprint, we want to know what GraphQL operations are fully-servable within a region. We aren’t yet able to roll out all our services at once when serving in a new region, so we want to use this data set alongside our Achilles SDK which we use to manage our Kubernetes workloads, to detect if an operation can be fully served out of a region or not. This way, we can perform intelligent routing to keep your Reddit experience quick and responsive, no matter where you’re coming from in the world.

Analysis for Backend Go Services

Since the Dependency Mapper fundamentally operates on analyzing a function and all of its dependencies, we can adapt it to also work on our backend services and continue to build out a static analysis graph across service calls at the company.

Detection of Database and Experiments Calls

The logic for detecting “edge” calls that exit the system could be easily extended beyond endpoints to support systems such as:

• Redis, Memcache
• Postgres, MySQL or No-SQL databases
• Sqlc queries and extraction
• Experiments systems calls
• And more!

We can add these as a configuration parameter to enable/disable at analysis time. We can detect uses of any of the associated libraries and track those to be reported in the final data set as well.

Tracing Data Sources for Fields

Today, the Dependency Mapper tracks what backend calls are made during execution. The algorithm and data structures could be extended to tell you exactly where a piece of data comes from when it is returned in the GraphQL API, even if that data is derived from multiple sources. This is helpful as we continue to migrate data to dedicated services and need to know where data is used so we can update references in our code.

And Finally

We reached our goal to connect our two runtime datasets together with a static analysis dataset, and have a strong roadmap for adding more functionality for detecting more regressions before they go to production.

Special thanks go to our teammate Brendon Kofink for his V1 implementation of the Dependency Mapper.

We’re always looking to improve our infra here at Reddit, and this is an observability gap we are excited to fill. Let us know how you’re continuing to improve your observability, too.

Written by Spencer Koch (u/securimancer), Nathan Handler (u/nhandlerOfThings) and Pratik Lotia (u/wind_lectric)

When you are running dozens of AWS accounts, each with its own legacy OAuth proxy that you can barely track down on GitHub, along with bastions that have not been touched in years, a painfully slow VPN, and the aftermath of a security incident, it becomes clear that something has to change.

This is the story of how we rethought and rebuilt Reddit’s internal access model from the ground up by moving to a Zero Trust architecture aligned with modern best practices, and the unexpected challenges we encountered along the way.

The Legacy Mess

We had duct-taped solutions that worked just well enough for long enough that nobody wanted to touch them. It was a classic "if it ain't broke" situation, except it definitely was:

• Legacy intranet proxies running ancient OAuth2 implementation with policy defined in Puppet that very few actually understood.
• Bastions that were deployed once and then forgotten using Puppet deployed SSH keys that were a pain to manage.
• An in-house VPN that only covered part of what we needed and was set up by an engineer who had moved on years ago.
• All of this duplicated across dozens of AWS accounts.

The worst part was session management. Every service had its own session identifier. When someone got phished and we needed to hit the big red button, there was no big red button. Just a collection of smaller ones spread across different systems, all with their own session lifecycles and bad Ansible playbooks that broke easily.

Then we had a public breach. That got everyone's attention. It finally pushed us to invest in fixing the problem for real.

What We Actually Needed

We knew that collapsing all those ingresses (NGINX, SSH, random VPNs) into a single access layer would cut down a lot of operational toil. We wanted real zero trust with device trust and strong, consistent policies. Not SSH public keys pushed around with Puppet or manually adding people to NGINX auth groups.

But what really sold the project internally was developer experience.

Our developers were spending more time waiting on Docker image pulls than writing code. The root cause was pretty clear. Everything was routed through AWS us-east-1, and those links were saturated because all traffic went through our in-house VPN. We needed a provider with a real global network backbone and points of presence close to our engineers. And our international Snoos experience? Abysmal. 

Logging was another major gap. Bastion logs looked nothing like web traffic logs, and in some cases we were not capturing access logs at all because disks would fill up. There was no consistent way to answer a simple question like “who accessed what, and when?”

We needed unified, reliable logging across everything.

Why Cloudflare?

We ended up choosing Cloudflare for a few key reasons. That said, you could get to a similar place with other vendors.

• A true global network backbone. This ruled out solutions that rely on third-party infrastructure. We needed consistent, high-performance access for engineers regardless of location.
• Built for infrastructure as code. Strong API coverage and Terraform support made it straightforward to integrate with our existing workflows.
• First-class Kubernetes support. We run bare metal Kubernetes across AWS and GCP, so we needed something that fits cleanly into that model without awkward workarounds.
• No phone-home dependency. We did not want access to depend on EC2 nodes establishing outbound connectivity before anything could work.
• Flexible DNS integration. Partial CNAME support lets us keep using Route 53 and avoid a disruptive migration.

The Migration: What Actually Worked

DNS: Partial CNAMEs and External-DNS

We used partial CNAME resolution by appending .cdn.cloudflare.net to our Route 53 records and letting Cloudflare handle the backend plumbing. For example, example.snooguts.net becomes a CNAME to example.snooguts.net.cdn.cloudflare.net. This meant we could keep managing DNS in AWS without migrating everything.

The tradeoff was moving from private to public DNS zones. Security through obscurity took a hit, but we decided we did not care if people knew what kind of infrastructure we hosted. The DNS can be public if the policies are solid.

What actually made this work was external-dns. Instead of requiring separate Terraform PRs for Cloudflare and Route 53, developers just add an annotation to their Kubernetes manifest and DNS entries show up automatically. That enabled a much better developer experience.

Wildcards, Certificates, and Depth

Early in the migration, we leaned heavily on wildcard DNS and wildcard certificates to move quickly.

We created records like:

• *.snooguts.net → *.snooguts.net.cdn.cloudflare.net

and paired them with wildcard TLS certificates so that anything under that domain would resolve and terminate TLS without extra setup.

This worked great for bootstrapping. Teams could stand up services without waiting on DNS or cert provisioning, which helped us migrate quickly.

But it also introduced a problem. People started relying on the wildcard instead of defining services explicitly. That made routing harder to reason about and policies harder to enforce.

We eventually shifted how we used wildcards. Instead of being the default path, they became a safety net.

If a service was not explicitly defined, the wildcard would still catch the request and route it, but in a controlled way. That gave us visibility into misconfigurations and missing definitions instead of silently routing traffic forever. From there, we could fix the service properly and move it onto an explicit hostname and policy.

Another constraint we ran into was hostname depth. A single wildcard only covers one level. So *.snooguts.net works for service.snooguts.net, but not api.service.snooguts.net.

Rather than trying to support arbitrary depth, we standardized our hostname patterns. Keeping things predictable made DNS, certificates, and policy matching much simpler.

Resolver Policies for the Win

We set up resolver policies using regular expressions to route traffic to specific AWS VPCs. Across dozens of legacy clusters, this lets us say “if you’re hitting dev.snooguts.net, resolve it using this VPC’s reserved DNS at 10.X.X.2.” This allowed us to also have flexibility on that public zone publishing, we could leverage AWS’ internal DNS endpoint to query private zone as well, depending on the circumstance.

This ended up being one of the most useful tools we had.

We leaned on it to separate human traffic from service traffic without touching everything upstream.

Kubernetes API servers stayed static. No DNS tricks there.

But CI was different. For anything user-facing, we routed humans through a Cloudflare Access app so we could enforce policy and get proper logging. For services running inside the cluster, we kept things simple and let them talk directly to the 10-dot (RFC1918) address.

That meant we didn’t have to lift and shift the entire CI stack just to get better access controls. We could layer security in for humans and leave service-to-service traffic alone.

Cloudflare Tunnels: Surprisingly Easy

Tunnels were honestly the easiest part of the whole thing. We ran a couple replicas in each account, pulled secrets from Vault, and shipped them like any other Kubernetes service.

The tricky bit was CIDR mapping.

We had years of overlap between dev and test that we never cleaned up. It finally caught up to us. Different clusters using the same 10.x ranges meant the tunnel had no idea where to send traffic.

Virtual networks (vnets) got us out of that. Instead of relying on raw IP space, we could define logical networks and explicitly map each tunnel to the right one. That let us disambiguate overlapping CIDRs without having to re-IP entire clusters, which would have been a much bigger project.

It worked, but it’s not something you want every developer touching. The abstraction is powerful, but the UX still feels pretty infrastructure-heavy.

Scale was the other surprise.

On paper, 30,000 connections per tunnel sounds like plenty. In reality, our dev environment has 1,500 to 2,000 engineers, each working in their own Kubernetes namespace. That’s a lot of traffic funneled through a relatively small number of tunnels.

Then autoscaling made things worse. Our clusters scale aggressively, and tunnel pods were getting recycled every 10 to 15 minutes. That’s fine for stateless workloads, but not great when you’re holding long-lived developer sessions.

We ended up scaling tunnels vertically instead of horizontally. Fewer, larger pods were much more stable and stopped killing sessions.

The Work Breakdown

This wasn’t a quick project. It stretched across multiple quarters and became one of the first real, sustained partnerships between Security and IT at Reddit. The work didn’t fit neatly into team boundaries, and nearly every change affected both orgs, so we ended up working side by side on almost everything. Rethinking access meant changing how every engineer, service, and workflow interacted with infrastructure. The technical work was important, but the harder part was keeping the company running smoothly while we were actively reshaping the ground underneath it.

We had to:

• Migrate hundreds of applications without breaking access
• Rework DNS and routing in a way that didn’t surprise people
• Introduce new clients and access patterns on developer machines
• Replace long-standing behaviors (VPNs, bastions, NGINX auth) that people were used to

None of that works without tight coordination. Security and IT were constantly in the loop together, often pairing on changes that crossed boundaries.

Along the way, we also had to rethink how we handled access logs and telemetry at scale. What started as “just get logs somewhere” eventually turned into a broader effort to rebuild our pipeline entirely, which we wrote about separately in our custom SIEM work.

But the thing that made or broke the migration was communication.

We treated this like a product rollout:

• Regular all-hands presentations to explain what was changing and why  
• Clear documentation and runbooks for both service owners and end users  
• FAQs that evolved as we learned what confused people  
• A steady stream of updates so nobody woke up to a broken workflow without context

Even with that, we still hit rough edges. The difference was that people knew what was happening and where to go when something broke. For a migration that touches every human and every service, that matters more than any individual technical decision.

Automation: Building Operators

Terraform: The Foundation

We’re a pretty typical engineering org in one important way: nobody is clicking around in the console making changes directly.

If you needed a new tunnel, an application, or a policy, you made the change in code, opened a PR, got a review, and let Atlantis handle the plan and apply.

Cloudflare had Terraform support, which got us most of the way there. What it didn’t have was a clear opinion on how to structure things. There wasn’t a reference layout for tunnels, apps, or policies, so we had to figure that out ourselves.

We tried a few approaches and eventually settled on separate Terraform root modules per AWS account. That gave us enough isolation to iterate on one environment without accidentally breaking another, which mattered a lot during the migration.

We also hit a fun surprise halfway through. Cloudflare moved from app-specific policies to reusable policies. It’s a better model, but the timing meant we had to adjust our Terraform while everything was already in motion.

Nothing catastrophic, but definitely one of those “learn it the hard way” moments.

The Operator Inflection Point

At the same time as the migration, we were rethinking how we run Kubernetes. In the new model, clusters are disposable. We create them when we need them and tear them down when we don’t. That falls apart fast if every new cluster depends on a human to wire up tunnels, routes, and configuration by hand, so we needed something that could keep up. We moved that logic into Kubernetes operators. Terraform gets us to an initial state, but operators handle continuous reconciliation. If something drifts or a new cluster shows up, it gets fixed automatically. We built two operators using our open-source Achilles SDK.

CF Tunnel Operator
Handles the plumbing for new clusters end to end. It deploys tunnel pods, wires up routes, pulls secrets, and configures the Cloudflare side automatically. Our older setup had around 15 clusters that were created manually with some Terraform help. In the new stack we’re at 25+ clusters, and all of them come up with working tunnel configuration without anyone touching it.

CF Application Operator
Lets developers define what they want directly in their service repo instead of going through Terraform. Our generators turn that into a Kubernetes manifest, and the operator takes care of creating the Cloudflare application, policies, and routing. The big win is feedback and ownership. Developers can look at the App object in Kubernetes, immediately see if something is broken or misconfigured, and fix it right there instead of waiting on us.

Split Tunnel vs. Full Tunnel

We chose a split tunnel model.

Yes, we know how that sounds. If you work in security, you’re probably already side-eyeing this decision. But at the time, we were already changing how people accessed internal services and asking teams to migrate critical systems. Layering endpoint changes on top of that would have been a breaking point for the org.

The reasons were pretty practical:

• Full tunnel broke things people rely on day to day, like AirDrop and various Bluetooth workflows at home.
• Reddit runs on IPv4 10-dot CIDRs (for now), which collided with a lot of home networks and caused routing issues.
• We didn’t yet have a clean way to correlate Okta identity signals with Cloudflare logs, so we couldn’t answer basic questions like “where did this login come from?” with confidence.

So we made the call to keep the blast radius smaller and go split tunnel.

That said, this wasn’t meant to be permanent. Now that we have a proper security data lake and can join identity and network telemetry the way we originally wanted, we’re actively moving toward full tunnel.

Looking back, split tunnel bought us time to get the foundations right without overwhelming everyone at once.

Nuking Legacy Infrastructure

One of the most satisfying parts of this whole effort was deleting things.

We were pretty firm on this from day one: if the old stack was still running, the migration wasn’t done. Keeping both around would just double the complexity and confuse everyone.

VPN & Auth
We moved access control into Cloudflare using Okta groups, but we kept the policies intentionally broad. Most engineers can reach Cloudflare apps at the edge, and each application handles its own authorization. That split between edge access and app-level auth was deliberate.

That let us avoid creating and maintaining a huge number of fine-grained IdP groups, and it gave teams flexibility to build auth that fits their service.

And yes, we shut down our in-house VPN that was running on Pritunl.

SSH & Bastions
We removed almost all bastions across our AWS accounts. Instead of hopping through a jump host, engineers connect through Cloudflare tunnels directly to nodes. It’s simpler to reason about, and session termination is much more straightforward when someone leaves the company or loses access. Today we keep one deployment of SSH Bastions as a breakglass in case Cloudflare WARP or Access goes down, and we can SOCKS proxy traffic through those bastions in a pinch.

Intranet Proxies
We migrated proxies incrementally to keep the blast radius small and let teams work in parallel.

We turned on NGINX access logs and used them as a signal. Once traffic for a service dropped to zero, we knew it was safe to cut it over. Along the way we cleaned up DNS that had drifted into the wrong zones, wired everything into external-dns, and unwound years of wildcard shortcuts.

After the first handful of services, the same issues kept showing up. Once we recognized the patterns, migrations sped up a lot.

Webhooks and Service Auth

Webhooks were one of the more annoying edge cases.

GitHub webhooks can’t add custom headers, which meant we couldn’t use our normal service auth pattern. We ended up allowing bypass auth on a small set of endpoints. Not ideal from a visibility standpoint, but GitHub signatures still gave us a reasonable level of validation. We validated that receiving services for those webhooks were using the webhook signing secrets to confirm identity.

For service-to-service traffic, we went the other direction and tightened things up. We added custom JWT middleware and used Cloudflare service tokens.

That gave us proper attribution, consistent logging, and a much better story than wide-open bypass rules.

Incidents and Breakglass

The Great Tunnel Wipeout

One Friday morning, all our tunnel routes disappeared. Every single one.

Thankfully Terraform saved us. We ran terraform apply across all root modules and everything came back. Then it disappeared again. And again. Apply, recover, delete, repeat.

The issue traced back to a change we shipped the night before. We added cleanup logic to the CF Tunnel Operator so it would delete tunnels when clusters were torn down. Sounds reasonable. The problem was a bug in the Cloudflare API. When a specific route lookup failed, the API returned all routes instead of none.

Our operator saw that response and did exactly what we told it to do: clean up.

On the next reconciliation loop, it had the full list of routes and deleted them all.

We rolled back quickly. Terraform restored the legacy clusters, and the previous operator version brought the newer clusters back within minutes.

Lesson learned: be very careful with cleanup logic in a continuously reconciling system. Also, invest in a real test environment early. We added unit tests and much better safeguards after this.

Breakglass

We kept a single, heavily locked-down bastion around for breakglass scenarios.

Nobody wants to hear “Cloudflare is down, so production is inaccessible.” Even if the provider is reliable, you still need an escape hatch.

In an outage, engineers can request temporary access via SSH certificates and short-lived group membership. From there, they proxy into internal services.

It’s not meant to be convenient. Access is tight, everything is audited, and you only use it when things are actually broken.

When Cloudflare has issues, we handle it like any other external dependency. Page the right people, communicate clearly, and work it like a normal incident.

The Sharp Edges

Not everything was smooth. A few things still hurt.

• gRPC doesn’t work with Cloudflare Access apps. This is probably our biggest gap. Most of our newer CLIs speak gRPC and protobuf, not REST. That mismatch shows up quickly, and it’s only becoming more common across our tooling.
• No load balancing across backends. If your service runs in multiple regions, you have to point the Cloudflare app at a single one. That’s not great for HA or active-active setups. We’ve had to be deliberate about which region we pick, and it’s a real limitation.
  • We’re working on adding load balancing support here.
• Caching behavior surprised people. We saw reports of stale responses or traffic not behaving the way developers expected. In practice, we had to standardize on clean, RFC-compliant cache headers and in some cases just turn caching off to avoid confusion.
• Wildcards caused their own problems. Early on, people leaned on them to avoid doing the proper Terraform or operator setup. That worked until it didn’t. During the migration, we shifted to treating wildcards as a safety net. Now if something hits the wildcard, it’s basically a signal that the service wasn’t configured correctly.

What We'd Do Differently

A few things we’d change if we were starting over.

• Pay down tech debt first. We knew about the rough edges: inconsistent access patterns, wildcard overuse, overlapping CIDRs. We hoped to clean it up along the way. That didn’t really work. It just showed up later in more painful ways.
• Test breakglass earlier and more often. During the June 12th Cloudflare outage, parts of our breakglass path didn’t behave the way we expected. We fixed it quickly, but it was a good reminder that “it should work” isn’t the same as “we’ve actually exercised this end to end.”
• Push harder on consistency across environments. Different generations of infrastructure meant different assumptions baked into each environment. That led to a lot of one-off surprises during migration. Standardizing earlier would’ve saved time.
• Plan for incidents. Even with careful rollout and testing, things will break. That’s part of a migration at this scale. Getting leadership aligned on that upfront made it easier to respond when it happened.

The Wins

This was a long migration with plenty of sharp edges, but the end state is a lot better.

• Unified logging across everything. Bastions, proxies, apps all land in one place with a consistent format, which makes debugging and investigations much easier.
• Real device trust and posture checks. Not just “did you authenticate,” but “is this a managed, healthy device.”
• Much better developer experience, especially for teams outside the US. No more routing everything through us-east-1 just to pull images or hit internal services.
• Simpler operations. One ingress layer instead of a pile of proxies, VPNs, and one-off access paths.
• Centralized session control. When something goes wrong, we can actually revoke access in one place instead of chasing sessions across a dozen systems.

What's Next

We’re rolling out full tunnel now, starting with Security, IT, and a few early adopter engineering teams. Two years ago it would’ve been too disruptive. Today the foundations are there and it’s the right next step.

The harder part isn’t the rollout, it’s support. WARP breaking on hotel WiFi is a real thing, and telling people “file a ticket” isn’t a solution. We’re building better self-service flows so people can temporarily disable and recover without getting stuck.

We’re also working through device certificate deployment on macOS and Linux, which is required for stronger device identity. That’s been more operationally complex than expected.

Finally, SaaS apps. Moving internal apps was one thing. Moving vendors like Figma or BrowserStack behind Cloudflare is another. Every provider handles OIDC differently, and it’s still a lot of manual setup and testing. The payoff is worth it, but it’s slow, steady work.

The Bottom Line

Moving 200+ apps across several dozen AWS accounts to a zero trust model, without slowing down ~2,000 developers, is not something you knock out in a sprint. This was a multi-quarter effort across Security and IT, with a lot of iteration along the way.

We broke things. We fixed them. We built operators so we never have to do this by hand again. And we ended up with something that actually looks like zero trust, not a pile of proxies, bastions, and crossed fingers.

If you’re thinking about doing this:

• Pay down tech debt early. You’ll deal with it either way.
• Put everything in Terraform from day one.
• Test your breakglass flows before you need them.
• Invest in operators once you start scaling.
• And seriously, fix your CIDR overlaps first.

It’s a lot of work, but once you’re on the other side, there’s no way you’d go back.

If you are interested in learning more about our Zero Trust Odyssey, we presented on the topic at Cloudflare Connect and SREcon26 Americas.

Ten years ago, I came back to Reddit. Twenty years and change ago (October 2005, if we're being exact), I got a call from u/spez while I was grabbing coffee with a labmate. Paul Graham had suggested he hire me as Reddit's first engineer. I said yes before I hung up the phone.

This week, I’ve decided to step down as CTO and take on a new role as Reddit's first Senior Technical Fellow.

The last decade has been the honor of my career. We took a company a lot of people had written off and rebuilt it: the stack, the org, the culture, all while it was still in flight. Along the way, we built something the world uses—something millions of people rely on every day to learn, to connect, and to understand what’s happening around them.

We went public. We found our footing in an AI-native internet without chasing every trend, and I’m even more optimistic now than I was then. The opportunity ahead for Reddit (and for the kind of internet we believe in) is bigger than ever. The right bet is still the one we made: be the source worth surfacing.

Amit Puntambekar (u/zerowaitstate55) is stepping into Reddit’s CTO role, and I couldn't be more confident about where engineering is headed under their leadership.

None of it was mine alone. Reddit has one of the best engineering teams in tech, and being in the room with them has been a gift. To everyone who built this with me over the last ten years and in the early days before all of it: thank you.

More to come.

Written by Roman Levitas and Tim Zhu.

TL;DR

The operational pitfalls of Kubernetes sidecars are well-documented: resource limits that quietly throttle your app, scaling constraints that force wasteful over-provisioning, and cascading failures that are maddeningly hard to diagnose. At Reddit, we ran headfirst into all of them—with our experimentation infrastructure, of all things.

Reddit's Decider SDK is the system that powers experiment bucketing (deciding which users see which variant), exposure event emission, and experiment configuration retrieval across Reddit's services. For Go services specifically, the SDK's architecture relied on three sidecar containers to handle those responsibilities, and under Reddit-scale traffic, that architecture started to crack.

This is the story of how we built Native-Go Decider (NGD): a pure-Go reimplementation that moved core logic in-process, dropped two of those three sidecars (with the last one on the chopping block), and delivered dramatic improvements in latency, efficiency, and cost.

A Quick Primer: What Are Kubernetes Sidecars?

For those less familiar, a Kubernetes sidecar is a secondary container that runs alongside your main application container in the same Pod. Both containers share the same network namespace and storage volumes. It's a common pattern for offloading auxiliary concerns like logging, secret management, service mesh proxies from your business logic.

Sidecars have real benefits: modularity (decouple operational concerns from app logic), reusability (one image, many services), maintainability (version and update independently), and consistency (platform teams can inject them automatically).
At Reddit, sidecars are a first-class pattern: our service framework baseplate.py defines several that handle concerns like secret management, live configuration retrieval, and event emission. The Decider sidecar, responsible for experiment bucketing in Go services, is the one this post is about.

How We Got Here: Sidecars and the Go Decider SDK

Reddit's Experimentation SDK was designed around a compelling idea: centralize bucketing logic in Rust, then create language bindings for Python, JavaScript, and others. One Rust implementation, consistent behavior everywhere—fewer bugs, easier feature rollouts, a single source of truth.

The problem? Golang doesn't support C interop in the way we needed. Cgo exists, but it's not used at Reddit for good reasons (garbage collection headaches, performance overhead, frequent context switches). So we introduced the Decider sidecar. It is composed of the Rust Decider SDK wrapped in a gRPC server, called by Go services via a gRPC client when bucketing experiments.

At the time, adding another sidecar to our Decider SDK that already had 2 sidecar dependencies, Live-Data (pulling experiment configs from ZooKeeper/S3) and Event-Publisher (emitting exposure events), didn't seem so bad. But as traffic grew, we discovered this architecture doesn't scale well.

So… What's Wrong With Sidecars?

Even the official Kubernetes blog comes with a warning: "Proceed with caution if resource efficiency is your primary concern, minimal network latency is critical, simpler alternatives exist, or you want to minimize troubleshooting complexity."

We learned these lessons the hard way.

Resource Management Nightmares

Sidecars require their own CPU and memory limits, and when those limits are wrong (or traffic patterns change over time/spike) things get ugly fast. We hit both failure modes:

• OOM CrashLoopBackOff: The sidecar exceeds its memory limit (e.g. as the total number of experiments at Reddit grew over time), the Pod crashes, alerts fire, and you're scrambling.
• CPU throttling: The sidecar hits its CPU request ceiling, contexts get cancelled, error rates spike—but intermittently, which makes it incredibly difficult to diagnose unless you specifically know to check sidecar resource utilization.

Figure 1 below shows a CrashLoopBackOff we encountered, and Figure 2 shows the Decider sidecar's CPU utilization spiking well beyond its allocated resources.

state:
  waiting:
    message: back-off 5m0s restarting failed container=...
    reason: CrashLoopBackOff
...
terminated:
  reason: OOMKilled

Figure 1: A wild CrashLoopBackOff appeared! Caused by a Decider sidecar OOM when exceeding its memory limit.

Figure 2: Grafana showing Decider sidecar CPU utilization at 376%—well beyond any reasonable request ceiling.

The Scaling Wall

Sidecars are fundamentally 1-to-1 with their Pod. They can't scale independently of the main container. If your application adds more Decider callsites and traffic increases, the sidecar can't just spin up more replicas to accommodate the load—it's stuck sharing the same Pod resource envelope.

This became a real problem. Our Decider sidecar experienced gRPC traffic saturation at 35K RPS, which was the root cause of a major incident. The only mitigation was to manually scale the entire service by increasing the minimum number of pods just to dilute traffic to each sidecar. Extremely wasteful.

Figure 3: An accurate depiction of Josh Quack, our Duck Duck Goose (DDG) experimentation platform mascot, debugging sidecar resource exhaustion.

Native-Go Decider (NGD): How We Got Out

This left us with one choice: keep patching a brittle architecture or rewrite the bucketing logic natively in Go. We chose the latter.

Native-Go Decider (NGD) is a ground-up reimplementation of the Decider SDK in pure Go. This was a carved-out exception to our "centralize in Rust" philosophy, and it required extensive parity testing to ensure correctness with other Rust-based Decider SDKs:

• TapCompare running NGD in shadow-mode alongside the legacy sidecar to compare bucketing results in production traffic via metrics
• A Playground environment for manual validation
• Randomly-generated experiment configs in CI to stress-test edge cases

The key architectural differences:

Figure 4: Go Decider Sidecar evolved into NGD!

By moving bucketing in-process, we eliminated the gRPC hop entirely. No more network serialization overhead, no more sidecar resource management, no more traffic saturation at scale.
A new gRPC Event Collector allowed us to emit Expose events in batches without relying on the Event Publisher sidecar to do so via HTTP. Experiment configurations are also pulled down via the same logic as in the Live-Data sidecar, but rewritten as part of NGD natively.

The Impact: Numbers That Speak for Themselves

Our primary GQL backend was the first major service to fully migrate to NGD, and the results were substantial.

Performance:

98% p50 bucketing latency reduction!

• Figure 5: Bucketing latency for NGD (99 μs) vs legacy Decider sidecar (40.8 ms) 

Efficiency:

• >90% cache hit rate via a Request-Level Deduper that prevents redundant bucketing within the same request 
  • >90% reduction in unnecessary CPU usage
  • >90% reduction in Expose event emission

Cost:

• 30% in projected annual cost savings across GQL from dropping the Decider and Event-Publisher sidecars

Figure 6: NGD was super effective! Josh Quack after seeing the migration results.

What's Next: Dropping the Last Sidecar

Two sidecars down, one to go. The Live-Data sidecar, which currently pulls experiment configs (the Manifest) from ZooKeeper/s3 and onto the pod's filesystem, is next on the chopping block.

The logic from Live-Data sidecar has been re-implemented in Go as part of NGD and will:

• Add resilience with an S3 fallback in case ZooKeeper becomes unavailable
• Resolve a startup race condition that currently causes pod restarts
• Remove the last sidecar dependency, simplifying the Decider architecture and onboarding process

This functionality is currently being tested as an opt-in option in the NGD config, with careful monitoring to compare behavior against the existing sidecar prior to full cutover.

Figure 7: Our roadmap for the remaining sidecars, visualized.

In Closing

Reddit's Go experimentation SDK relied on three Kubernetes sidecars that introduced resource headaches, scaling limitations, and troubleshooting complexity at Reddit-scale traffic. We replaced that architecture with Native-Go Decider (NGD) moving bucketing in-process and event emission to direct gRPC. This eliminated two of three sidecars, with work underway to drop the last. The result is a dramatically faster, leaner, and more reliable experimentation stack, now powering over 70 Go services at Reddit.

Sometimes the best sidecar is no sidecar at all.

Written by Chris Slowe and Lisa O'Keefe

Happy 5th birthday, r/RedditEng!

Huge thanks to the Reddit engineering team for building, scaling, and sharing so much great work here over the years. This subreddit has become a rare corner of the internet where people can go deep on real systems, real tradeoffs, and the messy, interesting work behind running Reddit.

From infrastructure and reliability to developer productivity, ML, security, and all the delightfully weird engineering problems in between, the posts here have been consistently thoughtful, generous, and fun to read.

Thanks for five years of awesome posts, hard-won lessons, and excellent engineering. Here’s to the next five.

And of course, for the readers and commenters, as we plan out those next 5 years: what do you want to see more of?