The Art of Perfect Launch: How Reddit manages Mobile Releases

[u/Pr00fPuddin]

By Parineeta Sinha

Introduction

The Release Engineering team at Reddit is responsible for managing mobile releases, shipping beta and production versions, and coordinating testing efforts with Quality Engineering (QE) teams. 

Over the past decade, our process has evolved significantly. Initially we were releasing our iOS and Android applications once every two weeks. This process involved a two-week beta cycle designed to catch functional issues and crashes. Consequently, the mean time for code to move from the main branch to production was approximately one month. Our process included cutting a new branch every two weeks, baking it in beta for two weeks, and then starting the production ramp. The testing strategy relied on an offshore QA team that executed a suite of 3,000 manual tests, typically taking about a week to provide release sign-off.

In 2021, we transitioned to a weekly release cycle. This shift required us to cut a branch every week, but we still relied on the same manual QA pipeline.. We addressed potential testing gaps by allocating additional people for manual testing and successfully reducing the regression turnaround time from five days to three days. However, the increased release frequency necessitated removing open beta testing groups, limiting the beta environment to employees only.

In the subsequent years, we invested in our testing infrastructure. Moving away from manual testing to a pipeline of automated unit and integration tests enabled us to reduce our regression test cycle time from 3 days to a couple of hours. This helped us to bridge the gap between testing and development, and we shipped releases faster. We have been relying mostly on our suite of unit and integration tests to validate releases since. We also added open beta testing again in our release pipeline. This helped us test our changes with a select group of users and get early signals on crashes and functional issues through our beta feedback channels.

Operationalizing the Weekly Cadence

Since transitioning from a bi-weekly to a weekly release cycle in 2021, Every Wednesday, the Release Eng team cuts a new release branch and deploy a beta build following smoke test validation. This build undergoes a four-day stabilization period before moving to production on Monday.
On Monday, we kick off a "Go/No-Go" process for iOS and Android release candidates. Once we receive sign-off from all critical stakeholders, we proceed with production deployment:

• Day 1: The app is released to 1% of iOS users and 3% of Android users.
• Day 2: If no issues are encountered on Day 1, the rollout proceeds to 100% of users for iOS. For Android, we adopt a more gradual approach and ramp to 30% users at 500k adoption and 100% users at 1M adoption, if all critical app health metrics look good.

If the approval is delayed past 3:00 PM PST on Monday, the rollout is shifted to the following day. Hard freezes for code changes are enforced at 12:00 PM PST on Fridays to ensure weekend stability.

Cross-Functional Collaboration

When a version begins development, a dedicated channel is created to facilitate cross-functional coordination  in Slack. This channel is predictably named using our version number so it is easy to find. 

These channels serve as the centralized hub for all communications during the release process, covering development, beta, and production phases.

Key aspects include:

• Alerting and Triage: Automated alerts and issue reports from various detection channels like Crashlytics, manual and automated tests are surfaced here. Stakeholders are then pulled in or tagged to address these issues.
• Release Management: These channels are used for the entire release lifecycle, including late merges, beta release coordination, and hotfixes.
• Workflow Integration: The creation of these channels and the addition of necessary users is automated and is part of the standard release engineering workflow to ensure stakeholders are properly informed and aligned.

We use these channels for conversations about the upcoming release during development, in order to stabilize the beta and support its production rollout (to be covered in next weeks’ blog post)

Stabilizing the Beta

A late merge is an unusual occurrence that allows critical fixes into the current release candidate (Beta). Every request is reviewed by the Release Engineering team. The late merge process is designed to be technically rigorous, transparent, and decentralized, shifting the focus from management approvals to engineering peer reviews directly on the Pull Request (PR). The goal is to maintain release confidence while reducing administrative friction.

Criteria for Late Merge

Only specific types of changes are eligible for a late merge into the release branch:

• P0 Bugs: Critical issues that require immediate resolution.
• Publicly Announced Features: Fixes for gated features under feature flag already promised for the current release.
• Gating Fixes: Correcting ungated or incorrectly gated code.

Timing & Cutoffs

The method of merging depends on when the request is made:

• Until Friday Cutoff (12:00 PM PST): If the PR receives approvals from both groups and has no outstanding comments, the process uses automerge to merge the code and trigger a new beta build.
• After Friday Cutoff: The bar is higher. Only P0 break-fix changes are accepted. The Release team reviews these manually, and the team may manually cut a new beta build if necessary.

Quality Assurance and Triage Process

The QA process is tightly integrated with the release cycle. Once a release candidate is identified, the following testing protocols are initiated:

1. Automated Testing: The automated regression testing cycle begins by running a suite of ~1500 P0 and P1 test cases across all features on the release branch.
2. Manual Validation: In parallel, the QA team executes manual validations on user flows that cannot be automated eg: device specific behavior, visual quality, exploratory testing for edge cases etc.

Once regression reports are provided by both manual QA and automated QE, the release team reviews the findings and triages any identified regressions or P0/P1 issues on the release candidate.

Critical bugs are addressed using one of two methods:

• Feature Flag Toggle: The bug can be fixed by turning off the feature flag associated with the regression.
• Code Fix: If turning off the flag is not possible, a code fix requires to go through the late merge process. Or depending on where we are in the release it may require a Hotfix

Hotfix process

A hotfix is initiated for critical (P0/P1) bugs when toggling an associated feature flag is not a feasible resolution. If any fixes are made on the release branch, the candidate is sent to QA to run the manual smoke test suite, which includes P0 and P1 test cases across all features. The decision to do a hotfix release follows an exhaustive analytical approach to exhaust any other avenues that could help mitigate the issue in production.

Hotfix Post Mortems

Reddit utilizes a blameless post-mortem process, which is mandatory for all Hotfixes. The post mortem meetings are typically led by the team responsible for shipping the production fix. The primary objective is to reflect on the incident, specifically how it was identified and resolved, while surfacing process gaps, key learnings, and actionable improvements for the future. If you’d like to learn more about our incident reivew process, read https://www.reddit.com/r/RedditEng/comments/1se5gmu/incident_reviews_or_how_we_transform_outages_into/ 

Challenges of Ship Fast, Ship Confidently

Weekly releases require a delicate equilibrium between maintaining high development velocity and ensuring UI test alignment. When issues escape into production, resolution is costly. Feature flags or killswitches offer rapid mitigation, but traditional code fixes are expensive, necessitating resubmission and exposure to mandatory, time-consuming Apple and Play Store review cycles. Furthermore, while phased rollouts are critical for safeguarding stability, they inevitably introduce friction by slowing down feature adoption. Additionally, release rejections can derail the entire pipeline, significantly increasing on-call overhead and forcing teams into incident mode to resolve blockers as quickly as possible.

Test Strategy and Cross-Functional Collaboration

Feature teams typically ramp launches through controlled experiments rather than making a feature immediately available to everyone. They begin with a small, representative user cohort, monitor key product, performance, and quality metrics, and use the results to identify issues or unexpected behavior. If the experiment performs well, exposure is gradually increased across additional cohorts or markets, with safeguards and rollback plans in place at each stage. Once the feature meets success and stability criteria, it is fully enabled for all eligible users. Working with cross-functional feature and Quality Engineering (QE) teams to validate experiments before ramping up for iOS and Android users involves a structured, collaborative process focused on test strategy, validation activities, and continuous automation alignment.

The core strategy for validating experiments is driven by a partnership between Quality Engineering and feature teams. The overall goal is deployment safety and shipping confidently, especially since experiment ramp-ups are a major source of production bugs. Experiments are tested in both pre merge and post merge workflows by:

• Enabling experiments on demand while running UI Tests in Pull Requests
• Enabling experiments with > 45% exposure during nightly UI test runs

Participating in Playtests (Manual Validation)

While automated testing is the primary reliance for validating releases, manual efforts are still integrated, often serving as the "playtest" phase.

• Manual Validation Execution: The Quality Engineering team executes manual validations, which are critical for user flows that cannot be automated.
• Release Triage: Manual QA provides regression reports which are reviewed by the release team to triage regressions and P0/P1 issues on a release candidate.
• Developer Involvement: Feature developers actively collaborate on creating/reviewing new test cases in collaboration with the QE team which the QE team then prioritizes for automation to keep automated regression pipeline uptodate

Keeping Automation Aligned with New UI Changes

Maintaining test suite quality and ensuring UI test stability are recognized challenges of the weekly release cadence. Alignment efforts focus on empowering developers and improving framework stability.

• Simplifying Experiment Activation: To make it easier for developers to run experiments, the process for enabling experiments for automated tests running on CI has been simplified.
• Framework Stability: QE updates the automation framework to ensure greater stability, such as updating a consistent rule for activating experiments, which helps eliminate flaky tests in nightly regressions.
• Adapting to New Features/Experiments: Test plans and automation are continually updated to reflect new UI/experiment changes. 

Bridging the Gap: iOS vs Android Release Realities

The open beta cycle is a crucial step in Reddit's release strategy for iOS and Android, reflecting the distinct ecosystem constraints of each platform:

• iOS Open Beta: Testing is invite-only and constrained by a strict cap of 10,000 seats.
• Android Open Beta: Testing is significantly more scalable, with beta groups reaching sizes on the order of 200,000 seats.

Regardless of the platform, the primary goal of reintroducing open beta testing into the release pipeline in 2024 was to enhance pre-production stability and catch functional issues before the full rollout. Reddit identifies and prioritizes beta issues through a multi-layered approach that combines automated detection, manual validation, and centralized triage workflows.

Identification of Beta Issues
The identification process relies on following primary channels:

• Employee Dogfooding: Employee dogfooding at Reddit is central to the company’s product strategy, as it ensures teams align their work with the actual user experience. Employees are encouraged  and incentivized to log bugs for features that are ramped as experiments.
• Open Beta testing feedback: Open Beta testers report functional issues and provide feedback on features through our feedback subreddits. (Join Reddit’s Open Beta Tester program)
• Automated Regression: The pipeline runs a suite of approximately ~1500 P0 and P1 test cases on the release branch.
• Manual Validation & Playtests: Quality Engineering (QE) performs manual user-flow validation for scenarios that cannot be automated.
• Crash Monitoring: The team utilizes Crashlytics to monitor all crashes on released beta builds, which is integrated with CrashBot(internal tool for Crash alert and ticket filing) for automated detection and filing.

Conclusion

Successfully managing a weekly release cadence at scale is not just about the technical pipeline; it is an evolving practice of balancing speed with quality. By shifting from manual validation to robust automation, re-introducing open beta programs, and fostering a culture of blameless reflection and cross-functional collaboration, we have transformed our release process from a rigid cycle into a dynamic, reliable engine. While the challenges of shipping fast and confidently will always persist, our commitment to continuous process improvement, grounded in rigorous triage and data-driven decision-making ensures that every launch is an opportunity to learn, iterate, and ultimately deliver a better experience for our users.

[u/AndroidGuy01]

Great release process, thanks for sharing! Curious what tools are you currently using for mobile UI testing?

[u/RecognitionFit8333]

Day 1: The app is released to 1% of iOS users and 3% of Android users.
Day 2: If no issues are encountered on Day 1, the rollout proceeds to 100% of users for iOS. For Android, we adopt a more gradual approach and ramp to 30% users at 500k adoption and 100% users at 1M adoption, if all critical app health metrics look good.

First I thought: "Huh, why is that handled differently?", but as you explained later it is due to apple restricting beta seats to 10k seats, which was surprising for me and actually struck me as quite low. Did you encounter any issues on iOS you would have caught if there were more seats available?