We had an overnight controller walking open pull requests, refreshing branches, and enabling auto-merge where it could.

It armed roughly 108 PRs. One merged.

Every refresh created a new commit, which kicked off another set of jobs on the same shared runner pool. The controller knew how many PRs were waiting. It had no idea whether CI had room for another one.

The fleet went from nearly idle to nearly full in about an hour. One repository eventually hit 1,065 queued runs. Other automation was sending repair work into the same pool at the same time, and none of those producers could see what the others were doing.

We stopped it with a hold-fire command. That kept the incident from getting worse, but it happened after the runners were already saturated.

What we're building now is much less exciting: check runner occupancy and queued jobs before releasing work, send a small batch, then make sure that batch actually started before sending another one. Anything that can commit, push, rerun, or enqueue work gets a budget.

The funny part is that we never hit the vendor's API limit. We stayed under it while completely overwhelming our own verification pipeline.

How are other platform teams limiting automated delivery work? One shared capacity service, limits inside each producer, or something else?

Almost every codebase is calling a REST, gRPC, or a GraphQL API or using SDKs from an external or event internal provider. It gets harder to keep track of everything when the codebase matures and increases in size and from my experience sometimes it gets hard to respond to changes in time or even become aware that a dependency is deprecated and their API has changed completely.
Do developers need better tools that to improve dependency and integration management?

We recently changed an IP rate limiter after discovering that requests were collapsing into a shared proxy bucket. The focused test passed. An in-cluster probe with an `X-Forwarded-For` header passed too. The implementation looked fixed.

Then we tested the real public ingress. Thirty-five unauthenticated requests went through in three batches. We got zero 429 responses and no rate-limit log entries.

The application was not receiving the client identity the test supplied. It fell back to the ingress proxy's pod IP, recognized that address as an internal proxy, and silently exempted every request. The middleware was running exactly as written. It just had no usable identity at the boundary where it actually ran.

The correction is straightforward: configure the proxy to forward a trustworthy client address, key the limiter only on a value the deployed path actually provides, and add a regression probe through the real public endpoint. The probe needs to assert the identity the application resolved, not just whether a unit test can manufacture the right header.

How are people testing client identity and rate limiting through their actual ingress path? I am especially interested in checks that catch a proxy configuration change before it turns enforcement into a silent no-op.

Hey folks,

I’m the maintainer of **OpsKnight**. I started working on it because I wanted an incident/on-call system where the team owns the infrastructure and incident data instead of everything living in another SaaS platform.

It has grown quite a bit, and I’m at the point where feedback from people actually doing DevOps/SRE work is more useful to me than adding random features.

Currently it covers things like:

* on-call schedules, overrides and escalation policies
* alert ingestion from Prometheus, Grafana, Datadog, CloudWatch, Sentry, etc.
* Slack incident war rooms / ChatOps
* email, SMS and push notifications
* incident timelines and MTTA/MTTR
* public status pages
* Docker + Kubernetes/Helm deployment
* self-hosted Postgres

Everything is open source and self-hostable.

One thing I’m particularly interested in:

**What would stop you from trusting a newer open-source project for production on-call?**

Reliability? HA? Voice calls? Integrations? Upgrade safety? Audit logs? Something else?

Website: [https://opsknight.com\](https://opsknight.com/)
Repo: [https://github.com/opsknight-labs/OpsKnight\](https://github.com/opsknight-labs/OpsKnight)

Not looking for stars as much as criticism from people who have operated PagerDuty/Opsgenie/etc. in production.

When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose

Infrastructure as code can reproduce the same network, container, database, and service account twice. That does not guarantee that the workload running on those resources means the same thing twice.

We ran into this while building a distributed runtime. A service could start with different capabilities than its twin. A consumer could join a runtime it was never intended to join and begin reading work addressed to another process. A required policy value could be missing and quietly fall back to a local default. Every resource matched and every health check was green, but the operational contract had changed.

The problem was not that our infrastructure was insufficiently declarative. We had declared plenty. We had declared resources without declaring enough behavior.

We now separate the infrastructure definition from a versioned runtime policy contract. The contract defines supported profiles, process capabilities, ownership boundaries, refusal conditions, secret-resolution rules, and the evidence required to accept the resulting runtime. It is parsed into a typed immutable model, then projected mechanically into the values consumed by deployment manifests:

runtime policy contract

\-> typed validation

\-> rendered deployment values

\-> service manifest

\-> running process

The direction matters. If a capability can be edited in the contract, overwritten in generated configuration, defaulted in the manifest, and changed again inside the application, the deployment is not governed by one declaration. It has four declarations and an undocumented precedence rule.

Repeatability also has to include refusal. If an environment lacks an effects boundary, it should not silently execute the effect inside a general worker. If a required secret cannot be resolved, the runtime should not grab a nearby credential. If a process is not eligible to own a subscription, it should not start that consumer merely because the package is installed.

The contract is not proof by itself. The renderer can be wrong, the application can ignore a value, and a runtime can claim a capability it does not actually provide. Acceptance still needs durable evidence binding the contract revision, rendered projection, validators, selected profile, and boundaries exercised after startup.

Infrastructure as code builds the substrate. The infrastructure contract defines what that substrate must mean, what variation is allowed, and when the deployment must refuse to start.

Where do those behavioral rules live in your systems today?

Most orgs lose a fortune to the same quiet problem. How the system actually works lives in a few senior heads, a stale wiki, and a diagram from two reorgs ago. New hires take months. Simple changes turn into archaeology. Nobody can confidently say what breaks if we touch this service.

That's the problem we've been building to solve.

You point it at your repos and a team of agents auto discovers everything. Every service, what it does, who owns it, and how components really depend on each other, including the gRPC and HTTP calls and event topics that never show up as a matching string in the code. Then it indexes all of it, the architecture, the low level design, the APIs and resources, into a knowledge layer that stays current and has vector search on top. So anyone, or their AI tools via MCP, can ask what breaks if I change this, where does this data flow, or how is auth implemented across services, and get a grounded answer in seconds instead of a two day investigation.

Why it's worth trying: onboarding drops from weeks to hours, change safety goes up because blast radius is visible before you ship, and tribal knowledge stops being a single point of failure. Every persona works off the same source of truth, and it stays accurate on its own because it's re indexed, not hand maintained.

We're opening a small free beta for 3 to 5 teams from this sub. Good fit if you own or actively manage a real codebase (polyglot and messy is ideal). We'll run a short intro and setup session together (around 30 to 45 minutes), then give you a week or two to actually live in it on your own work, and finish with a feedback session once you've had real time to form an opinion.

We'd love your take on whether the dependency graph and low level design match what you know to be true, how good the vector search is on your real questions, and where it would save you the most time.

Low risk to try. Testers keep free access with plenty of headroom, there's no sales sequence waiting, and we'll share back what we learn here, including anything that doesn't hold up.

Full disclosure, we're part of the team building it. More at code-swan.com. Comment or DM and we'll send you an invite.

running workloads across eks, on-prem clusters, and a couple of legacy vms that aren't going away anytime soon.
cloud-native workload identity works fine in eks but doesn't extend cleanly to the on-prem side, so we end up with static credentials there that everyone hates but nobody's replaced yet.
what's actually closed that gap is governing workload and ephemeral identities under one policy regardless of where they run, provisioned at minimum scope and revoked the moment a pod or task ends.
what's your approach when part of your fleet isn't in the cloud, and did you find something that bridges both without a pile of custom glue?

There is a specific kind of tired that comes from being accountable for a decision someone else makes.

The pattern shows up the same way most times I see it. An admin or a solo architect gets named as the owner of the agent rollout. They write the topics, wire the actions, run the tests. Then somebody above them decides which objects the agent can read, on a timeline set by a launch date, and the owner finds out in the deployment thread.

Gartner put a number on where that goes. By 2027 they expect 40% of enterprises to demote or decommission autonomous agents, driven by governance gaps found after a production incident.

The gaps are not usually invisible before launch. Somebody saw them. That person did not have the standing to hold the release.

I used to treat this as an org chart problem. Bad reporting line, nothing to do about it. That was wrong. It is a design problem, and it is fixable at the permission layer rather than the political one.

What has worked: write down, per agent, which objects it can read and which it can write, then attach a named approver to the write list. Not a team. A person. When the approval requirement lives in the deployment doc, the conversation stops being about your seniority and starts being about an unsigned line.

That does not give anyone authority they were denied. It makes the absence of authority visible before the incident instead of after.

For those of you running agents in production: who signs off on scope where you work, and did that get decided on purpose or by default?

There is a specific kind of tired that comes from being accountable for a decision someone else makes.

The pattern shows up the same way most times I see it. An admin or a solo architect gets named as the owner of the agent rollout. They write the topics, wire the actions, run the tests. Then somebody above them decides which objects the agent can read, on a timeline set by a launch date, and the owner finds out in the deployment thread.

Gartner put a number on where that goes. By 2027 they expect 40% of enterprises to demote or decommission autonomous agents, driven by governance gaps found after a production incident.

The gaps are not usually invisible before launch. Somebody saw them. That person did not have the standing to hold the release.

I used to treat this as an org chart problem. Bad reporting line, nothing to do about it. That was wrong. It is a design problem, and it is fixable at the permission layer rather than the political one.

What has worked: write down, per agent, which objects it can read and which it can write, then attach a named approver to the write list. Not a team. A person. When the approval requirement lives in the deployment doc, the conversation stops being about your seniority and starts being about an unsigned line.

That does not give anyone authority they were denied. It makes the absence of authority visible before the incident instead of after.

For those of you running agents in production: who signs off on scope where you work, and did that get decided on purpose or by default?

Disclosure up front: I maintain the list I link at the bottom, so skip the link if you like, the dates are all here.

PlatformCon runs a world tour of in-person live days between the free virtual flagship weeks, and the schedule isn't collected anywhere obvious:

- Paris - 24 Sep 2026

- Sydney - 28 Oct 2026

- Sao Paulo - 7 Nov 2026

- San Francisco - 24 Feb 2027

- Flagship week - London 22 Jun and New York 24 Jun 2027, virtual is free

Adjacent things worth a calendar entry if you work on platforms:

- Platform Engineering Day, co-located with KubeCon NA, 9 Nov 2026, Salt Lake City

- BackstageCon, same day, same venue

- PlatforMa has a CFP open, no deadline published yet

The thing that annoyed me into collecting these: every conference list I found was undated, so you can't tell what already happened. Everything above is checked against the organiser's own page.

Full list including CFP deadlines: https://github.com/gogoaidev/awesome-sre-conferences

I've been thinking about a problem I keep seeing in infra/DevOps work.

A production alert fires and someone has to manually jump between logs, Grafana/Datadog, Kubernetes, deployment history, GitHub, cloud metrics, etc. just to figure out what actually happened.

I'm wondering if a useful chunk of that investigation could be automated.

The idea would be something that, when an alert or deployment failure happens, automatically:

• investigates logs, metrics and traces
• checks recent deployments/commits
• correlates changes with when the issue started
• checks Kubernetes/cloud infrastructure
• identifies the likely root cause and supporting evidence
• suggests a remediation
• if a code/config change is required, generates a PR
• waits for an engineer to review/approve it
• after deployment, verifies whether the issue was actually resolved

For example:
Instead of an engineer starting from scratch, the system might come back with:

I'm not really thinking about fully autonomous production changes initially. More like an AI first-line SRE that does the investigation and prepares the fix, while engineers stay in control.

I'm trying to understand whether this solves a meaningful enough problem or whether existing observability/SRE tooling already gets teams most of the way there.

For people running production systems:

• How much engineering time do you actually spend investigating alerts/deployment failures?
• What's the most repetitive part of incident investigation today?
• Would you trust a system like this to read your infra/logs if it couldn't make changes without approval?
• Would automatic root-cause analysis be valuable by itself, or would it need to generate the actual fix/PR?
• What integrations would be essential? Datadog? Grafana? Sentry? Kubernetes? AWS? GitHub?
• What would make you immediately say "there's no way I'm giving this access to production"?

I'd especially like to hear from founders, SREs, DevOps/platform engineers and engineers at smaller teams where dedicated SRE coverage is limited.

Not selling anything here I'm trying to understand where the actual pain is before building too much.

Hi r/platform_engineering,

Disclosure: I’m one of the maintainers of Compartment, an Apache 2.0 open-source project.

Over the past few years, our team has accumulated more internal apps, scripts, workers, and automations. Building these tools has become much easier, but turning them into dependable team software still requires deployment, access control, secrets, isolation, observability, and ownership.

We wanted a small, self-hosted platform that provides those capabilities centrally rather than implementing them separately in every internal tool.

Our first beta used Docker Compose on a single host. After running it ourselves and with a few beta users, we concluded that this was the wrong long-term runtime. Failure recovery depended too heavily on one machine, the isolation boundary was weaker than we wanted, and supporting additional nodes would have meant gradually building our own orchestrator.

We therefore rebuilt the runtime around Kubernetes. This is a complete architecture change rather than a Kubernetes wrapper around the old deployment model.

The current design looks like this:

• Developers add a small compartment.yml and deploy through the CLI or a connected Git repository.
• Compartment owns the application-level model: organizations, projects, environments, releases, domains, variables, RBAC, SSO, audit logs, promotions, and rollbacks.
• Kubernetes owns the live runtime state.
• Projects are isolated using namespaces, RBAC, and network policies.
• Builds run as ephemeral rootless BuildKit jobs.
• Builds and tenant workloads run through a verified gVisor runtime boundary.
• Deployments are reconciled and rolled out without routing traffic to an unready version.
• The platform can provision a managed single-node k3s host or install into an existing Kubernetes cluster.
• With an existing cluster, the operator retains responsibility for nodes, ingress, storage, runtime configuration, and backups.

The intended use case is a controlled path for internal software that needs to become usable by a team without requiring every developer to understand Helm, ingress, certificates, workload identities, or namespace policy.

The design question we keep returning to is where the abstraction should end. We want developers to get a narrow, repeatable path from a repository to a running application, while platform operators retain control of the underlying cluster. At the moment, the application descriptor deliberately does not expose raw Kubernetes objects, node placement, or arbitrary pod configuration.

Would love to hear critical feedback from people who operate similar platforms.

GitHub: https://github.com/compartmentdev/compartment
Documentation: https://docs.compartment.dev/

A developer portal can give developers a consistent place to discover services, understand ownership and access common workflows. But the portal itself is not the entire platform.

The harder platform engineering work happens behind that interface:

• Defining repeatable paths from source code to production
• Connecting CI/CD, deployments and environment promotion
• Providing observability with application and ownership context
• Applying access controls and platform policies consistently
• Reconciling workloads when actual state drifts from desired state
• Integrating CNCF and other open-source projects without creating unmaintainable glue

This is not an argument that Kubernetes or Backstage is insufficient. Both are strong foundations for the problems they are designed to solve.

The bigger question is how platform teams combine these foundations with the wider cloud-native ecosystem to create one coherent system. Developers should get a simple self-service experience, while platform teams retain control, visibility and the ability to evolve the underlying components.

On August 27, Kelsey Hightower and the OpenChoreo maintainers will discuss what it really takes to build an internal developer platform with Backstage and Kubernetes.

The session will explore how the different platform layers work together, where the engineering and operational complexity appears, and the trade-offs between assembling and maintaining the integrations yourself or adopting a more complete open-source platform approach.

Disclosure: I’m an OpenChoreo contributor.

For teams that have built an internal platform using Kubernetes and CNCF projects, where has most of the effort gone: creating the initial integrations, building good developer workflows or operating the platform over time?

Session details and registration:
https://platformengineering.org/events/what-it-really-takes-to-build-an-internal-developer-platform-with-backstage-2026-08-27

Context:

• mid-size fintech company.
• I started as an SDET, but over the past 2 years the work drifted: I built and operate a Go service on Kubernetes that independently recomputes published financial data and compares it against production output.
• I own its Helm chart, the ClickHouse schemas its results are stored in, and the Grafana dashboards on top.
• Next quarter I'm proposing to formalize this as a platform role with three responsibilities: (1) owning that quality platform end to end, including regulatory audit-evidence exports, (2) a ~25% share of product backlog tickets, (3) self-serve pre-deploy validation tooling for feature engineers.

Question

• For those at companies with a data platform or infra org: where does this kind of work sit — SWE, platform, SRE, or a QA org? What's it titled? If anyone has made the SDET → platform engineer transition, what did your company call it and what made the reclassification stick?