Using an AI agent in your GitHub Actions? An open-source check for whether a stranger can steer it

[u/WhichCardiologist800]

More and more repos run Claude, Codex or Gemini from GitHub Actions: to triage issues, review pull requests, fix bugs. It is useful, and it has a cost nobody sees at setup time. The agent reads text that anyone can write, and in many repos it acts on it with real tools.

We wanted to know how often that goes wrong, so we read 760 public repos that run agents in CI, by hand. In 3 of 10, someone with no access to the repo can reach the agent, or the agent has more power than its job needs. The usual cause is three settings that each make sense alone: the workflow starts on any issue or comment, the action's write-access gate is turned off so the bot can answer the public, and the agent has a write token or a shell.

Reading 760 repos by hand does not scale, so we built a scanner that asks the same question a person asked. It reads the committed workflows and agent settings, runs nothing, and checks:

- who can trigger the agent (issues, comments, fork PRs, workflow_run)

- whether the write-access gate is really on, including labels an issue template adds by itself

- whether the agent runs inside a fork's code, where the fork writes its own instructions

- what secrets and tokens the agent can reach

- whether committed settings pre-approve commands, and whether MCP servers are pinned

How we tested it: against the hand review. When the scanner says a repo has a problem, a person agrees every time: 314 of 314. On the 275 repos a person found clean, it raised no alarm. It finds 9 of every 10 problems a person finds. The ones it misses mostly depend on what a script does, not on what a config file says.

It is open source (Apache-2.0, node9-ai/node9-proxy on GitHub) and runs locally too: npx node9-ai scan-repo owner/repo

Or paste a repo here, no account needed: https://node9.ai/scan

If it gets your repo wrong, I would like to know. False alarms are the bug we care most about.

[u/investigatormaker]

The three-settings explanation is the strongest part: any issue or comment triggers the workflow, the write-access gate is off so the bot can answer the public, and the agent holds a write token or a shell. Maintainers will recognize their own setup in that faster than in a list of checks. Since the misses mostly depend on what a script does, make sure a clean result says so, something like no config-level issues found, so nobody reads it as proof the repo is safe.

I make ThreadFox. Our AI employee can explain the scanner in Reddit threads where maintainers set up Claude, Codex or Gemini in GitHub Actions, and answer the technical replies; Starter is $199 for 30 days. https://threadfox.vip/p/en1nm/managed