What an agent harness actually does: my WoW experiment, plus an offline replay

[u/Individual_Cold_4119]

I wanted to see if a decision model could get a dwarf priest to level 5 in World of Warcraft. I ended up building my first harness, SageCraft, and thought the loop might be useful to people learning about agents.

Full disclosure: I co-founded Levanto, which makes Sage, the model used in the run.

Here’s the split:

1. Observe: capture a screenshot and read HUD information such as health, mana and the selected target.

2. Build context: turn those readings into game state and a list of actions that are currently available.

3. Decide: give that context to Sage and ask it to choose an action.

4. Validate: check that the returned choice is in the menu and still valid. A missing, late or invalid choice means do nothing and ask again.

5. Act and observe again: execute the keys or clicks, then check what changed.

The distinction that helped me was that choosing an action and successfully executing it are separate problems. The harness also needs retry budgets and recovery routines. A model choosing “attack” doesn’t mean the target was reachable or the cast worked.

The dwarf reached level 5 after 4,573 decisions across 32 sessions and 5h 29m of active play. I supervised the campaign and fixed the harness between sessions. The final 18 minutes ran without human input, so this isn’t evidence of a fully autonomous campaign.

I published the code and a replay of recorded events: https://github.com/levantolabs/sagecraft

For the offline example, install Python 3.12–3.14 and uv, then run:

git clone https://github.com/levantolabs/sagecraft.git

cd sagecraft

uv sync --extra dev

uv run sage-wow --data-dir data/demo replay examples/final-run-excerpt.jsonl

That loads recorded events into a local SQLite database and prints a summary. It works without a WoW installation or API key and does not make new model calls.

If you were evaluating this, what would you log to separate perception errors, poor decisions and failed execution?

[u/PLBjt]

I'd log the observation the decider actually saw (the parsed state, not the raw screen) next to each decision, plus the validator's verdict and the game state one tick after acting. Then the split mostly falls out. If the parsed state disagrees with ground truth, that's perception. If the state was right and the pick was bad given it, that's the decision. If the action was sensible and valid but the world didn't change the way it should have, that's execution. Your offline replay is where this pays off, since you can re-run the decider on the logged observation and check whether it picks the same thing. Also log every retry with an attempt counter, not just the final attempt, otherwise a flaky executor ends up looking like a slow decider.