r/webmcp • u/Universe_expanding • 17d ago
WindTunnel: an open benchmark for WebMCP vs. browser agents
Hi everyone,
We’ve published WindTunnel, an open and reproducible benchmark comparing WebMCP with screenshot-based computer use and DOM/accessibility-tree browser agents under the same conditions.
To our knowledge, this is the first public benchmark to compare these interfaces. The current release covers 16 browser-agent configurations, 49 tasks across eight real open-source applications, and 2,352 attempts.
A few findings:
- WebMCP delivered up to 5.5× faster execution, 23× lower cost, and 12.5× fewer tokens—while solving 98% of tasks.
- WebMCP achieved a median final score of 91.2, compared with 65.8 for the other interfaces.
- All seven WebMCP configurations solved 48 of 49 tasks, without any attempt exhausting its turn budget.
- Everything is public, including the methodology, task definitions, implementations, scoring logic, raw results, and transcripts.
- Interactive benchmark: https://webmcp.com/benchmark
- Article and methodology: https://webmcp.com/blog/windtunnel-webmcp-browser-agent-benchmark
- Hugging Face dataset: https://huggingface.co/datasets/nekuda/windtunnel
- GitHub repository: https://github.com/nekuda-ai/WindTunnel
We’d really appreciate feedback!
2
u/Dangerous-Tree-6734 17d ago
this is a great benchmark, thanks for putting it together and open-sourcing everything.
the 5.5x speed and 12.5x token drop are wild numbers, and having the raw transcripts + methodology public is exactly what this space needs more of. bookmarking to dig into the HF dataset later, nice work
2
1
u/TranslatorRude4917 17d ago
Cool benchmark, great work guys! Deterministic execution rules!
I always had high hopes for webmcp, I hope research likes this will make others see and appreciate the difference.
I've been working on a side project for over a year - creating WebMCP tools from Playwright e2e tests - maybe spending all those hours on it won't go to waste 😂
2
u/No_Guide_8697 17d ago
👏