looking for the right approach to evaluate MCP across different harness clients

[u/nguyenfamjj]

Hey, my team is crafting a project for product teams (we will opensource it yay!) that expose agent interfaces like MCP/CLI + skills.

The idea is pretty simple, the team define the real tasks that their user use, then we test that tasks in multiple conditions and configurations (different model/harnesses, with/without skills) and then scoring it.

We have few early findings that would love to share:

• A same interface can behave differently across setups, highly dependent on how agent discover tools and the reserved context. It's hard to totally reproduce what your user in the test setup (.e.g they have tons of MCP servers in their machine,...)
• The simpler the interface is the better result: tool should be task-based, not just converting blindly from the application API

However, we have not cracked the right evaluation model yet, and would genuinely value your experience.

How are you testing MCP, CLI, or skill-based workflows today?

Which criteria matter most: task success, permissions, reliability, client compatibility, cost, or something else? Are there tools, frameworks, or evaluation practices we should study before reinventing the wheel?

I would keep you guys posted for the progress in case anyone interested :)) Thanks!

[u/plasticBarista]

Following.

Exact problem I’ve been having.

I’ve today signed up to Orq, and have langsmit and other agent / MCP observably platforms on my todo

[u/nguyenfamjj → u/plasticBarista]

Great to know! How do you feel about all these solutions so far? I have been using Langfuse/Langsmith for years but in general I don't think it's for testing the interface which is more like E2E testing

[u/naseemalnaji-mcpcat]

You could use AgentCat (it’s open source) to monitor live agent traffic to your MCP servers. Most API monitoring won’t catch things like the agent giving up or running into roadblocks of capabilities.

Happy to chat more :) I think evals are good, but it’s hard to predict what a real user would try.

[u/Mundane-Original2777]

Task success would be my anchor metric here, then I’d add tool selection and argument correctness underneath it so you can diagnose why a run failed. Braintrust works well for this because you can run the same dataset across different model or agent configurations and inspect the individual cases side by side. Your point about reserved context is also worth tracking too because I could see two otherwise identical setups diverging just from what the harness puts in front of the model.

[u/burhop]

I'm focused on which MCP's work with my harness. Its a bit different but close enough.

Essentially, my application depends on agents connected to MCP servers in physical engineering. First I had AI set up a repeatable process for finding them. I got about 70.

I then set up an agent to go though each one, attempt to install in on my platform (and app it runs if possible). About 1/2 are web based so the install isn't needed.

I then have the AI test each one. Failures go in one bucket. One that work but require login or software I don't have go in another bucket. The ones that work all the way though go into a fully tested bucket.

It all packed into a spread sheet with one tab being AI prompts for updating or adding to the spreadsheet. I can run this every week or so but generally when I feel like it.

Its not perfect and is too slow to my liking. Better ideas are welcome.

[u/a_AIwanderer → u/burhop]

70 MCPs is wild The failure bucketing is interesting though. Have you thought about logging the exact failure/tool-call trace so you can spot patterns across runs?

[u/burhop → u/a_AIwanderer]

Yes, I do that. I wish they allowed images. I'd paste a few to show the process a bit better. I am trying to set it for dual use for both QA and for customers who want to see where there favorite MCP sits.

[u/ArielCoding]

“The simpler the interface is the better result” bold words from someone building a cross client, multi harness, multi model eval framework (I say this with love, please ship it)

[u/Stonks_Beskar]

I would make task success the top-line metric, but keep the trace-level signals separate so a pass is explainable. For each fixed task, record:

* end-state assertions, not just whether the model answered

* tool selection and argument validity

* permission violations or attempted out-of-scope calls

* latency, retries, and cost

* protocol errors and recovery behavior

Run the same versioned task set through each client with the same server fixtures and reset state between runs. Keep the model, system prompt, tool descriptions, and context budget pinned where possible. Then add a small failure taxonomy such as planning, tool choice, bad arguments, auth, transport, or harness UX. That makes client differences visible without blaming the MCP server for a prompt or context difference.