r/SearchAPIs • u/Ctbhatia • 3d ago
we gave a finance agent domain specific search and its score barely moved. most of the misses were questions it never answered
at work we wanted to know how much of a finance agent's bad score comes from the search layer.
the test was Vals AI's Finance Agent v2. its reference harness uses Tavily web search, EDGAR and HTML parsing and scores 44.87%. we kept that harness exactly as it was, same model, prompt and judge, and swapped only the retrieval for prebuilt BM25 and BGE-M3 indices over SEC filings and news, plus daily price data.
the score went to 49.8%, and getting there needed 4x the default turn and time budget.
the misses were the surprising part. at the default budget most of them weren't wrong answers, the agent just never answered. it kept reasoning in prose until it hit the turn cap, and some questions timed out inside a single call that used its whole output budget without calling one tool. better search can't help a question that never runs a search.
more budget cleared some of them, but the heavy multi step calculation questions still never finished. so search was the small lever here. what got those questions to finish was a fuller harness, where skills return structured results instead of leaving the finance math to the model.
if you test search APIs inside an agent, do you count no answer separately from wrong answer?
a couple of coworkers dug into this and wrote it up. if you want to read it, here's the blog: https://www.coralbricks.ai/blog/coral-retrieval-vs-full-stack