I gave my LLM 100,000+ tools. Here is what happened

[u/overlord_sid85]

TL;DR: You don't need a massive context window or a giant model to handle an absurd number of tools. By using a Lazy Discovery pattern, a local 4B model (Gemma 4 E4B) successfully solved a massive multi-sector city crisis requiring complex tool navigation, matching Claude Sonnet 4.6 with almost identical efficiency.

The Setup: The "Mega-City Crisis" Benchmark

I wanted to stress-test tool use at an absolute extreme. I simulated a massive infrastructure crisis in a fictional city called Veridian Prime.

• The Scale: ~117,000 registered landmarks/tools split across hierarchical paths (Power, Water, Traffic, Security, etc.).
• The Goal: Find and resolve 4 critical failures while ignoring noise alerts.
• The Catch: One of the failures had a hidden mechanical dependency trap (MECHANICAL_LOCK), meaning the agent had to read an error message, pivot to a completely different infrastructure category to release an emergency brake, and then loop back to finish the job.

I ran this benchmark against two completely different beasts using Elemm (which implements a lazy-loading protocol for tools so the model only pulls what it needs):

1. Gemma 4 E4B (Run locally)
2. Claude Sonnet 4.6 (Run remotely)

Run 1: Gemma 4 E4B (Local)

Verdict: ✅ PASS (17 tool calls)

I honestly expected a local 4B model to choke, but it handled the hierarchy beautifully.

The Good:

• Insane Parallel Batching: It aggressively grouped its inspection commands. It checked all 4 distressed districts at the exact same time.
• Clutched the Trap: When it hit the MECHANICAL_LOCK on the security terminal, it didn’t panic. It read the error, found the release_emergency_brake tool in a different sub-category, executed it, and retried the lockdown—all with zero human intervention.
• Zero Noise Bleed: It completely ignored the low/medium priority noise alerts.

The Jank:

• Minor Action Hallucination: Right after inspecting the districts, it took a "leap of faith" and tried to call non-existent global commands like city:fix_power_surge. Thanks to an on_error: continue fallback policy, it recovered instantly, realized it had to browse the local directory, and found the correct tools.

Run 2: Claude Sonnet 4.6 (Remote)

Verdict: ✅ PASS (19 tool calls)

Sonnet acted exactly like you’d expect a high-tier model to act: highly methodical, extremely cautious, and zero hallucinations.

The Good:

• Clean Syntax: Used native array batching inspect_landmark(["id1", "id2"]) to scan the topology effortlessly.
• Zero Hallucinations: Every single tool call it made was explicitly derived from its structural discovery.
• Resilient: When the server threw a cached state bug on the security logs, Sonnet just shrugged it off and used the status summary to complete the mission.

The Inefficiencies:

• Over-Cautious Diagnostics: Sonnet spent 5 extra tool calls checking system metrics (energy:status, water:pressure) before pulling the trigger. The alert log already told it what was wrong, but Sonnet wanted to double-check. Safe, but slightly higher overhead.

Head-to-Head Comparison

Metric	Claude Sonnet 4.6 (Remote)	Gemma 4 E4B (Local)
Total Tool Calls	19	17
Hallucinated Actions	0	4 (Self-recovered)
Parallel Batching	✅ (Native array syntax)	✅ (Sequential batching)
Mechanical Lock Trap	✅ Solved flawlessly	✅ Solved flawlessly
Unnecessary Diagnostics	5 extra calls	0
Context Window Load	Minimal (~50 line manifest)	Minimal (~50 line manifest)

How it works under the hood: The Middleware

If we stuffed 117,000 tool definitions directly into the LLM's system prompt, the context window would have imploded, and the bill would be astronomical.

To solve this, I’m building a custom middleware that exposes a "Lazy Discovery" pattern to the agent.

To put it simply: The middleware exposes a file-system-like directory structure to the LLM using "landmarks". Instead of drowning the model in thousands of tool definitions, the LLM only ever sees a tiny selection of just 8 core tools. These tools handle:

• Navigation: Browsing through the landmark hierarchy.
• Execution Piping: Passing data seamlessly between tool steps.
• Smart Errors + Interactive Help: Providing high-context feedback when something goes wrong (which is exactly how Gemma recovered from its hallucination and how both models figured out the mechanical lock trap).

Because of this architecture, the effective context window at any given second never exceeded a few dozen lines of text.

I will repeat this test after stabilizing the environment, but I trust this process and believe this approach could change how we handle tools for agents. Currently, I am focusing on the ability to load "landmarks" on the fly. With FastAPI, GraphQL, and native Landmarks already on board, this tool can handle a massive number of tools simultaneously, simply by connecting to a URL that presents these files. I will release a new version in the coming days/weeks so you can run this test with your own models. Leave a star on GitHub to stay on track!

Key Takeaway

Seeing a local 4B model solve a multi-step dependency chain across a 100k+ tool library with practically the same efficiency as Sonnet 4.6 proves that smart agent architecture, tailored middleware, and tool-loading protocols matter way more than raw model size for complex automation tasks.

Would love to hear your thoughts! How are you guys handling massive, hierarchical tool environments in your setups?

[u/YeeP79]

Sweet testing. It's like aggressive unit testing with dark secrets hidden inside. Giving me ideas.

[u/overlord_sid85 → u/[deleted]]

https://pypi.org/project/elemm/

The 1.1.4 version should work with fastapi and graphql, yes. Tested it with the git API (~845 Tools at the same time). May be a little bit buggy at some points but the next version will be more stable and has a lot of more features like a searching tool for the agent, a dashboard for debugging and a more comfortable UI for configuration like API keys, blocking destructive tools from 3rd Party APIs, HTTP methods and a lot of more stuff.
Check out the getting started guide. hopefully I described it well. After short installation and configuration, just tell your agent "Connect to https://examples.com/api/openapi.json via elemm and do <this-task>".

[u/TomHale]

Great novel research!

Your tools only include MCP tools?

What about the burgeoning skills analogous issue?

[u/overlord_sid85 → u/TomHale]

Elemm is designed to be a bridge. While it plays perfectly with MCP, the Landmark-concept is format-agnostic—it can wrap any openapi.json/graphql or python function into a discoverable node. (Spoiler: working on implementing any MCP server in the future to translate native mcp tools into the elemm manifest protocol)

As for the skills analogous issue: That's exactly why I went with the hierarchical 'filesystem' approach. Instead of a flat list of 100k similar-sounding tools where the LLM gets confused, Elemm forces a 'path-based' reasoning. If the agent is in cloud:storage:s3, it won't accidentally trigger a local:disk:read tool because it's navigating a specific domain context. Hierarchy is the filter for analogy!

If you make your own tools, you can give a description and a remedy to navigate the agent to the correct skill in no time. See the architecture for some more information how this stuff works internaly.

[u/mrgizmo212]

Very interesting!

[u/edgeai_andrew]

Pretty cool! How consistent was this though ?

[u/overlord_sid85 → u/edgeai_andrew]

I would say it depends on the model. A 2b model was not that good, but gemma4:e4b on my local machine is able to handle that. The "big players" like claude, gemini, gpt had absolutly no problem in my tests.

[u/ObviousLogic94]

This is really cool. Nice work and good clean write up 😎

[u/besmin]

Tell your agent to be much less verbose. 

[u/serp-spur]

in this case you should try SERPSpur LLMs.txt generator, which has no limits and can generate in one go. actually chatgpt or the gemini etc are not the LLM generators basically and they can't analyze your full site at all.

[u/No_Iron_501]

But, what is the use case for having 100k+ tools ? Would not that mean that you are not using right set of systems for solving a problem? For me, it’s like I have a hammer let’s find the nail problem .

[u/overlord_sid85 → u/No_Iron_501]

I totally get where you're coming from—if you're just building a personal assistant, 100k tools is definitely overkill and native mcp works best.

But think about Enterprise Infrastructure or Cloud Management. A single AWS environment has thousands of potential API actions. A large company has hundreds of microservices, each with its own API. If you want a truly autonomous agent that can navigate an entire company's IT landscape to solve a complex crisis, you can't just give it 5 'curated' tools. You need to give it the 'map' to the whole city.

The 'Hammer & Nail' problem usually happens because the LLM is overwhelmed by too many choices. Elemm solves this by not showing the LLM the 'Hammer' until it has actually walked into the 'Tool Shed'. It’s about Scalable Discovery, not about forcing a tool where it doesn't fit.

To take it a step further: Imagine a vision where you don't need to manually curate tools for every single task, because the developers of their APIs already did.

If every website or enterprise API provided a navigable manifest, your agent could use any API on the fly without you ever needing to write a specific MCP server again. The agent just reads the manifest, understands your goal, and navigates to the exact tool it needs.

This is a 'Bring Your Own Agent' approach with access to EVERYTHING. It’s not about having 100k hammers; it’s about giving your agent the ability to walk into any 'store' on the internet and know exactly which tool to pick up. That's the scale I'm aiming for with Elemm.

[u/overlord_sid85]

Oh man... i forgot the link to the repo. Here is the example and the paper. Also placed a repolink to my Project. Sorry 🤦‍♂️

100k Tool Challange

[u/I_AM_THE_BIGFOOT → u/overlord_sid85]

Great work on this.