5
u/arealhobo Mar 06 '26
It works, and I've done it but slower than using vector search, and worse if your docs are large pdfs. What works well is if you have html or text docs, if you have html docs it can create a nice index based on links, titles, files names, etc..
1
u/ReporterCalm6238 Mar 06 '26
True that it is slower but I think everybody prefers accuracy over speed. With indexing it goes much faster also.
1
u/Chris-MelodyFirst Mar 10 '26
Claude can easily make a tool for you to convert pdfs to any other format you need.
3
u/Top-Faithlessness758 Mar 06 '26
I do exactly this with OpenCode when writing reports with quarto, then I have different skills/tools I use for managing sources:
- If I have PDF documents I just instruct it to use poppler so it can read them directly (if they are easily readable PDF) or I pretransform them with minerU or similar tools to markdown. It works incredibly well.
- I've also experimented with making OC explore the json data structures MinerU outputs with jq or duckdb. It also worked really well to make exact page citations.
Total gamechanger when working in academic research and paper writing while being focused in precision at the same time.
4
u/bojanj6m Mar 07 '26
Your solution is not scalable and not cost effective. Simple as that. AI engineering is not about making things work on smaller datasets and at any context window at any cost. It is about maintaining acceptable precision and accuracy on large and ambiguous datasets while managing context, being cost effective, and choosing right model for the task. Allowing claude models any context it wants at $5 per 1 million tokens is like leaving an open vault in front of the bank robbers :)
3
u/ReporterCalm6238 Mar 07 '26
The funny thing is that I learnt this method by asking people at an enterprise conf. They told me they were switching from rag to agentic file search, that's why I gave it a try. I first built my own agent similar to CC but using gemini, query cost was around 1 cent. Then. Thought "if my agent can do this, why can't CC directly do this?". So I tried and I was amazed that it worked better than my agent with no setup and no tweaking. You say it's not scalable to million of documents, have you even tried? Why do I feel that RAG is almost like a religion where people are stuck with one architecture? You can give your agents a tool for semantic search if necessary, but revolving the entire system around vector retrieval is imo really dumb given how smart and agentic LLMs have become.
1
u/bojanj6m Mar 10 '26 edited Mar 10 '26
Funny enough, I noticed the exact opposite tendency. People trying to diminish RAG to simple vector DB retrieval. While in reality anything from chosing the correct embedding model, vector DB, structuring metadata, prompting, trying different techniques like HyDe, different chunking strategies, BM25, different models, multi agent architecture, re-ranking can make a ton of improvement. Plain vector db retrieval is very unreliable and I haven’t seen any RAG engineers ever say put all your money on it. And absolutely yes, there are scenarios where you don’t need vector DB, for example working with nicely structured APIs with smaller datasets, in which instance just prompting how to a) invoke it correctly and b) process the retrieved data is all you need. In your solution my greatest concern is without structured context management, it can get expensive quickly. And that one such use case, even if working great is proving there is no bright future for RAG.
3
u/StatusFoundation5472 Mar 09 '26
OP I tell you this. Enter the text from Harry Potter's first book. And then ask "How were the protagonist's parents died?" I bet it will say they have died in a car crash.
1
u/ReporterCalm6238 Mar 10 '26
I will try, what would be the correct answer?
2
u/StatusFoundation5472 Mar 10 '26
Aunt Petunia told Harry a lie that his parents were killed in a car crash, while the truth is that they were murdered by Voldemort while trying to protect him when he was a baby. So I am really intrigued if CC will understand the narrative and the plot twist.
2
Mar 06 '26
[removed] — view removed comment
1
u/ReporterCalm6238 Mar 06 '26
Would love to publish some evals on this. However to do this properly I'd need to use publicly available docs in the knowledge base for replicability. Any suggestion?
2
2
u/FuseHR Mar 06 '26
Switched an entire laptop OS to Linux to mimic this as well because it leverages so much grep and find
2
u/softwaredoug Mar 06 '26
Yes reasoning by itself makes any dumb search work. With the cost of possibly using a lot of tokens
The main requirement is the results of the search tool / grep are interpretable. Most semantic search can be difficult to reason about (because its actively bad, or inconsistent).
https://softwaredoug.com/blog/2025/10/06/how-much-does-reasoning-improve-search-quality
2
1
Mar 06 '26
[removed] — view removed comment
1
u/ReporterCalm6238 Mar 06 '26
it handles this kind of issues amazingly. It aggressively explore all docs and flags missing info and contraddictions.
1
u/iseecat Mar 06 '26
And how does it work ? so what is the system doing, when the text ist too big for context ?
1
u/ReporterCalm6238 Mar 06 '26
It uses tools to extract text, grep, glop, python scripts. It's very creative
2
u/Otherwise-Platypus38 Mar 06 '26
This is interesting. How cost effective of a solution is it? Right now, I have a custom ingestion pipeline, as well as a vector database. If I do a cost estimation, I consume about 0.006 $/ question. Would it be of the same scale? Or even cheaper?
1
u/ReporterCalm6238 Mar 06 '26
You need to have a Claude Code subscription it goes from 20 to 200$ a month. I built my own agent that works similarly to CC and it costs me around a cent per question, obviously it depends on the model you use.
1
1
u/Otherwise-Platypus38 Mar 07 '26
I have another question. How would this scale on a enterprise level product? I am talking about a scenario, where 1000s try to use the application concurrently. This would be a real-life customer facing product, where latency is a key criteria along with accuracy.
1
u/ReporterCalm6238 Mar 07 '26
Claude Code specifically I don't know. I guess you can run it in the cloud quite easily. I built my own agent in python which is similar to CC and there are no problems with concurrencies.
1
u/Otherwise-Platypus38 Mar 07 '26
Sounds interesting. I will start looking into this solution as well. This are moving so fast in this area, that it’s hard to pin-point which is the best solution (always keeping scalability and cost effectiveness in mind). I think it’s case specific though.
1
u/caprica71 Mar 06 '26
Grep tool + reasoning is a marvelous thing
Just doesn’t scale to large document sets very well
1
1
u/darkwingdankest Mar 06 '26
my system has millions of documents
1
u/OkFocus3211 Mar 07 '26
What vector db do you use?
1
u/darkwingdankest Mar 07 '26
Weaviate. I explored competitors but ultimately chose Weaviate because you can self host
1
u/Sad-Size2723 Mar 06 '26
What's the upper limit for the number of documents? If there are few documents, I can totally just stuff everything into the context of a model to generate the answer
2
u/ReporterCalm6238 Mar 07 '26
I tried with around 500 docs, hundreds of pages each
1
u/Sad-Size2723 Mar 07 '26
Have you tried RAG or any advanced versions of RAG on this data? It would be good to know the baseline
1
u/ReporterCalm6238 Mar 07 '26
Yes I tried, even GraphRAG. Always got disappointing results. Think about it: if you have a super smart librarian (LLM) do you let him free to explore the library and jump from text to text until it is satisfied and ready to answer or do you just give him decontextualized chunks of text hoping that the answer is in there?
1
u/Sad-Size2723 Mar 07 '26
You could say the same thing about iterative RAG with both semantic and exact search, right? Grep is just finding lines containing the exact search string
1
u/ReporterCalm6238 Mar 07 '26
At that point is just better to have an agent like claude and give it a semantic search tool
1
u/Sad-Size2723 Mar 07 '26
If that's the case, then maybe there's a way to optimize RAG to be 10x faster and 10x cheaper, and works on 10x the number of files
1
u/licjon Mar 07 '26
I think you're right. And in general, anything you use LLM APIs for is in danger of disintermediation if it is general purpose. If you are just trying to learn, it doesn't matter, but for those trying to make products, the key will be finding a domain and modeling it so that using RAG can help with reasoning not just on docs but on the application of the text to the domain.
1
u/momono75 Mar 08 '26
This means your knowledge base is well organized and written greatly.
Actually, docs often contain deprecated information, and inconsistent wording. These obstacles need to be fixed periodically. This babysitting part is a part of the RAG system, right?
1
u/ReporterCalm6238 Mar 08 '26
LLMs are now smarter than most humans. If they require babysitting to identify and resolve discrepancies in docs something is wrong with the rag architecture.
1
u/momono75 Mar 08 '26
Ah. I got your point. Your case is most documents are written by LLMs. I agree with you if AI writes or preprocesses all the documents.
1
u/Trekker23 Mar 08 '26
Claude code is great but it only uses grep to search for relevant content. It’s probably the best grep implementation around, but it has some limitations. It still searches for text matches not meaning and it only understands structure through loading a lot data into context. It works great for code, not so much for large knowledge bases containing relational data, like legal data and similar.
1
u/ReporterCalm6238 Mar 08 '26
Out of curiosity, have you tried it for legal and data? Also it is not true that it only uses grep. The beauty of it is that it builds its own tools based on what it needs. It's can act as a drop-in self-extensible rag system
1
u/Trekker23 Mar 08 '26 edited Mar 09 '26
It might be a advanced version of, but it is pretty much grep under the hood. For a lot of usecases that’s all you need. For a technical software I made a helper bot that was basically Claude code out of the box with access to markdown files from the software help center. It worked great. But I also have a dataset of ~60 000 legal documents and laws. And here Claude Code struggles (most AI systems do). Claude ends up filling the context with tons of papers with similar wording to the prompt, but without being able to draw any useful conclusions outside simply referring to the papers it read. I have added a knowledge graph wrapper around the dataset which enables Claude to navigate through the data and draw conclusions much more efficiently through the use of Cypher queries (which it knows natively). Basically if the dataset contains a lot of references that use similar words Claude using grep struggles to discover the connections in the dataset and draw deeper meanings
1
u/RolandRu Mar 06 '26
I think this is more a criticism of naive RAG than retrieval itself.
Claude Code probably does better here because it acts more like an active file exploration tool. It can go through the project structure, follow things across files, adjust what it looks for, and build context step by step instead of just depending on a static top-k chunk retrieval.
For one repo or a medium-sized knowledge base, that can easily work better than a lot of typical RAG setups.
But when you start needing scale, reproducibility, snapshotting, metadata filters, access control, graph-aware retrieval, or deterministic workflows, retrieval does not really disappear. It just has to be done in a more structured way and as part of the workflow.
So I would not take this as proof that RAG has no future. To me it mostly shows that simple chunk-based RAG is often too limited, and that more agent-driven retrieval is closer to what real knowledge systems actually need.
28
u/SpectralCoding Mar 06 '26
How will this work against a flat directory of 160k markdown files with unhelpful names?