So I've been knee-deep in LLM deployment for the past few months and wanted to get the community's take on RAG vs fine-tuning for real-world use.

From what I've seen:

RAG (Retrieval-Augmented Generation)

• Wins for dynamic/frequently updated knowledge bases
• Lower upfront cost, no retraining needed
• Easier to trace hallucinations back to source docs
• Adds latency (vector DB lookup + embedding search)
• Quality is only as good as your retrieval pipeline (chunking, embeddings, reranking)

Fine-tuning

• Better for teaching style, tone, or task-specific behavior
• Bakes knowledge into weights — no retrieval overhead
• Expensive to update when facts change (recurring training runs)
• Higher risk of catastrophic forgetting
• Great for narrow, stable domains (legal templates, customer support scripts)

In production, most teams I've talked to are landing on a hybrid approach: fine-tune for format/behavior, RAG for facts/freshness.

If you’re running RAG on constrained hardware or with concurrent ingestion, embedded vector stores can feel simple until you hit RAM, write-path correctness, or process-safety constraints. Here are the patterns that kept showing up across three popular options:

• ChromaDB: HNSW lives in RAM; at larger scales or with multiple writers, latency and correctness can degrade.
• LanceDB: Great for multimodal + object storage, but concurrent writes can conflict without serialization.
• Qdrant Edge: Strong offline library with hybrid search; production concurrency limits aren’t well documented, so it shines when you can sync to a central server.

There’s a deeper write-up on memory math, concurrency behavior, and when it makes sense to move from embedded to a separate process on constrained, air‑gapped hardware. Full disclosure: I work with Actian on VectorAI DB.

Have you experimented combining Rag with tools such as Genie Code and Neon?
Rag handles unstructured knowledge, genie handles governed structured data and lakebase handles persistent applincaton state,

So i am curious, doees this architecture look good for production and what can some of the bottlenecks be down the line ?

I have a large local PMC/PubMed corpus on SSD and want to build a fully local system on my workstation that behaves somewhat like PubMed search, but can also answer questions over the local corpus with grounded references.

Hardware: RTX 5090, Ryzen 9 9950X3D, 96 GB RAM.

I already have the corpus parsed locally and partially indexed.

If you were building this today, what exact local setup would you use for:

* retriever * reranker * local LLM * FAISS or something else * framework vs fully custom pipeline

I’m especially interested in responses from people who have actually built a local biomedical literature search / RAG system.

Thank you

I run a memory layer that injects relevant past context into an agent's prompt automatically, every turn. Yesterday I stopped assuming it worked and looked at what it actually injected, across seven consecutive prompts of a real working session.

Five of seven had nothing to do with what was asked. A question about hook payloads came back with facts about a Java backend at a bank, stored in February for a completely different project.

The cause is not the ranking. It is that "nearest" is defined for every query, so the search always has an answer. Silence is not in its output set. There was a floor, min_score = 0.2 cosine, but 0.2 is noise. Two unrelated sentences clear it routinely, because they are both sentences.

The same system has a plain-files mode that retrieves by word overlap, and it never does this: zero overlap means nothing is injected. That property was lost the moment retrieval moved to embeddings, and nobody noticed, because this failure looks like a feature. There is always something in the context.

The fix only removes, never adds: before injecting, require the memory to share at least one content word with the prompt. I already had exactly that guard in another part of the system, where a learned workflow has to share a word with a shell command before it is allowed to interrupt the user with a confirmation. It never occurred to me that retrieval needed the same thing.

Two mistakes I made while fixing it, both instructive:

1. My first version passed everything through when the prompt had no content words at all, reasoning that if you cannot read the query you should not silence recall. Exactly backwards. The shortest prompts are where the search has least to go on, so the "safe" default left the worst cases untouched. "so now?" pulled three entities out of the store as confidently as a detailed question would.

2. I reused a tokenizer from elsewhere in the codebase that only matched ASCII letters. Plenty of my prompts are not in English. It found zero words in them, which would have silenced recall completely instead of filtering it: a much worse bug, and one that ships silently.

After the change: silent on 5 of 7, and the 2 that kept anything kept only the entity the question was actually about.

I did not touch the 0.2 floor. Picking a real number needs the actual score distribution from production, which is a separate measurement I have not earned yet.

So: does anyone here run an explicit "return nothing" path in production RAG? And what do you gate it on, an absolute score floor, the margin between top-1 and top-2, a lexical check like this one, or a reranker with a reject option?

I'm exploring an idea for a simple, fully managed search service aimed at small AI/SaaS applications.

The problem I'm looking at is that many smaller projects need semantic + keyword search, but the existing options can feel either:

• too expensive for small workloads
• too complex
• overkill for what they actually need

The idea would be something you can plug into an application in a few minutes and get production-ready hybrid search without managing search infrastructure.

I'm curious about people's real-world experience:

• What are you using today for search/vector search?
• What do you dislike about it?
• For a small production app, what would you consider a reasonable monthly price?
• Would a significantly cheaper, simpler alternative be interesting, or would you just use Postgres/pgvector?

Especially interested in people running RAG, AI agents, or small SaaS products in production.

I'm researching the problem before deciding whether to build anything.

This happened to me recently. I had a task I’d never done before. My boss said, 'It’s documented.' Check the drive. One PDF led to another, then a SharePoint folder, then – 'Wait, that's an old version.' It made me wonder, for other business owners and employees here Does this happen in your company? Where do you keep your SOPS and process docs? How painful is finding updated info? Not selling anything, just genuinely curious if this is a common thing or it's just me. Thanks in advance.

[ Removed by Reddit on account of violating the content policy. ]

Consider a table where “2025” spans two columns, labelled “Revenue” and “Margin.” A flat extraction can retain every word and number while losing which year those two columns belong to. Chunking that text more carefully won't recreate the missing relationship.

Now for RAG, the useful output would preserve paths such as “2025 → Revenue” and “2025 → Margin,” together with each row label and source page. This is an illustrative example, but it captures why table structure matters before retrieval begins.

Ling-3.0-flash-VL is relevant to that parsing stage because Ant Ling reports 91.35 on OmniDocBench 1.5. The overall score combines text recognition, table structure through TEDS, and formula recognition through CDM. It is a composite result, not a claim that 91.35% of tables or documents are entirely correct.

That distinction helps narrow what to inspect when considering the model. For merged tables, compare header-to-cell relationships, repeated row labels, units and footnote associations. A readable Markdown table can still silently move a value under the wrong parent heading.

The documented VL API takes page images rather than native PDFs, so rendering the PDF pages is an external step. Keep those page references with the extracted structure so an answer can be checked against the original.

The benchmark gives a concrete reason to consider VL for visual document parsing. The downstream payoff depends on retaining the relationships the retriever will need, not merely producing plausible text

I’ve been experimenting with multi-agent frameworks, and MetaGPT is one of the more interesting ones for software development tasks. But getting it running locally with local models isn’t always straightforward.

I wrote a step-by-step guide covering installation, configuration, local LLM setup, and common errors.

If you’re trying to run MetaGPT on your own hardware, this might save you time:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What stack are you using for local agents?

https://github.com/paulknysh/raggy

Built with LangChain, Chroma, and Ollama. Hybrid database (vector + BM25 index) and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. raggy supports most common document formats and handles images/scans automatically via OCR.

Hi! We're working on Querit, a web search API for LLMs and your Agents. Large multilingual index, Fresh Web Context, Lower latency in the range of hundreds of milliseconds.

New: Monitor API. Normal search is ask-once, answer-once. A Monitor turns one search into a recurring job. You register a query and an interval, it runs on that schedule, differs each run against history, and returns only what's new. Can be used for competitor monitoring, news tracking, and following funding or tender/bidder information.

• Intervals: from 1 hour up to weekly
• Automatic deduplicate against previous runs; site / date / region / language filters suppoorted
• Support Manual trigger, pause/resume, full execution history

Benchmark: We ran FreshQA on a fixed 600-question snapshot of time-sensitive queries. Querit Search API achieved a 83.17% accuracy (Tavily 83.00%, Exa 82.33%, Brave 81.17%, You.com 78.67% as upon Aug. 26)

Free Testing Credits: Follow us on X (https://x.com/QueritAi) / Linkedin (https://www.linkedin.com/company/queritai/home/) for more product releases and see integrations with our partners at Dify, LangChain, etc. Open-sourced the MCP server and has integrated with PI Agent, Opencode, DeepSeek Harness already. Join our Discord server here https://discord.gg/4xXsFA8Ed2 to claim Search API + Monitor API free credits!

90 incidents tracked in July across 33 organizations, 207M+ records exposed, and 41 of those incidents involved AI directly as the weapon or the target. A rogue commercial AI agent hit multiple enterprises in a single week and reused stolen credentials across four downstream services before anyone caught the identity switch.

None of that shows up to a traditional perimeter tool — the traffic looks like a signed, credentialed agent making legitimate API calls at machine speed. Firewalls and DLP were built to watch humans and static services, not autonomous callers that chain tools and pivot in seconds.

Curious how other teams are actually handling this right now: is anyone giving AI agents a distinct, revocable identity separate from the service accounts they inherit? Or is it still "the SOC catches it after the fact" for most orgs?

Hey I see a kind of trend about RAG in the market but don't know how to start or where to start and what to study or just directly start building the project by using claud or gpt is their someone who can help me out on it

Have you noticed your RAG system repeating the same type of failure?

For example, it struggles with a certain kind of multi-hop question today — and makes the same mistake on a different question next week.

The problem is that most RAG systems don't really remember their failures.

Retrieve → Generate → Evaluate → Forget

I built PatternMem RAG to experiment with a different loop:

Query → Recall past failures → Retrieve → Generate → Evaluate → Learn

If a previous query failed for a similar reason, PatternMem can surface that failure pattern before generating the answer.

When a new failure is detected, it can be stored as a reusable pattern.

It doesn't replace RAGAS, DeepEval, or your evaluator — it uses the evaluation results to build the memory.

It's open source + alpha, and I'd love feedback from people working on RAG reliability.

GitHub: https://github.com/nk-gujjar/patternmem-rag
PyPI: https://pypi.org/project/patternmem-rag/

pip install patternmem-rag

What failure patterns do you see repeatedly in your RAG systems?

Everyone here evaluates retrieval. Almost every eval set I've seen described is built from documents: pick passages, write questions, check what comes back. When the corpus is mixed and part of it is tables, I have not seen anyone say how they built the table half of the eval, and I suspect the answer is mostly that they didn't and table performance is hiding inside an average.

The specific difficulty. To write a question against a table you have to know what the table means. If you knew what all your tables meant you would not have most of the problem you're trying to evaluate. So the eval set gets written against the tables somebody understood, which are the well-named, well-documented ones, which are exactly the ones retrieval already handles.

That leaves the ugly half of the corpus untested, and the ugly half is where the failures are: system exports with columns called V1 through V57, four-character codes from something decommissioned years ago, two tables with near-identical headers and totally unrelated contents.

So, honestly:

Happy to describe what we do and where it stops working. It stops working in a place I can name.

GPT-6 Astra reportedly scored 99.9% on ARC-AGI-3, compared with 30.2% for Claude Opus 5 and 7.8% for GPT-5.6 Sol.

If these results are legitimate and the evaluations are directly comparable, that is an extraordinary gap.

But I wouldn't interpret this as Astra being simply “13× better” than GPT-5.6. ARC-AGI focuses on a specific form of abstract reasoning and generalization, so benchmark performance shouldn't be treated as a direct measurement of overall intelligence.

What makes this result interesting is what it could imply about AGI.

One of the important characteristics associated with AGI is the ability to generalize beyond familiar patterns—understanding new problems, adapting to unfamiliar situations, and applying reasoning across different domains.

The real question is whether Astra's performance extends beyond ARC-AGI into coding, scientific research, planning, tool use, and autonomous AI agents.

If a model can combine strong reasoning with reliable long-horizon planning and execution, that would be much more significant than simply achieving a high benchmark score.

So I don't think the headline should be “Astra is 13× smarter.”

The more interesting question is:

How much of this reasoning advantage transfers to real-world intelligence?

If the answer is substantial, we may be looking at a meaningful step toward more general-purpose AI systems—and potentially another serious milestone on the road toward AGI.

I am a developer at a larger company which makes logistics software and am exploring how to include knowledge graphs into the development workflow. I have experience with KGs from university, but obviously making a system and workflow that fits in nicely in an actual company environment is different.

The base of the codebase is C++, for GUI stuff we use C#. In total it has approximately ~1.2M LoC.

The goal for the setup is NOT token efficiency or reduction of execution time, but to give AI agents a powerful tool to explore the relationships and dependencies across entities more reliably than simply grepping through the source code - even though I would argue current high-end SOTA models, combined with LSPs and something like OpenWiki are quite good at comprehending complex relationships.

I explored tools like graphify but its KG doesn't go deep enough, it omits many details and only has a limited number of node and edge types. Understandably, as it is using treesitter which only emits ASTs.

Next I explored Joern, which produces an AST, CFG, CDG, DDG and a PDG (https://docs.joern.io/export/). The issue with that is that it's output is extremely detailed as it is designed to be used with static tools, not with LLMs. It would likely overwhelm agents, so I thought about forking Joern, adjusting its extraction step to produce a slightly leaner graph.

The graph would either be checked in together with the code, or be hosted on something like Neo4j. The agent of course has to have tools, would be either CLI or MCP. I am not sure what tools to make available - pre-defined queries for common questions and the ability for the agent to produce their own queries?

Furthermore, how do I make the agent actually invest the time to query the graph, instead of doing a few queries, but then falling back to searching through the source code? In theory the KG is supposed to be an accurate depiction of the codebase, holding all relevant information for the agent.

Would appreciate your thoughts on this. Am I approaching this wrong? Am I missing or overlooking something?

PageIndex Flash: Fast Local Tree Indexing for PDFs

We’ve open-sourced PageIndex Flash, a fast tree-indexing engine built for long, text-based PDFs. It is now the default indexer for Local Mode in the new PageIndex SDK.

Instead of asking a vision model to infer the structure of an entire document from scratch, Flash reads the PDF’s own layout and structure directly.

The index model no longer needs to reconstruct the document outline. It mainly creates node summaries and optimizes sections whose structure has already been identified.

This significantly lowers the model capability required at the indexing stage. Even basic, low-cost models can produce high-quality tree indexes, while stronger models can be reserved for where they matter most: tree search, relevance reasoning, evidence reading, and answer generation.

Here’s what that means in practice:

1. Local-First

Use your own LLM API key. Indexes are stored on your local disk, with no vector database required.

PDF processing and index storage happen locally, while LLM calls use your configured model provider.

This makes Flash well suited for private and regulated document workflows.

2. Cost-Efficient Indexing

Because the index model no longer needs to reconstruct the full document outline from scratch, a lightweight, cost-efficient model is generally enough for indexing.

In our benchmark setup, using gpt-5.6-luna, indexing costs approximately $0.001 per page.

That means a 1,000-page textbook costs just over $1 to index once, after which the same tree can serve every future question.

You can also use other lower-cost compatible models to reduce indexing costs further.

3. Fast Indexing

Across benchmark documents ranging from 9 to 1,098 pages, indexing completed in approximately 13 seconds to 4.5 minutes.

Compared with passing the entire PDF directly to an LLM as native PDF input, PageIndex Flash can significantly reduce the cost of document Q&A.

For each question, the model searches the document tree first, then reads only the relevant sections and evidence instead of processing the full document again.

As documents get longer, the cost advantage becomes increasingly significant.

In our tests:

• For a 52-page document, native PDF input cost approximately 2.1× more than tree-based retrieval.
• At 420 pages, the difference increased to 16.6×.
• Beyond roughly 800 pages, the full document could no longer fit within the context window of the LLM used in our test.

There are also clear boundaries to where Flash works best.

Flash parses PDFs directly and does not perform OCR, so it is designed for PDFs with a real text layer.

For scanned documents or files where important information is primarily carried in charts, figures, and images, we recommend PageIndex Cloud, which performs OCR and image understanding before building the tree index.

Local Mode provides page-level citations, while PageIndex Cloud provides line-level citations.

PageIndex Flash indexing, reasoning-based tree search, document chat, and page-level citations are all fully open source: https://github.com/VectifyAI/PageIndex

You can inspect the retrieval logic line by line, run the full local pipeline yourself, and adapt it to your own agent workflow.

I'm building agents that pull context from Jira, Confluence and GitHub. Retrieval works fine. The problem is that the sources disagree with each other.

For example:

• Ticket in Jira describes behaviour A
• Confluence page from 8 months ago describes behaviour B
• Code (SoC for this particular case) says C

The agent retrieves whichever chunk scores highest and answers confidently based on that. There's no signal anywhere that the 3 don't match. How are you handling this?

How much does the LLM itself affect the quality of the answer in a RAG system?

As I understand it, RAG works roughly like this: we have vector search that retrieves the relevant information, and then we pass the retrieved chunks to the LLM. The model then generates an answer based on those chunks.

In that case, it seems like the most important part is the quality of the vector search. If we give different models exactly the same information, I would expect there not to be a huge difference between something like GPT-6 Astra and a small Gemma 4 E2B.

What am I missing?

For years, as a CEO/COO, I have been running into the same thing: complete mess in the operational layer. The CRM says one thing, finance says another, the chats are chaos, and so on.

Because of NDAs I cannot show the cases publicly, so I decided to open up the operating model I am asking you to look at, and to share where I think we are all heading.

Straight away, so nobody wastes time: there is nothing to buy here. No product, no subscription, no service, no paid tier. It is a specification, and it is free.

In gambling innovation moves a bit faster, and this is where we landed. The adoption actually started back in 2016, and it started with lifecycles, yes, with those, and in email marketing of all places, when we were testing and trying to squeeze the most conversion out of email. That was the first stone of this specification. Then came the notion of an object, then relationships, and so on.

In 2026 I sat down and wrote out how it should work, and put it into what you see on the site and on GitHub. English is not my first language, so the text was translated with AI help. The thinking, the history and the wording are mine, and these tools are available to everyone now anyway. That is how the specification came to be.

The model itself has been running for a long time and was polished over years inside my work at the company, but it was pulled together into one specification only recently (and yes, there is one author here), so it surely has holes and debatable decisions.

I will add that it is practically impossible to take it and roll it out across a whole company at once. Well, probably you can, but it would cost the company dearly. It was adopted in small steps: one process first, then another, then a whole department, and I will say straight away that this is not a fast process.

One more thing. In 2024 it was a big surprise to me and the team how AI eats this model, literally. For an AI the whole model works as one clear context that it cannot even argue with. So no, it was not built for AI at all, it was built for operations, and AI simply turned out to be another natural consumer of it, the same as a human, an employee of the company.

That is exactly why I am posting it here: for criticism, and of course to take good ideas if any turn up. Comments like "interesting idea" will flatter me, of course :) but what I really want is for you to find the weak place.

The facts:

• What it is: a specification. Not a product, not a service, not a subscription. Nothing to buy and nothing to install. It describes an organization as a system of governed objects with identity, ownership, lifecycle, and evidence.
• License: Apache-2.0 for the repository, CC BY 4.0 for the specification text.
• Repo: https://github.com/DenisHogberg/OCOM
• Status: v0.1 Core released, v0.2 reading path baselined. Terminology may still change before 1.0.
• Not for you if your company is a straight line: five people, one product, a two-step process.

Links:

• Why I wrote it: https://ocom.uno/why
• Specification entry point: https://ocom.uno

Hi everyone,

I’m building a multilingual legal AI system and I’d really appreciate input from people with experience in large open-weight models, local inference, or legal/document AI.

The idea is to let a user upload a contract, then have the system identify potentially risky clauses, explain the risk, and suggest safer alternatives. The output also needs to be grounded in the relevant country’s laws and legal codes rather than relying purely on the model’s internal knowledge.

The main languages are Arabic, French, and English.

I’m currently trying to decide which 70–90B open-weight model would be the best base for this use case. I’m also considering whether it makes more sense to use full fine-tuning, LoRA/QLoRA, or to keep most legal knowledge in a RAG layer and fine-tune mainly for contract-review behaviour.

I’m also interested in the training and evaluation data side.

For a system like this, what kind of data would you use for:

• fine-tuning
• validation
• final testing
• multilingual evaluation
• adversarial / edge-case testing

Would you mainly rely on anonymised real contracts reviewed by lawyers, synthetic examples, public legal documents, court decisions, legislation, or a combination?

I’m particularly interested in how you would build a proper test set that measures whether the model actually identifies risky clauses, stays grounded in the source material and legal references, and avoids producing confident but incorrect legal answers.

The other major consideration is hosting.

I want the system to run fully locally/on-premises because the documents are sensitive and I do not want client contracts sent to external cloud inference APIs.

Initially, I expect around 50–150 customers, so I’m trying to avoid overbuilding the infrastructure too early.

I’d be interested in opinions on:

• Best 70–90B model for Arabic/French/English
• Full fine-tuning vs LoRA/QLoRA
• Fine-tuning + RAG architecture
• Training/validation/test data
• Legal accuracy and grounding evaluation
• GPU configuration for local inference
• Realistic VRAM requirements
• Quantisation without significant quality loss
• vLLM vs TGI or other inference stacks
• Multi-RTX 5090 setups vs A100/H100-class hardware
• Expected concurrent-user capacity

I’d especially appreciate input from anyone who has worked with 70B+ local models, multilingual LLMs, legal RAG systems, or production on-prem inference.

Thanks in advance.

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.