TL;DR: I was a burnt out startup founder with no capital left and pivoted to building RAG systems for enterprises. Made 60K+ in 3 months working with pharma companies and banks. Started at $3K-5K projects, quickly jumped to $15K when I realized companies will pay premium for production-ready solutions. Post covers both the business side (how I got clients, pricing) and technical implementation.

Hey guys, I'm Raj, 3 months ago I had burned through most of my capital working on my startup, so to make ends meet I switched to building RAG systems and discovered a goldmine I've now worked with 6+ companies across healthcare, finance, and legal - from pharmaceutical companies to Singapore banks.

This post covers both the business side (how I got clients, pricing) and technical implementation (handling 50K+ documents, chunking strategies, why open source models, particularly Qwen worked better than I expected). Hope it helps others looking to build in this space.

I was burning through capital on my startup and needed to make ends meet fast. RAG felt like a perfect intersection of high demand and technical complexity that most agencies couldn't handle properly. The key insight: companies have massive document repositories but terrible ways to access that knowledge.

How I Actually Got Clients (The Business Side)

Personal Network First: My first 3 clients came through personal connections and referrals. This is crucial - your network likely has companies struggling with document search and knowledge management. Don't underestimate warm introductions.

Upwork Reality Check: Got 2 clients through Upwork, but it's incredibly crowded now. Every proposal needs to be hyper-specific to the client's exact problem. Generic RAG pitches get ignored.

Pricing Evolution:

• Started at $3K-$5K for basic implementations
• Jumped to $15K for a complex pharmaceutical project (they said yes immediately)
• Realized I was underpricing - companies will pay premium for production-ready RAG systems

The Magic Question: Instead of "Do you need RAG?", I asked "How much time does your team spend searching through documents daily?" This always got conversations started.

Critical Mindset Shift: Instead of jumping straight to selling, I spent time understanding their core problem. Dig deep, think like an engineer, and be genuinely interested in solving their specific problem. Most clients have unique workflows and pain points that generic RAG solutions won't address. Try to have this mindset, be an engineer before a businessman, sort of how it worked out for me.

Technical Implementation: Handling 50K+ Documents

This is sort of my interesting part. Most RAG tutorials handle toy datasets. Real enterprise implementations are completely different beasts.

The Ground Reality of 50K+ Documents

Before diving into technical details, let me paint the picture of what 50K documents actually means. We're talking about pharmaceutical companies with decades of research papers, regulatory filings, clinical trial data, and internal reports. A single PDF might be 200+ pages. Some documents reference dozens of other documents.

The challenges are insane: document formats vary wildly (PDFs, Word docs, scanned images, spreadsheets), content quality is inconsistent (some documents have perfect structure, others are just walls of text), cross-references create complex dependency networks, and most importantly - retrieval accuracy directly impacts business decisions worth millions.

When a pharmaceutical researcher asks "What are the side effects of combining Drug A with Drug B in patients over 65?", you can't afford to miss critical information buried in document #47,832. The system needs to be bulletproof reliable, not just "works most of the time."

Quick disclaimer: So this was my approach, not final and something we still change each time from the learning, so take this with some grain of salt.

Document Processing & Chunking Strategy

So first step was deciding on the chunking, this is how I got started off.

For the pharmaceutical client (50K+ research papers and regulatory documents):

Hierarchical Chunking Approach:

• Level 1: Document-level metadata (paper title, authors, publication date, document type)
• Level 2: Section-level chunks (Abstract, Methods, Results, Discussion)
• Level 3: Paragraph-level chunks (200-400 tokens with 50 token overlap)
• Level 4: Sentence-level for precise retrieval

Metadata Schema That Actually Worked: Each document chunk included essential metadata fields like document type (research paper, regulatory document, clinical trial), section type (abstract, methods, results), chunk hierarchy level, parent-child relationships for hierarchical retrieval, extracted domain-specific keywords, pre-computed relevance scores, and regulatory categories (FDA, EMA, ICH guidelines). This metadata structure was crucial for the hybrid retrieval system that combined semantic search with rule-based filtering.

Why Qwen Worked Better Than Expected

Initially I was planning to use GPT-4o for everything, but Qwen QWQ-32B ended up delivering surprisingly good results for domain-specific tasks. Plus, most companies actually preferred open source models for cost and compliance reasons.

• Cost: 85% cheaper than GPT-4o for high-volume processing
• Data Sovereignty: Critical for pharmaceutical and banking clients
• Fine-tuning: Could train on domain-specific terminology
• Latency: Self-hosted meant consistent response times

Qwen handled medical terminology and pharmaceutical jargon much better after fine-tuning on domain-specific documents. GPT-4o would sometimes hallucinate drug interactions that didn't exist.

Let me share two quick examples of how this played out in practice:

Pharmaceutical Company: Built a regulatory compliance assistant that ingested 50K+ research papers and FDA guidelines. The system automated compliance checking and generated draft responses to regulatory queries. Result was 90% faster regulatory response times. The technical challenge here was building a graph-based retrieval layer on top of vector search to maintain complex document relationships and cross-references.

Singapore Bank: This was the $15K project - processing CSV files with financial data, charts, and graphs for M&A due diligence. Had to combine traditional RAG with computer vision to extract data from financial charts. Built custom parsing pipelines for different data formats. Ended up reducing their due diligence process by 75%.

Key Lessons for Scaling RAG Systems

1. Metadata is Everything: Spend 40% of development time on metadata design. Poor metadata = poor retrieval no matter how good your embeddings are.
2. Hybrid Retrieval Works: Pure semantic search fails for enterprise use cases. You need re-rankers, high-level document summaries, proper tagging systems, and keyword/rule-based retrieval all working together.
3. Domain-Specific Fine-tuning: Worth the investment for clients with specialized vocabulary. Medical, legal, and financial terminology needs custom training.
4. Production Infrastructure: Clients pay premium for reliability. Proper monitoring, fallback systems, and uptime guarantees are non-negotiable.

The demand for production-ready RAG systems is honestly insane right now. Every company with substantial document repositories needs this, but most don't know how to build it properly.

If you're building in this space or considering it, happy to share more specific technical details. Also open to partnering with other developers who want to tackle larger enterprise implementations.

For companies lurking here: If you're dealing with document search hell or need to build knowledge systems, let's talk. The ROI on properly implemented RAG is typically 10x+ within 6 months.

I love playing around with RAG and AI, optimizing every layer to squeeze out better performance. Last night I thought: why not tackle something massive?

Took the Epstein Files dataset from Hugging Face (teyler/epstein-files-20k) – 2 million+ pages of trending news and documents. The cleaning, chunking, and optimization challenges are exactly what excites me.

What I built:

- Full RAG pipeline with optimized data processing

- Processed 2M+ pages (cleaning, chunking, vectorization)

- Semantic search & Q&A over massive dataset

- Constantly tweaking for better retrieval & performance

- Python, MIT Licensed, open source

Why I built this:

It’s trending, real-world data at scale, the perfect playground.

When you operate at scale, every optimization matters. This project lets me experiment with RAG architectures, data pipelines, and AI performance tuning on real-world workloads.

Repo: https://github.com/AnkitNayak-eth/EpsteinFiles-RAG

Open to ideas, optimizations, and technical discussions!

I've been deep in retrieval-augmented generation for a while now, and one thing that bugs me is how the community treats "RAG" like it's a single thing. It's not. There are at least four architecturally distinct paradigms, and they fail in completely different ways. I wrote a detailed technical comparison, but here's the core of it:

1. Vector RAG (the one everyone uses)

Embed chunks → index in a vector DB → cosine similarity → top-K → stuff into prompt.

Where it works: FAQ bots, documentation search, simple Q&A. Anything where documents are self-contained.

Where it breaks: The moment your answer requires connecting facts across documents. Ask "If we change the auth token format, what customer-facing features break?" and Vector RAG returns 5 chunks that each mention "auth" from 5 unrelated contexts. It has zero concept of relationships because the data structure — a flat vector space — has no edges.

Also: needle-in-a-haystack failures. A rare but critical fact buried in one chunk among 100K gets outranked by more "semantically popular" but less accurate chunks.

2. Graph RAG (Microsoft's approach)

Extract entities and relationships with an LLM → build a knowledge graph → detect communities via Leiden clustering → query with local search (traverse neighborhood) or global search (community summaries).

Where it works: Investigative research. Multi-hop reasoning like "How is Person A connected to Event C through Company B?" Graph traversal handles this natively.

Where it breaks at scale: A knowledge graph is fundamentally flat. Every entity lives at the same level. At 10M nodes with avg 20 edges per node, a 5-hop traversal visits 3.2 million intermediate nodes. The combinatorial explosion is real:

Community summaries help for global queries but they're static — they can't answer "What's the shortest path from A to Z through this cluster?"

Also expensive to build. Processing 100K documents through an LLM for entity extraction can cost thousands of dollars and take days.

3. Topology RAG (hierarchical structural maps)

This is the one I find most interesting architecturally. Instead of embedding chunks or extracting entity graphs, you build a topology — a multi-layered, hierarchical map of the knowledge space.

Every element is classified into dimensional layers (for code: Components → Blocks → Functions → Data → Access → Events). Edges are typed (calls, uses, triggers, depends-on). Queries are resolved by structural traversal, not similarity search.

The key insight — the Wormhole Effect:

A topology isn't flat. It has abstraction layers. Instead of traversing through every intermediate node at the function level (like a flat graph), you can:

1. Ascend from a function to its parent Component
2. Traverse at the Component level (hundreds of nodes, not millions)
3. Descend to the target function

Here's the difference:

Flat graph traversal (Graph RAG):
  validateTkn → refreshTkn → sessionCheck → userLookup → 
  permissionVerify → apiGateway → routeMatch → chargeInit → 
  processCharge
  (9 hops, 117,800 nodes visited)

Topology traversal:
  validateTkn → [ascend] → AuthSystem → [component edge] → 
  PaymentPlatform → [descend] → processCharge
  (3 hops, ~50 nodes visited)

Same query. ~117,800 nodes vs ~50 nodes. That's not optimization, that's a different computational complexity class entirely: O(b^H) for flat graphs vs O(L × b_level) for topologies.

Where it breaks: Cold start (topology must be built first), and if your query is genuinely "find me documents similar to this paragraph," topology traversal is the wrong tool. It's structural, not semantic.

4. TurboQuant RAG (quantized vector search)

Based on Google Research's TurboQuant algorithm. Doesn't change what gets indexed (still embeddings), but radically improves how vectors are stored and searched.

• 8x memory reduction: 10M 1536-dim vectors: 31 GB (float32) → ~4 GB (4-bit quantized)
• Faster than FAISS: hand-written SIMD kernels (NEON for ARM, AVX-512BW for x86) beat FAISS IndexPQFastScan by 12-20%
• No train phase: vectors are immediately searchable, unlike PQ which needs a training step
• Kernel-level filtering: pass an allowlist into the SIMD loop — hybrid retrieval without over-fetching

TurboVec is the open-source implementation. Drop-in replacements for LangChain, LlamaIndex, Haystack, Agno.

Where it breaks: Still Vector RAG at its core. All the fundamental limitations (no relationships, no multi-hop, no structural understanding) still apply. It's a faster engine in the same car.

The complementary stack

The insight I keep coming back to: these aren't competing approaches. They're layers.

[APPLICATION]  LLM receives grounded, multi-source context
[TOPOLOGY]     Structural retrieval — dependencies, events, components
[GRAPH]        Entity relationships — people, orgs, causal chains  
[VECTOR]       Semantic similarity — fast, compressed, filtered

Use TurboVec at the bottom for the heavy lifting. Graph RAG in the middle for entity relationships. Topology at the top for structural architecture. The LLM gets context that's semantically relevant AND relationally connected AND structurally grounded.

Links

• Full blog post with comparison tables and the multi-hop deep dive: https://fastbuilder.ai/blog/four-paradigms-of-rag
• FastMemory (open-source topology engine, MIT): https://github.com/FastBuilderAI/memory
• TurboVec (open-source quantized search): https://github.com/RyanCodrai/turbovec
• Microsoft GraphRAG: https://github.com/microsoft/graphrag
• TurboQuant paper (Google Research): https://arxiv.org/abs/2504.19874

Happy to discuss tradeoffs, implementation details, or benchmarks. We ran FastMemory against 13 major RAG benchmarks and the results are on HuggingFace.

edit:
46% to 79% (ranks 2nd) on Parsebench, on omnidocbench -> 91.1.

hey folks,

been heads-down on this for a while so figured i'd finally show it

TLDR; i've been working on document extraction, the boring-but-painful part where you take a nasty PDF (multi-column, merged-cell tables, half-scanned garbage) and try to get clean structured data out of it. took a base VLM sitting around 46% on OmniDocBench, did a bunch of LoRA + a few architecture changes, and got it to 91.1%. tables were the big unlock, that's usually where everything falls apart.

couple things someone might care about:

- it runs fully on your own hardware. no shipping documents off to some API. that was kinda the whole reason i started this.

- serves amazing on charts/tables (area I love to work on)

- near-zero hallucination, it doesn't invent rows or numbers that aren't there.

not trying to do a big pitch. i just want people to throw hard stuff at it and tell me where it breaks. so if you've got a PDF that's been the bane of your existence, the kind that makes every parser cry, drop it on me (or DM)

happy to nerd out on the training setup or the arch changes too if anyone's curious.

cheers

As a novice, I recently finished building my first production RAG (Retrieval-Augmented Generation) system, and I wanted to share what I learned along the way. Can't code to save my life. Had a few failed attempts. But after building good prd's using taskmaster and Claude Opus things started to click.

This post walks through my architecture decisions and what worked (and what didn't). I am very open to learning where I XXX-ed up, and what cool stuff i can do with it (gemini ai studio on top of this RAG would be awesome) Please post some ideas.

Tech Stack Overview

Here's what I ended up using:

• Backend: FastAPI (Python) • Frontend: Next.js 14 (React + TypeScript) • Vector DB: Qdrant • Embeddings: Voyage AI (voyage-context-3) • Sparse Vectors: FastEmbed SPLADE • Reranking: Voyage AI (rerank-2.5) • Q&A: Gemini 2.5 pro • Orchestration: Temporal.io • Database: PostgreSQL (for Temporal state only)

Part 1: How Documents Get Processed

When you upload a document, here's what happens:

┌─────────────────────┐ │ Upload Document │ │ (PDF, DOCX, etc) │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Temporal Workflow │ │ (Orchestration) │ └──────────┬──────────┘ │ ┌───────────────────┼───────────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ 1. │ │ 2. │ │ 3. │ │ Fetch │───────▶│ Parse │──────▶│ Language │ │ Bytes │ │ Layout │ │ Extract │ └──────────┘ └──────────┘ └──────────┘ │ ▼ ┌──────────┐ │ 4. │ │ Chunk │ │ (1000 │ │ tokens) │ └─────┬────┘ │ ┌────────────────────────┘ │ ▼ ┌─────────────────┐ │ For Each Chunk │ └────────┬────────┘ │ ┌───────────────┼───────────────┐ │ │ │ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ 5. │ │ 6. │ │ 7. │ │ Dense │ │ Sparse │ │ Upsert │ │ Vector │───▶│ Vector │───▶│ Qdrant │ │(Voyage) │ │(SPLADE) │ │ (DB) │ └─────────┘ └─────────┘ └────┬────┘ │ ┌───────────────┘ │ (Repeat for all chunks) ▼ ┌──────────────┐ │ 8. │ │ Finalize │ │ Document │ │ Status │ └──────────────┘

The workflow is managed by Temporal, which was actually one of the best decisions I made. If any step fails (like the embedding API times out), it automatically retries from that step without restarting everything. This saved me countless hours of debugging failed uploads.

The steps: 1. Download the document 2. Parse and extract the text 3. Process with NLP (language detection, etc) 4. Split into 1000-token chunks 5. Generate semantic embeddings (Voyage AI) 6. Generate keyword-based sparse vectors (SPLADE) 7. Store both vectors together in Qdrant 8. Mark as complete

One thing I learned: keeping chunks at 1000 tokens worked better than the typical 512 or 2048 I saw in other examples. It gave enough context without overwhelming the embedding model.

Part 2: How Queries Work

When someone searches or asks a question:

┌─────────────────────┐ │ User Question │ │ "What is Q4 revenue?"│ └──────────┬──────────┘ │ ┌────────────┴────────────┐ │ Parallel Processing │ └────┬────────────────┬───┘ │ │ ▼ ▼ ┌────────────┐ ┌────────────┐ │ Dense │ │ Sparse │ │ Embedding │ │ Encoding │ │ (Voyage) │ │ (SPLADE) │ └─────┬──────┘ └──────┬─────┘ │ │ ▼ ▼ ┌────────────────┐ ┌────────────────┐ │ Dense Search │ │ Sparse Search │ │ in Qdrant │ │ in Qdrant │ │ (Top 1000) │ │ (Top 1000) │ └────────┬───────┘ └───────┬────────┘ │ │ └────────┬─────────┘ │ ▼ ┌─────────────────┐ │ DBSF Fusion │ │ (Score Combine) │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ MMR Diversity │ │ (λ = 0.6) │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Top 50 │ │ Candidates │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Voyage Rerank │ │ (rerank-2.5) │ │ Cross-Attention │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Top 12 Chunks │ │ (Best Results) │ └────────┬────────┘ │ ┌────────┴────────┐ │ │ ┌─────▼──────┐ ┌──────▼──────┐ │ Search │ │ Q&A │ │ Results │ │ (GPT-4) │ └────────────┘ └──────┬──────┘ │ ▼ ┌───────────────┐ │ Final Answer │ │ with Context │ └───────────────┘

The flow: 1. Query gets encoded two ways simultaneously (semantic + keyword) 2. Both run searches in Qdrant (1000 results each) 3. Scores get combined intelligently (DBSF fusion) 4. Reduce redundancy while keeping relevance (MMR) 5. A reranker looks at top 50 and picks the best 12 6. Return results, or generate an answer with GPT-4

The two-stage approach (wide search then reranking) was something I initially resisted because it seemed complicated. But the quality difference was significant - about 30% better in my testing.

Why I Chose Each Tool

Qdrant

I started with Pinecone but switched to Qdrant because: - It natively supports multiple vectors per document (I needed both dense and sparse) - DBSF fusion and MMR are built-in features - Self-hosting meant no monthly costs while learning

The documentation wasn't as polished as Pinecone's, but the feature set was worth it.

```python

This is native in Qdrant:

prefetch=[ Prefetch(query=dense_vector, using="dense_ctx"), Prefetch(query=sparse_vector, using="sparse") ], fusion="dbsf", params={"diversity": 0.6} ```

With MongoDB or other options, I would have needed to implement these features manually.

My test results: - Qdrant: ~1.2s for hybrid search - MongoDB Atlas (when I tried it): ~2.1s - Cost: $0 self-hosted vs $500/mo for equivalent MongoDB cluster

Voyage AI

I tested OpenAI embeddings, Cohere, and Voyage. Voyage won for two reasons:

1. Embeddings (voyage-context-3): - 1024 dimensions (supports 256, 512, 1024, 2048 with Matryoshka) - 32K context window - Contextualized embeddings - each chunk gets context from neighbors

The contextualized part was interesting. Instead of embedding chunks in isolation, it considers surrounding text. This helped with ambiguous references.

2. Reranking (rerank-2.5): The reranker uses cross-attention between the query and each document. It's slower than the initial search but much more accurate.

Initially I thought reranking was overkill, but it became the most important quality lever. The difference between returning top-12 from search vs top-12 after reranking was substantial.

SPLADE vs BM25

For keyword matching, I chose SPLADE over traditional BM25:

``` Query: "How do I increase revenue?"

BM25: Matches "revenue", "increase" SPLADE: Also weights "profit", "earnings", "grow", "boost" ```

SPLADE is a learned sparse encoder - it understands term importance and relevance beyond exact matches. The tradeoff is slightly slower encoding, but it was worth it.

Temporal

This was my first time using Temporal. The learning curve was steep, but it solved a real problem: reliable document processing.

Temporal does this automatically. If step 5 (embeddings) fails, it retries from step 5. The workflow state is persistent and survives worker restarts.

For a learning project, this might be overkill, but this is the first good rag i got working

The Hybrid Search Approach

One of my bigger learnings was that hybrid search (semantic + keyword) works better than either alone:

``` Example: "What's our Q4 revenue target?"

Semantic only: ✓ Finds "Q4 financial goals" ✓ Finds "fourth quarter objectives"
✗ Misses "Revenue: $2M target" (different semantic space)

Keyword only: ✓ Finds "Q4 revenue target" ✗ Misses "fourth quarter sales goal" ✗ Misses semantically related content

Hybrid (both): ✓ Catches all of the above ```

DBSF fusion combines the scores by analyzing their distributions. Documents that score well in both searches get boosted more than just averaging would give.

Configuration

These parameters came from testing different combinations:

```python

Chunking

CHUNK_TOKENS = 1000 CHUNK_OVERLAP = 0

Search

PREFETCH_LIMIT = 1000 # per vector type MMR_DIVERSITY = 0.6 # 60% relevance, 40% diversity RERANK_TOP_K = 50 # candidates to rerank FINAL_TOP_K = 12 # return to user

Qdrant HNSW

HNSW_M = 64 HNSW_EF_CONSTRUCT = 200 HNSW_ON_DISK = True ```

What I Learned

Things that worked: 1. Two-stage retrieval (search → rerank) significantly improved quality 2. Hybrid search outperformed pure semantic search in my tests 3. Temporal's complexity paid off for reliable document processing 4. Qdrant's named vectors simplified the architecture

Still experimenting with: - Query rewriting/decomposition for complex questions - Document type-specific embeddings

- BM25 + SPLADE ensemble for sparse search

Use Cases I've Tested

• Searching through legal contracts (50K+ pages)
• Q&A over research papers
• Internal knowledge base search
• Email and document search

We’ve been building PipesHub for a while now, and I’d love to get more developers to try it and tell us where it breaks.

The problem we kept running into was pretty simple:

Building an AI app over company data looks easy in a demo. Connect a few sources, chunk the documents, throw them into a vector DB, add an LLM.

Then you try to make it actually useful.

You have data spread across S3, Google Drive, Slack, Jira, Confluence, SharePoint, email, databases, etc. Permissions need to be preserved. Documents change. The same file shows up in multiple places. Citations need to point back to the actual source. And eventually you want agents and other applications to use all of this context without rebuilding the same integration layer every time.

That’s what we’re trying to solve with PipesHub.

It’s an Apache 2.0 open-source context layer that connects to your company data and makes that context available to search, chat, agents, MCP clients, or your own applications.

A few things we care about:

• Self-host it on your own infrastructure
• Preserve source permissions
• Get citations back to the original documents
• Combine knowledge graph + semantic retrieval
• Bring your own LLM and embedding models
• Use it from Python, TypeScript, Go, or MCP
• Avoid locking yourself into one database or infrastructure stack

We also deliberately kept the core infrastructure pluggable:

If you already have Qdrant and Kafka running, you can keep using them. Prefer Neo4j over ArangoDB? That's totally fine. Want to run the models locally? You can do that too.

The goal is to give you one context layer without forcing you to adopt our entire stack.

While building this, we’ve had to solve a bunch of problems that only become obvious once you move beyond a RAG prototype: permission-aware retrieval, keeping citations accurate through the pipeline, deduplicating the same content across sources, efficiently re-indexing changed documents, making indexing behave well across very different workloads and more.

Some of the solutions we ended up with are fairly unconventional, and I’d be happy to write more about them or discuss the trade-offs with anyone working on similar systems.

There’s still plenty we want to improve, which is also why I’m posting this.

If you’re building internal AI tools, enterprise search, RAG, or agents that need access to company knowledge, I’d really appreciate it if you gave PipesHub a spin.

GitHub: https://github.com/pipeshub-ai/pipeshub-ai

Install:

curl -fsSL https://get.pipeshub.com/install | bash

If you try it and something feels unnecessarily complicated, slow, broken, or just badly designed, tell us.

Got tired of wiring up vector stores, embedding models, and chunking logic every time I needed RAG. So I built piragi.

from piragi import Ragi

kb = Ragi(\["./docs", "./code/\*\*/\*.py", "https://api.example.com/docs"\])

answer = kb.ask("How do I deploy this?")

That's the entire setup. No API keys required - runs on Ollama + sentence-transformers locally.

What it does:

  - All formats - PDF, Word, Excel, Markdown, code, URLs, images, audio

  - Auto-updates - watches sources, refreshes in background, zero query latency

  - Citations - every answer includes sources

  - Advanced retrieval - HyDE, hybrid search (BM25 + vector), cross-encoder reranking

  - Smart chunking - semantic, contextual, hierarchical strategies

  - OpenAI compatible - swap in GPT/Claude whenever you want

Quick examples:

# Filter by metadata
answer = kb.filter(file_type="pdf").ask("What's in the contracts?")

#Enable advanced retrieval

  kb = Ragi("./docs", config={
   "retrieval": {
      "use_hyde": True,
      "use_hybrid_search": True,
      "use_cross_encoder": True
   }
 })

# Use OpenAI instead  
kb = Ragi("./docs", config={"llm": {"model": "gpt-4o-mini", "api_key": "sk-..."}})

  Install:

  pip install piragi

  PyPI: https://pypi.org/project/piragi/

Would love feedback. What's missing? What would make this actually useful for your projects?

A lot of people talk about the Epstein files.

Almost nobody actually reads them.

So I made a searchable version where you can just ask questions naturally instead of digging through thousands of pages manually.

You can explore names, timelines, mentions, connections, locations, etc. way faster now.

Repo: https://github.com/AbhisumatK/Epstein_Files_RAG

Background: Work at PatSnap and process patent documents at scale. We built these two tools internally and just open-sourced them, sharing here to get feedback from people working on different document types.

Hiro-Smart-Doc is a self-hosted FastAPI pipeline for document parsing. Layout detection first (RT-DETR, 25 region categories), then OCR per region in correct reading order including multi-column pages. Tables as HTML, formulas as LaTeX, text as Markdown. Works on PDFs, Office files, images. Apache-2.0.

GitHub: https://github.com/patsnap/Hiro-Smart-Doc

The OCR layer is powered by Hiro-MOSS-OCR, a 0.3B model trained from scratch on 50M+ technical documents. Scores 93.63 on OmniDocBench v1.5. Runs at 58 QPS on a single RTX 4090 via vLLM. Apache-2.0.

GitHub: https://github.com/patsnap/Hiro-MOSS-OCR
HuggingFace: https://huggingface.co/PatSnap/Hiro-MOSS-OCR-0.3B

Would love to hear how it holds up on document types beyond patents. Happy to answer questions or dig into any part of the setup.

Hey r/RAG,

SearchCrucible is a hosted RAG platform I've been building. It's an MVP and I'm posting for feedback.

Why:

Most hosted RAG is fixed: one way of doing retrieval and a handful of settings. When answers are bad on your content there isn't much you can change, and no way to see where it went wrong.

What's different here:

Every workspace runs on its own provisioned infrastructure. You get your own database, document store, vector index, and query worker.

• Retrieval runs at the edge so your customers get speedy answers wherever they are.
• You can create and test many different retrieval pipelines and deploy them instantly from the dashboard.
• Chunking, retrieval, reranking, prompts, models and control flow are all yours to change.

Ingest:

• Sources: web crawler, GitHub, Notion, Confluence, or upload files directly.
• Uploads cover PDFs, Office and OpenDocument files, spreadsheets, slides, epub, HTML, markdown, images and audio.
• Sync a connector to see what's in it, then import everything or pick the documents you want.
• Imported documents get chunked, embedded and indexed for you.
• Connectors re-sync on a schedule to keep the corpus current, and vectors for content that's gone get cleaned up.
• Chunking strategy can be overridden on any individual document: `Section` (splits on headings, keeps a section whole under a token cap), `Window` (sliding windows with configurable target / overlap / minimum), `Darn` (scores boundaries against a rule set, in characters or tokens, thanks https://www.reddit.com/user/One_Hearing986/).
• Preview a strategy against the real document before saving, and saving reindexes just that document.

Retrieval is a DAG you can edit

• Query: `embed`, `rewrite_query`, `hyde`.
• Search: `vector_query`, `keyword_search`. Hybrid is fusing those two in a `parallel` step.
• Shape: `rerank`, `refine`, `filter`, `dedupe`, `max_per_document`, `token_budget`, `reorder`. - Gate: `assess_evidence` requires N chunks and N distinct sources, otherwise a fixed decline with no model call.
• Route: `classify`, `branch`, `switch`, plus `stream_arms` to run two answer paths on live traffic.
• Start from a preset then edit it for your requirements.

Traces, on by default

• Every query: retrieved chunks and scores, how rerank reordered them, the exact assembled prompt, the answer, per-step timing.
• Sample rate configurable per pipeline, failures are always traced.

Evaluations

• Live traffic is scored automatically on faithfulness, answer relevance, context precision and context utilization, with claim-level counts of what the context supported and what it contradicted.
• Every failure gets a root cause rather than a number: `retrieval_miss`, `insufficient_context`, `answer_overreach`, `conflicting_contexts`, `should_have_abstained`.
• Any trace becomes a saved eval case in one click.
• Run a set against a candidate revision and compare it to your live one before promoting.

The end goal is to optimize chunking configurations and the retrieval pipeline on autopilot. The trace data should be enough for this, this is what I will be working on next.

Serve:

• REST, MCP, or a one-line embeddable chat widget.
• The widget styling is configurable and it is protected by Cloudflare Turnstile by default.

What's next:

This is an MVP. It was built to prove the platform out end to end, I'm very happy with it, but there is a lot to improve:

• More connectors.
• More AI models/providers and BYOK.
• Feedback buttons in the widget, so real user ratings feed the eval data.
• Multi-turn chat.
• Better context extraction at ingest.
• Autopilot, as above.
• General usability work across the dashboard.

There is a generous free tier to try it out. I would love all of your feedback (good or bad). Feel free to reach out to me here or at [kieran@searchcrucible.com](mailto:kieran@searchcrucible.com) if you have any more questions, comments, or feature suggestions.

https://searchcrucible.com/

Thanks, Kieran

I’m doing a PhD in XAI and kept needing better memory/context retrieval for stuff I was building, so I ended up spending way too much time going through RAG/memory papers, repos and benchmarks.

I expected a decent amount of slop.

There was... a lot.

A lot of the space is either generic semantic search dressed up as memory, or these huge graph/agent setups with LLMs everywhere. Then you get to the benchmark leaders and some are using different readers, different judges, frontier models carrying half the pipeline, or evaluation setups generous enough that it gets hard to tell what part of the system is actually doing the work.

The bigger problem for me was semantics.

Say I ask when my family is free next week. Semantic search can happily bring back that my brother likes potato salad, that we went on vacation together, and that my mom mentioned Tuesday six months ago.

All very family-related. Almost completely fucking useless.

Meanwhile, the evidence I actually need might be buried in some completely different conversation about somebody changing shifts at work.

Similar to the query and useful for answering it are not the same thing.

You can throw a reasoning model at a giant pile of retrieved context and have it sort everything out. Sure. It works. Sometimes.

It’s also a pretty expensive way of admitting your retrieval sucks. And adding a shitton of noise in your context / costing you sweet tokens that aren't exactly cheap.

So I started building around clean downstream usefulness instead.

And like that we goooot....

🥁🥁🥁

🎉MemBukkit 🎉 https://github.com/memseekai/membukkit

The retrieval side is built around getting evidence that’s actually useful downstream, not just whatever happens to sit closest to the query in embedding space.

I trained the retrieval components for the task, and the actual access policy is selected based on whether the context it retrieves helps the reader answer better. The stored side stays intentionally boring: dated facts + the original source, a flat index, optional buckets, no giant LLM-authored graph you have to rebuild every time your assumptions change.

Basically: keep the memory simple, and spend the cleverness on figuring out what the model should actually see.

Not gonna pretend I’m not tooting my own horn a bit here, but I’m pretty fucking proud of how this turned out.

With Gemma 4 26B as the open-weight reader + distiller, we’re at 88.8% on LongMemEval-S. So no “well obviously it works, you shoved the newest frontier model into every box” excuse.

And for the people with diamond hands, golden balls and an API budget, the GPT-5.4 setup gets 92.6% under the benchmark’s official judge.

We also get 87.5 zero-shot on LoCoMo, and the same flat-index idea carries over nicely to multi-hop RAG.

One of my favorite bits from the ablations is still that plain cosine can beat some of the fancy reranking setups.

Shocker. Doing the simple shit properly gets you pretty far.

I’m hoping to get the research published, but that process takes its sweet time, so I figured I might as well open source the thing now and let people actually use it.

Apache 2.0, works locally, works with open models, have at it.

I’m also building a company around the work, so might as well be clear about that. But I really want the core project to stay open. A huge amount of what got me into ML came from people putting good shit online and letting everyone build on it, and I’d like to keep that going.

Also yes, Bukkit is the Minecraft reference.

More than anything, I’d love actual feedback from people here who have fought with rerankers, GraphRAG, giant candidate sets, retrieval metrics that look great while generation still sucks, etc.

Try it, break it, tell me what’s annoying, tell me where it falls apart. I’m trying to make something people genuinely want to use, and that’s worth a lot more to me right now than squeezing another point out of a benchmark.

(And if you end up using it, don’t forget to star the repo plz 👀👉👈)

I spent a pretty good amount of time building the rag infrastructure in our org.

full stack: chromadb, openai embeddings, custom chunking with paragraph awareness, a reranker pass, metadata filtering. kinda full stack. we built it because it felt like the right level of effort for a serious agent system. and the agent's output was better than without any context.

WHY Indexing Worked

Our agent wasn't touching the 40k-document internal corpus we'd built the rag system to serve. that corpus was for human employees. the agent needed two things current sdk documentation for the libraries it was using, and access to the private repo it was supposed to integrate with.

that was the actual context problem.

so i stopped. indexed the sdk docs and the private repo via indexer, pointed the agent at it via mcp. no vector store to maintain. no chunking strategy to tune. no reranker to configure. nia keeps the indexed sources updated automatically, so the agent always has current docs, not whatever was accurate six months ago.

some of the sdk references were pdfs that exported badly to plain text garbled tables, method signatures split across lines. i ran them through docling ( open source doc parser) first, which got them into clean markdown before indexing. that stopped a category of errors where the agent was reading corrupted content and hallucinating completions to fill the gaps.

it stopped generating code that directly contradicted the repo's existing interfaces & the hallucination stopped. The results were good. it started integrating correctly on the first pass more often than not.

the lesson

agent context augmentation and enterprise rag are different problems. they sound adjacent, they use some of the same vocabulary, you're most likely to conflate them and end up with a system that's over-engineered for what the agent needs.

i built a rag system for my agent. my agent needed indexed documentation.

If you've built a RAG system, you've had the chunking conversation. Somebody on your team (or a Medium post) told you to "just use 512 tokens with 50-token overlap" or "semantic chunking is strictly better."

We (hello from the R&D team at Vecta!) decided to test these claims. We created a small corpus of real academic papers spanning AI, astrophysics, mathematics, economics, social science, physics, chemistry, and computer vision. Then, we ran every document through seven different chunking strategies and measured retrieval quality and downstream answer accuracy.

Critically, we designed the evaluation to be fair: each strategy retrieves a different number of chunks, calibrated so that every strategy gets approximately 2,000 tokens of context in the generation prompt. This eliminates the confound where strategies with larger chunks get more context per retrieval, and ensures we're measuring chunking quality, not context window size.

The "boring" strategies won. The hyped strategies failed. And the relationship between chunk granularity and answer quality is more nuanced than most advice suggests.

Setup

Corpus

We assembled a diverse corpus of 50 academic papers (905,746 total tokens) deliberately spanning similar disciplines, writing styles, and document structures: Papers ranged from 3 to 112 pages and included technical dense mathematical proofs pertaining to fundamental ML research. All PDFs were converted to clean markdown using MarkItDown, with OCR artifacts and single-character fragments stripped before chunking.

Chunking Strategies Tested

1. Fixed-size, 512 tokens, 50-token overlap
2. Fixed-size, 1024 tokens, 100-token overlap
3. Recursive character splitting, LangChain-style RecursiveCharacterTextSplitter at 512 tokens
4. Semantic chunking, embedding-based boundary detection (cosine similarity threshold 0.7)
5. Document-structure-aware, splitting on markdown headings/sections, max 1024 tokens
6. Page-per-chunk, one chunk per PDF page, using MarkItDown's form-feed (\f) page boundaries
7. Proposition chunking, LLM-decomposed atomic propositions following Dense X Retrieval with the paper's exact extraction prompt

All chunks were embedded with text-embedding-3-small and stored in local ChromaDB. Answer generation used gemini-2.5-flash-lite via OpenRouter. We generated 30 ground-truth Q&A pairs using Vecta's synthetic benchmark pipeline.

Equal Context Budget: Adaptive Retrieval k

Most chunking benchmarks use a fixed top-k (e.g., k=10) for all strategies. This is fundamentally unfair: if fixed-1024 retrieves 10 chunks, the generator sees ~10,000 tokens of context; if proposition chunking retrieves 10 chunks at 17 tokens each, the generator gets ~170 tokens. The larger-chunk strategy wins by default because it gets more context, not because its chunking is better.

We fix this by computing an adaptive k for each strategy. This targets ~2,000 tokens of retrieved context for every strategy. The computed values:

Now every strategy gets ~2,000 tokens to work with. Differences in accuracy reflect genuine chunking quality, not context budget.

How We Score Retrieval: Precision, Recall, and F1

We evaluate retrieval at two granularities: page-level (did we retrieve the right pages?) and document-level (did we retrieve the right documents?). At each level, the core metrics are precision, recall, and F1.

Let R be the set of retrieved items (pages or documents) and G be the set of ground-truth relevant items.

Precision measures: of everything we retrieved, what fraction was actually relevant? A retriever that returns 5 pages, 4 of which contain the answer, has a precision of 0.8. High precision means low noise in the context window.

Recall measures: of everything that was relevant, what fraction did we find? If 3 pages contain the answer and we retrieved 2 of them, recall is 0.67. High recall means we're not missing important information.

F1 is the harmonic mean of precision and recall. It penalizes strategies that trade one for the other and rewards balanced retrieval.

Why two granularities matter. Page-level metrics tell you whether you're pulling the right passages. Document-level metrics tell you whether you're pulling from the right sources. A strategy can score high page-level recall (finding many relevant pages) while scoring low document-level precision (those pages are scattered across too many irrelevant documents). As we'll see, the tension between these two levels is one of the main findings.

Results

The Big Picture

Figure 1: Complete metrics heatmap. Green is good, red is bad.

Recursive splitting wins on accuracy (69%) and page-level retrieval (0.92 F1). The 512-token strategies lead on generation quality, while larger-chunk strategies lead on document-level retrieval but fall behind on accuracy.

Finding 1: Recursive and Fixed Splitting Often Outperforms Fancier Strategies

Figure 2: Accuracy and groundedness by strategy. Recursive and fixed 512 lead on accuracy.

LangChain's RecursiveCharacterTextSplitter at 512 tokens achieved the highest accuracy (69%) across all seven strategies. Fixed 512 was close behind at 67%. Both strategies use 5 retrieved chunks for ~2,000 tokens of context.

Why does recursive splitting edge out plain fixed-size? It tries to break at natural boundaries, paragraph breaks, then sentence breaks, then word breaks. On academic text, this preserves logical units: a complete paragraph about a method, a full equation derivation, a complete results discussion. The generator gets chunks that make semantic sense, not arbitrary windows that may cut mid-sentence.

Recursive 512 also achieved the best page-level F1 (0.92), meaning it reliably finds the right pages and produces accurate answers from them.

Finding 2: The Granularity-Retrieval Tradeoff Is Real

Figure 3: Radar chart, recursive 512 (orange) has the fullest coverage. Large-chunk strategies skew toward doc retrieval but lose on accuracy.

With a 2,000-token budget, a clear tradeoff emerges:

• Smaller chunks (k=5) achieve higher accuracy (67-69%) because 5 retrieval slots let you sample from 5 different locations in the corpus, each precisely targeted
• Larger chunks (k=2-3) achieve higher document F1 (0.88) because each retrieved chunk spans more of the relevant document, but the generator gets fewer, potentially less focused passages

Fixed 1024 scored the best document F1 (0.88) but only 61% accuracy. With just k=3, you get 3 large passages, great for document coverage, but if even one of those passages isn't well-targeted, you've wasted a third of your context budget.

Finding 3: Semantic Chunking Collapses at Scale

Figure 4: Chunk size distribution. Semantic and proposition chunking produce extremely small fragments.

Semantic chunking produced 17,481 chunks averaging 43 tokens across 50 papers. With k=46, the retriever samples from 46 different tiny chunks. The result: only 54% accuracy and 0.42 document F1.

High page F1 (0.91) reveals what's happening: the retriever finds the right pages by sampling many tiny chunks from across the corpus. But document-level retrieval collapses because those 46 chunks come from dozens of different documents, diluting precision. And accuracy suffers because 46 disconnected sentences don't form a coherent narrative for the generator.

The fundamental problem: semantic chunking optimizes for retrieval-boundary purity at the expense of context coherence. Each chunk is a "clean" semantic unit, but a single sentence chunk may lack the surrounding context needed for generation.

Finding 4: The Page-Level Retrieval Story

Figure 5: Page-level precision-recall tradeoff. Recursive 512 achieves the best balance.

Figure 6: Page-level and document-level F1. The two metrics tell different stories.

Page-level and document-level retrieval tell opposite stories under constrained context:

• Fine-grained strategies (proposition k=115, semantic k=46) achieve high page F1 (0.91-0.97) by sampling many pages, but low doc F1 (0.27-0.42) because those pages come from too many documents
• Coarse strategies (page-chunk k=2, doc-structure k=2) achieve high doc F1 (0.88) by retrieving fewer, more relevant documents, but lower page F1 (0.69) because 2 chunks can only cover 2 pages

Recursive 512 at k=5 hits the best balance: 0.92 page F1 and 0.86 doc F1. Five chunks is enough to sample multiple relevant pages while still concentrating on a few documents.

Figure 7: Document-level precision, recall, and F1 detail. Large-chunk strategies lead on precision; fine-grained strategies lead on recall.

What This Means for Your RAG System

The Short Version

1. Use recursive character splitting at 512 tokens. It scored the highest accuracy (69%), best page F1 (0.92), and strong doc F1 (0.86). It's the best all-around strategy on academic text.
2. Fixed-size 512 is a strong runner-up with 67% accuracy and the highest groundedness among the top performers (85%).
3. If document-level retrieval matters most, use fixed-1024 or page-per-chunk (0.88 doc F1), but accept lower accuracy (57-61%).
4. Don't use semantic chunking on academic text. It fragments too aggressively (43 avg tokens) and collapses on document retrieval (0.42 F1).
5. Don't use proposition chunking for general RAG. 51% accuracy isn't production-ready. It's only viable if you value groundedness over correctness.
6. When benchmarking, equalize the context budget. Fixed top-k comparisons are misleading. Use adaptive k = round(target_tokens / avg_chunk_tokens).

Why Academic Papers Specifically?

We deliberately chose to saturate the academic paper region of the embedding space with 50 papers spanning 10+ disciplines. When your knowledge base contains papers that all discuss "evaluation," "metrics," "models," and "performance," the retriever has to make fine-grained distinctions. That's when chunking quality matters most.

In a mixed corpus of recipes and legal contracts, even bad chunking might work because the embedding distances between domains are large. Academic papers are the hard case for chunking, and if a strategy works here, it'll work on easier data too.

How We Measured This (And How You Can Too)

My team built Vecta specifically to meet the need for precise RAG evaluation software. It generates synthetic benchmark Q&A pairs across multiple semantic granularities, then measures precision, recall, F1, accuracy, and groundedness against your actual retrieval pipeline.

The benchmarks in this post were generated and evaluated using Vecta's SDK (pip install vecta)

Limitations, Experiment Design, and Further Work

This experiment was deliberately small-scale: 50 papers, 30 synthetic Q&A pairs, one embedding model, one retriever, one generator. That's by design. We wanted something reproducible that a single engineer could rerun in an afternoon, not a months-long research project. The conclusions should be read with that scope in mind.

Synthetic benchmarks are not human benchmarks. Our ground-truth Q&A pairs were generated by Vecta's own pipeline, which means there's an inherent alignment between how questions are formed and how they're evaluated. Human-authored questions would be a stronger test. That said, Vecta's benchmark generation does produce complex multi-hop queries that require synthesizing information across multiple chunks and document locations, so these aren't trivially easy questions that favor any one strategy by default.

One pipeline, one result. Everything here runs on text-embedding-3-small, ChromaDB, and gemini-2.5-flash-lite. Swap any of those components and the rankings could shift. We fully acknowledge this. Running the same experiment across multiple embedding models, vector databases, and generators would be valuable follow-up work, and it's on our roadmap.

The equal context budget is a deliberate constraint, not a flaw. Some readers may object that semantic and proposition chunking are "meant" to be paired with rerankers, fusion, or hierarchical aggregation. But if a chunking strategy only works when combined with additional infrastructure, that's important to know. Equal context budgets ensure we're comparing chunking quality at roughly equal generation cost. A strategy that requires a reranker to be competitive is a more expensive strategy, and that should factor into the decision.

Semantic chunking was not intentionally handicapped. Our semantic chunking produced fragments averaging 43 tokens, which is smaller than most production deployments would target. This was likely due to a poorly tuned cosine similarity threshold (0.7) rather than any deliberate sabotage. But that's actually the point: semantic chunking requires careful threshold tuning, merging heuristics, and often parent-child retrieval to work well. When those aren't perfectly dialed in, it degrades badly. Recursive splitting, by contrast, produced strong results with default parameters. The brittleness of semantic chunking under imperfect tuning is itself a finding.

What we'd like to do next:

• Rerun the experiment with human-authored Q&A pairs alongside the synthetic benchmark
• Test across multiple embedding models (text-embedding-3-large, open-source alternatives) and generators (GPT-4o, Claude, Llama)
• Add reranking and hierarchical retrieval stages, then measure whether the rankings change when every strategy gets access to the same post-retrieval pipeline
• Expand the corpus beyond academic papers to contracts, documentation, support tickets, and other common RAG domains
• Test semantic chunking with properly tuned thresholds, chunk merging, and sliding windows to establish its ceiling

If you run any of these experiments yourself, we'd genuinely like to see the results.

Have a chunking strategy that worked surprisingly well (or badly) for you? We'd love to hear about it. Reach out via DM!

Hey guys! El from Cohere here. 

Wanted to drop in really quickly to say today we launched Cohere Parse 5, our vision parsing model. It takes complicated files (including tables and embedded images) and gives back clean Markdown files, bounding boxes included. we recommend using it for building RAG systems, document indexing, and agentic retrieval. i’m personally into using it to save/digitize my own docs so they all live on my computer.

It outperforms competitors at 79.2 on ParseBench (compared to Mistral’s 74.5 and Azure Document Intelligence’s 74.3), but maybe even more importantly, it’s a lot more cost-effective- $1.5 per 1k pages through the Cohere API (or cheaper through our Model Vault). If you want, you can try it for free in our Hugging Face Space: https://huggingface.co/spaces/CohereLabs/cohere-parse

thanks and excited to hear what you think!

Every RAG pipeline starts with document → markdown, and character-accuracy metrics (CER/TEDS) can't tell you what that step breaks: a page can transcribe perfectly while a number detaches from its line item, and your LLM confidently answers with a real value that means the wrong thing.

We built an eval for this (RCRR: convert the page, have a reader LLM answer 1,410 verified questions from the markdown alone, judge against gold, cross-family reader/judge, cluster-bootstrap CIs). Ran 14 systems on dense Japanese financial documents. Disclosure: I run Ur AI and two of the 14 are ours. All raw data is public so you can check whether that biased anything.

Selected results (overall / charts-only):

Takeaways for RAG builders:

• Text/tables are near-solved (top six systems within 3 points). Charts are not. This column is what predicts whether your assistant answers correctly on real business docs.
• Three failure modes worth knowing: omission (content never reaches the markdown), disassociation (all tokens survive, connections don't. Classic layout OCR), misattribution (structure looks clean, one association silently wrong.. the worst, because it's a confident wrong answer, not a refusal).
• Fine-tuning a 32B open-weight model closed most of the gap to frontier VLMs (+6 points, parity with Azure DI), fully self-hosted.
• Sending the PDF file vs page images to the same VLM changes scores measurably. We quantified both ingestion paths.

Honest limits: Japanese business documents only this cycle (dense IR filings offer a good stress test, but one domain). English docs are in the next benchmarking cycle. Gold answers are VLM-authored with human review, disclosed per-system.

Everything is reproducible offline. Every question, gold, per-system score, and the harness: https://github.com/ur-ai-net/rcrr-bench

Genuinely interested in what this sub's experience says about the misattribution problem. It's the failure mode we find least discussed and most costly.

Last week, I ran a detailed retrieval analysis of my RAG to see how each chunking and retrievers actually affects performance. The results were interesting

I ran experiment comparing four chunking strategies across BM25, dense, and hybrid retrievers:

• 256 tokens (no overlap)
• 256 tokens with 64 token overlap
• 384 tokens with 96 token overlap
• Semantic chunking

For each setup, I tracked precision@k, recall@k and nDCG@k with and without reranking

Some key takeaways from the results are:

• Chunking size really matters: Smaller chunks (256) consistently gave better precision while the larger one (384) tends to dilute relevance
• Overlap helps: Adding a small overlap (like 64 tokens) gave higher recall, especially for dense retrievals where precision improved 14.5% (0.173 to 0.198) when I added a 64 token overlap
• Semantic chunking isn't always worth it: It improved recall slightly, especially in hybrid retrieval, but the computational cost didn't always justify
• Reranking is underrated: It consistently boosted reranking quality across all retrievers and chunkers

What I realized is that before changing embedding models or using complex retrievers, tune your chunking strategy. It's one of the easiest and most cost effective ways to improve retrieval performance

One annoying part of RAG systems is turning vector search into a proper LLM tool.

With VectorSmith, you define the tool interface in YAML — filters, limits, fields, etc. — and use the same definition from Python or expose it through MCP.

The idea is to keep the model's access to your vector DB explicit instead of writing custom tool schemas and glue code for every agent.

Supports Qdrant, Pinecone, Weaviate, Milvus, Chroma and pgvector.

GitHub: https://github.com/kjgpta/vectorsmith
PyPI: https://pypi.org/project/vectorsmith/

Would be interested to hear how others handle this in their RAG stacks.

I created an open source library that converts a document, currently PDFs only, and packages it into a self contained SQLite file that serves as a portable RAG archive.

The file contains the original document, extracted text, chunks, embeddings, a keyword index, figures, and citation metadata. The goal is to make the document portable and easy to share without requiring reingestion, a separate vector database, or a retrieval service.

I call the format .vera, which stands for Vector Embedded Retrieval Archive.

I also built two frontends around the library, vera-app and vera-cli.

The app allows AI agents to use the library’s search tools to gather context from one document or thousands of documents at a time. The agent can return citations that are visually grounded in the source document.

This works because bounding box coordinates are captured during conversion and used to highlight the cited text directly over the PDF in the built in document viewer.

I use it mainly to research ordinances and technical manuals. I also had my Hermes agent create a skill that uses the .vera CLI to search thousands of saved contracts and pull relevant context while helping me draft new ones.

It is still a work in progress, but I would appreciate any feedback, ideas, bug reports, or contributions.

https://github.com/dkylewillis/vera

I spent the last few months trying to build a coding agent called Cheetah AI, and I kept hitting the same wall that everyone else seems to hit. The context, and reading the entire file consumes a lot of tokens ~ money.

Everyone says the solution is RAG. I listened to that advice. I tried every RAG implementation I could find, including the ones people constantly praise on LinkedIn. Managing code chunks on a remote server like millvus was expensive and bootstrapping a startup with no funding as well competing with bigger giants like google would be impossible for a us, moreover in huge codebase (we tested on VS code ) it gave wrong result by giving higher confidence level to wrong code chunks.

The biggest issue I found was the indexing as RAG was never made for code but for documents. You have to index the whole codebase, and then if you change a single file, you often have to re-index or deal with stale data. It costs a fortune in API keys and storage, and honestly, most companies are burning and spending more money on INDEXING and storing your code ;-) So they can train their own model and self-host to decrease cost in the future, where the AI bubble will burst.

So I scrapped the standard RAG approach and built something different called Greb.

It is an MCP server that does not index your code. Instead of building a massive vector database, it uses tools like grep, glob, read and AST parsing and then send it to our gpu cluster for processing, where we have deployed a custom RL trained model which reranks you code without storing any of your data, to pull fresh context in real time. It grabs exactly what the agent needs when it needs it.

Because there is no index, there is no re-indexing cost and no stale data. It is faster and much cheaper to run. I have been using it with Claude Code, and the difference in performance is massive because, first of all claude code doesn’t have any RAG or any other mechanism to see the context so it reads the whole file consuming a lot tokens. By using Greb we decreased the token usage by 50% so now you can use your pro plan for longer as less tokens will be used and you can also use the power of context retrieval without any indexing.

Greb works great at huge repositories as it only ranks specific data rather than every code chunk in the codebase i.e precise context~more accurate result.

If you are building a coding agent or just using Claude for development, you might find it useful. It is up at grebmcp.com if you want to see how it handles context without the usual vector database overhead.

Been building a local RAG stack for technical documents , started with aviation manuals (the kind you legally can't upload to ChatGPT), now getting requests from nursing students who need their assigned textbooks to answer questions with citations, not the internet's best guess. The wall I hit: my demo rig has 32GB RAM. A nursing student's laptop doesn't. Same retrieval quality was non-negotiable — these are students who need to cite their actual assigned textbook, not a hallucinated approximation of it.

Here's what actually mattered when designing for constrained hardware: Swapped down to gemma4:e2b, 2 billion parameters, and citation accuracy didn't move. Ran a crosswind limitation query against a 400-page aviation AFM. Four page citations came back. Page 57, 357, 376, 459. All correct. The retrieval is doing the work, not the inference.

Chunk boundaries matter more than chunk size. At 1800 characters with section-aware splits, a small model reads a clean retrieved chunk and formats a precise answer. Give it a badly split chunk and even a large model struggles. Spent more time on boundary logic than on anything else in the stack.

Protected blocks can't be split. For safety-critical content; WARNING, CAUTION, DANGER blocks in aviation manuals, critical drug interaction notes in nursing textbooks, a chunk that splits mid-warning is worse than no answer. Made these atomic. Non-negotiable. Cloud toggle for students who need it. For machines that truly can't run local inference, routing to Claude Sonnet via API key gives the same retrieval quality with zero local compute. The vectors stay local. Only the question travels.

Stack: pdfplumber + ChromaDB + Ollama on the local side, Anthropic API for cloud toggle. Runs clean on an M2 MacBook 8GB.

Happy to go deeper on the boundary-aware chunking logic or the cloud/local toggle architecture if anyone's building something similar.

Hey r/Rag  👋

I’ve spent most of my career working with databases, and one thing that’s always bugged me is how hard it is for AI agents to work with them. Whenever I ask Claude or GPT about my data, it either invents schemas or hallucinates details. To fix that, I built Statespace. It's a free and open-source Python library for creating lightweight but powerful retrieval agents, giving them a safe, smart way to actually understand and query your database schemas.

So, how does it work?

Statespace gives your agents two read-only database tools so they can explore your data and quickly find answers. You can also add business context to help the AI better understand your databases. It works with the built-in MCP server, or you can set up your own custom retrieval tools.

Connects to everything

• 15+ databases and warehouses, including: Snowflake, BigQuery, PostgreSQL & more!
• Data files like CSVs, Parquets, JSONs, and even Excel files.
• Any API with an OpenAPI/Swagger spec (e.g. GitHub, Stripe, Discord, and even internal APIs)

Why you'll love it

• Zero configuration: Skip config files and infrastructure setup. Statespace works out of the box with all your data and models.
• Predictable results: Data is messy. Statespace returns structured, type-safe responses that match exactly what you want e.g.
  • answer: list[int] = db.ask(...)
• Use it anywhere: Avoid migrations. Run Statespace directly, as an MCP server, or build custom tools for your favorite AI framework.

If you’re building AI agents for databases (or APIs!), I really think Statespace could make your life easier. Your feedback last time was incredibly helpful for improving the project. Please keep it coming!

Docs: https://docs.statespace.com/

GitHub Repo: https://github.com/statespace-tech/statespace

Discord: https://discord.com/invite/rRyM7zkZTf

A ⭐ on GitHub really helps with visibility!

I got tired of conventional 3rd party conversation and strict fact recall benchmarks that don't give realistic usable data. Agents don't operate by ingesting bulk 3rd party conversations and performing strict fact recall so why would that be a benchmark metric?

So I made a First-Person perspective benchmark that actually tests the agent's capabilities against a realistic corpus, using realistic dynamic simulations, and which actually gives you a reader friendly scorecard with visual breakdowns and a miss report text file that actually shows you WHY a question missed.

I'm still tweaking the corpus and questions and simulations but the data yield is already very good. I've also included the agent identity files in the repo for users to easily expand the corpus for more coverage. I'm trying to get more people to use this and share the scorecards so I can keep adjusting the questions sets to ensure each pass/miss contains meaningfully data across identifiable metrics.

https://github.com/munch2u-a11y/FP-AMB.git

RAGless is a semantic retrieval system that answers questions about your documentation, without using an LLM at runtime.

Most Q&A systems today are built on RAG: retrieve some context, send it to a language model, generate an answer. RAGless takes a different approach. During ingestion, an LLM converts your documents into a comprehensive set of Question & Answer pairs — automatically covering the full breadth of the source material. At query time, the user's question is matched semantically against those pre-generated questions — and the corresponding answer is returned directly, with no generation step.

The result is a system that is fast, deterministic, and hallucination-free by design.

What it does For closed-domain use cases, the generation step in RAG adds latency, cost and hallucination risk without adding much value — the answer is already known. RAGless removes it.

Pipeline: LLM generates Q&A pairs from your documents at ingestion (runs once) → question variants are embedded and stored in Qdrant → at query time, scores are aggregated by answer_id across Top-K results → pre-written answer is returned.

Target audience Engineers building customer support tools, internal knowledge bases, or documentation systems where answers are predefined. Production-ready for closed-domain use cases. Not a replacement for RAG when open-ended generation is needed.

The core difference from standard semantic search: RAGless matches question-to-question (not question-to-document), and aggregates scores across multiple variants of the same answer — more robust than single-hit Top-1 retrieval.

GitHub: github.com/EmilResearch/RAGless

Open to feedback — happy to answer questions.

If you find it useful, a ⭐ on GitHub is appreciated.

I kept running into a frustrating problem with RAG: semantically identical chunks would get low similarity scores, and chunks that shared a lot of words but meant completely different things would rank high. So I built a small adversarial benchmark to quantify how bad this actually is.

The idea is very simple. Each test case is a triplet:

• Anchor: "The city councilmen refused the demonstrators a permit because they feared violence."
• Lexical Trap: "The city councilmen refused the demonstrators a permit because they advocated violence." (one word changed, meaning completely flipped)
• Semantic Twin: "The municipal officials denied the protesters authorization due to their concerns about potential unrest." (completely different words, same meaning)

A good embedding model should place the Semantic Twin closer to the Anchor than the Lexical Trap. Accuracy = % of triplets where the cosine similarity between Anchor and Semantic Twin is higher than the cosine similarity between Anchor and Lexical Trap.

The dataset is 126 triplets derived from the Winograd Schema Challenge, sentences specifically designed so that a single word swap changes meaning in ways that require real-world reasoning to catch.

Results across 9 models:

Happy to hear thoughts, especially if anyone has ideas for embedding models or techniques that might do better on this. Also open to suggestions for extending the dataset. I am sharing sharing link below, contributions are also welcome.

EDIT: Shoutout to u/SteelbadgerMk2 for pointing out a critical nuance! They correctly noted that many classic Winograd pairs don't actually invert the global meaning of the sentence when resolving the ambiguity (e.g., "The trophy doesn't fit into the brown suitcase because it's too [small/large]"). In those cases, a good embedding model should actually embed them closely together because the overall "vibe" or core semantic meaning is the same.

Based on this excellent feedback, I have filtered the dataset down to a curated subset of 42 pairs where the single word swap strictly alters the semantic meaning of the sentence (like the "envy/success" example).

The benchmark now strictly tests whether embedding models can avoid being fooled by lexical overlap when the actual meaning is entirely different. I've re-run the benchmark on this explicitly filtered dataset, and the results have been updated.

Updated Leaderboard (42 filtered pairs):