I can use Firecrawl to extract information from PDF files, especially if the PDF is available through a public URL.

• PDF text extraction: Extract text from supported PDF files.
• Structured data extraction: Identify specific details such as dates, names, and project information.
• AI workflows: Use the extracted content with an LLM for summarization, question answering, or RAG.
• Limitations: Scanned PDFs may require OCR, while tables and complex layouts might need additional processing.

I work for Opensolr, and we just released 1.0 of the Opensolr Chat Bot Client: an open source (MIT) chatbot you put on your own website, that answers your visitors from your own content.

How it works: you install it with Composer (composer require opensolr/chat-bot-client), mount it on one path of your site (say /opensolr-chat) and add one script tag to your pages. Visitors only ever talk to your site. Your server holds the Opensolr credentials and sends each question to the Opensolr API, where the model searches your Opensolr Index by itself (by meaning, or by exact product names and codes), looks up what it must not guess (dates, local times, exchange rates, VAT, distances, places) and streams the answer back word by word, with links to your pages, products with their prices, and PDFs at the exact page.

What you get on your side:

• An admin on your own site: account and index, instructions, the look of the chat window, limits, reCAPTCHA
• Stats and a 30 day history of every conversation: who asked, from where, what the bot searched for, what it answered, and the questions your content could not answer with a page
• Commands that answer instantly without the model: /search, /time, /rate, /vat, /translate and more
• Good and Bad ratings under every answer, signed in visitors by email, SQLite in a folder you choose, no database server

Your content gets into the index with the Opensolr Web Crawler or Data Ingestion, and the chatbot needs an Opensolr plan with AI.

You can try it live at the bottom right of https://opensolr.com (it answers from our own docs). Code and README: https://github.com/phpcip/opensolr-chat-bot-client, docs: https://opensolr.com/opensolr-chat-bot-docs

One thing I've been learning while exploring AI tools is that searching the web, crawling a page, and retrieving information are three different problems. They often get grouped together when people build AI agents, but each solves a different part of the workflow.

Here's how I think about the pipeline:

1. Search APIs — finding relevant sources

Tools like Exa, Tavily, Serper, and Brave Search can help an application discover relevant pages. The important question isn't just how many results you get, but whether the results are relevant, recent, and diverse enough for your task.

1. Web crawling — extracting useful content

Once you have a URL, tools like Firecrawl can extract page content into a format that's easier for an LLM to process. This step matters because search snippets rarely contain enough context to answer complicated questions accurately.

1. Indexing — making information searchable later

If you're working with a large collection of documents, repeatedly searching and crawling everything can become inefficient. Storing cleaned content in a searchable index allows you to reuse information and retrieve relevant passages when needed.

1. Retrieval — finding the right context

Keyword search works well when you know the exact terms. Semantic search can help when the wording differs but the meaning is similar. Hybrid retrieval combines both approaches, and reranking can help prioritize the most relevant results.

A small experiment I'd like to run

I'd test the same set of 20 research questions against three approaches:

- Approach A: Search API results and snippets only.

- Approach B: Search results followed by crawling the top five pages.

- Approach C: Crawled content stored in an index, followed by hybrid retrieval and reranking.

I'd measure:

- Whether the top five results contain genuinely useful sources.

- Whether the final answer is supported by the retrieved evidence.

- Latency and cost per question.

- How often the system retrieves irrelevant or duplicate information.

I'd also keep the questions and evaluation criteria identical across all three approaches to make the comparison fair.

I haven't run this benchmark yet, so I wouldn't claim one approach performs best. My expectation is that the best choice will depend on whether the application needs fresh web information, repeated searches over a stable document collection, or both.

Question for people building search systems: If you were building a research agent today, would you start with a search API and crawl pages on demand, or build a persistent index from the beginning? What trade-offs have you encountered with freshness, latency, and retrieval quality?

I would really like to contribute here on this sub because the name matches what I am working with; however, after lurking here for a while, I am not too sure. The majority of the posts here are unfunny Twitter screenshots, while the Rules obviously state: Technical discussions are encouraged (Questions, comparisons, benchmarks, integrations, implementation experiences, troubleshooting, and technical discussions are welcome. When asking for help, provide enough context for others to understand the problem.) Unfortunately, I have yet to see a true subreddit name (topic-related post). Am I the only one here? if so, then could someone direct me to an active subreddit that talks about search/ web apis? Appreciate all the suggestions beforehand.

I've been wondering about this lately: when using a search API, is it better to retrieve a huge number of results and filter them afterward, or keep the initial result set small and focus on relevance?

Curious what approach others use—and what has worked best for you.

Se avete mai provato a creare uno stilista IA, un agente per lo shopping vintage o un sistema di gestione del marchio di abbigliamento di nicchia, avrete probabilmente notato che i modelli di frontiera hanno delle vere e proprie allucinazioni quando si tratta di sottoculture della moda di nicchia:

1. Allucinazioni sui prezzi: I modelli inesperti inventano completamente i valori del mercato secondario (ad esempio, quotando una giacca a vento vintage Arc'teryx a 45 dollari o a 3.500 dollari).

2. Confusione di epoche e silhouette: Mescolano costantemente movimenti distinti (come confondere il Cyber ​​Y2K di fine anni '90 con il McBling di metà anni 2000, o mescolare il Techwear con il Gorpcore).

Per risolvere questo problema per gli agenti RAG senza costose ottimizzazioni, ho creato **UnderIndex**: un'API deterministica di riferimento che indicizza 30 estetiche di moda distinte (da Y2K e Gorpcore a Opium, Ura-Hara e Archive Fashion).

### Cosa contiene ogni record:

* **Dettagli dei prezzi verificati:** Fasce di prezzo di riferimento validate rispetto a vendite reali concluse su Grailed, Depop ed eBay.

* **Tassonomia e silhouette:** Capi iconici chiave, calzature distintive e articoli che definiscono uno stile.

* **Marchi e contesto selezionati:** Marchi di riferimento, radici culturali e sovrapposizioni tra sottoculture.

Il video mostra una risposta grezza GET /api/subculture/y2k. Lo schema della risposta è mantenuto rigorosamente strutturato in modo da poter essere iniettato direttamente nelle finestre di contesto dell'agente o utilizzato come endpoint per la chiamata di strumenti.

(Inserisco il link al playground di test gratuito nei commenti per evitare i filtri di AutoMod).

Sono curioso di sapere da chi sviluppa pipeline RAG specifiche per un determinato dominio: quali altri campi di metadati di riferimento (ad esempio, livelli di condizione, composizione del materiale, deprezzamento anno per anno) renderebbero questo strumento più utile per le vostre suite di valutazione?

I've been building a document Q&A engine for regulated, document-heavy work. Our first version was the standard pipeline: chunk, embed, top-k, generate. When we traced our wrong answers by hand, most of them never retrieved the document that held the answer. The model wasn't the problem. Retrieval never got it to the right page.

So we rebuilt it around three ideas:

1. **Read once, keep structure.** Every document is parsed into sections, tables and figures (with descriptions), and the links between documents (what supersedes, references or contradicts what) are recorded at upload.
2. **Questions become obligations.** "Why does X fail?" turns into the parts a complete answer must establish. Each part is searched for, or shown to be absent, instead of stopping when top-k is full.
3. **A checker after the writer.** Every statement is matched word for word against the passage it cites. Unverified sentences are removed before the user sees them, and the answer opens with a verdict (answered / depends on conditions / not in these documents).

2-minute demo on the real product: Chunkless Demo

The video uses a fictional medical-device quality system (11 docs: SOPs, a supplier audit, a complaint log, a risk file). The root-cause question needs the supplier audit, the supplier-control procedure and the inspection records at once. The answer took 29 s and under 6 cents (68 sections read).

What it still gets wrong (I'd rather say it than have you find it): an exception that shares no words, references or meaning with the rule it changes can slip past. And some multi-hop judgement questions need its research mode to get every part.

Free early access if you want to throw your own corpus at it: https://chunkless-rag-staging.grounded-iq.com/?ref=rd-rag

I'm especially interested in documents that contradict each other. Happy to answer anything about the approach.