October 26, 2026 / AI Stack

RagFlow vs AnythingLLM memory

I ran AnythingLLM as the brain of my self-hosted AI stack for a long time. It worked well until it didn’t. Then it started making things up, and I spent far too long fixing the wrong layer. This post is the lesson I paid for in time, plus the research I wish I had done first.

How it went wrong

My setup isn’t exotic: AnythingLLM in Docker, LM Studio serving a local model on an RTX 5070, and an Obsidian vault pushed into AnythingLLM through its API. About 95% of what my assistant knows comes from my own notes: projects, infrastructure, work history, personal details.

For months it was great. Then the network grew, I added a second AnythingLLM instance and a LiteLLM router in front of my models, and the answers started drifting. At first they were vague and generic. Then they were confidently wrong. I asked one instance for my dog’s name and it said “Bella.” That’s not my dog’s name. I asked again and got a different wrong name. When I asked it to pull together reports from the work material in my vault, like an outline of projects that had failed, I got shallow answers that read like guesses.

The costly mistakeI treated it as a model problem. I rewrote system prompts. I built new personas, including a “deep research analyst” whose only job was to stop making things up. I changed models. I rebuilt parts of the stack. Each change seemed to help for a day, then the problem came back. I was tuning the voice of a system whose real problem was what it was being handed to read.

Here’s what I eventually understood. A RAG system hallucinates in a specific way: when retrieval brings back the wrong chunks, or none, the model fills the gap with something plausible. No prompt fixes that, because the model is doing exactly what it was asked to do with the context it was given. I’d been debugging the last stage of a pipeline when the failure was upstream.

So I stopped tweaking and started researching. Should I move retrieval to a dedicated engine like RAGFlow? Or was the common advice to “graduate to RAGFlow” going to cost me even more time?

What most people recommend

Almost every comparison lands in the same place. AnythingLLM is the easy, friendly option for personal documents and small teams. RAGFlow is the “serious” engine for organizations that need precise retrieval over large, messy document sets. Reviewers usually describe AnythingLLM as a thin layer over standard chunking and RAGFlow as real infrastructure, and the main reason they give for RAGFlow is its DeepDoc parser, which keeps tables, scans and complex PDF layouts intact.

The specs back up the general picture:

	AnythingLLM	RAGFlow
Chunking	One strategy for every file type: a recursive character splitter. Defaults are 1,000 characters with only 20 characters of overlap.	Template-based chunking per document type, with a visual chunk view you can inspect and correct.
Retrieval	Vector similarity only. A handful of chunks (typically 4–6) go to the model.	Hybrid: keyword (BM25-style full text) plus vector search, with optional reranking.
Footprint	Light. One container.	Multi-container stack. Official floor is 4 cores, 16 GB RAM, 50 GB disk, plus kernel tuning for Elasticsearch.
Use from other tools	Built around its own chat UI and workspaces.	Retrieval API that returns raw chunks, an MCP server, and OpenAI-compatible endpoints.

All of that is accurate. But the advice behind it assumes a type of corpus and a type of failure that may not be yours. Here are four cases where following it backfires.

Four ways the common advice backfires

Scenario 1 · Paying for a feature you can’t use

Your corpus is already clean Markdown

DeepDoc is RAGFlow’s main selling point, and it exists to rescue structure from scanned contracts, tables and odd PDF layouts. An Obsidian vault has none of that problem. Markdown already marks its structure with headings and lists. If your knowledge base is mostly notes, you’d take on a 16 GB minimum, Elasticsearch tuning and several containers for a parser with nothing to do.

What actually helps with Markdown: chunking on headings, and adding keyword search alongside vector search. RAGFlow does provide that keyword search, and it matters (see below). But if you pick RAGFlow because of DeepDoc, you’ll judge it on the wrong thing.

Scenario 2 · Technical debt with a deadline

Building integrations right before a rewrite

RAGFlow is in the middle of a major transition. The v1.0 release candidate is a full rewrite in Go. The data migration from v0.27.2 is one-way: you can’t roll back. Deprecated API aliases are removed, and early v1 builds are missing some features, including GPU support for DeepDoc and the team permission model. The upgrade also has to go in order. You can’t jump from an older 0.x straight to 1.0.

If you write your sync scripts, MCP glue and agent integrations against the 0.x API today, you’re writing code that will soon need rework. The Python SDK is the one layer the project says is unaffected.

Scenario 3 · Swapping tools to fix the wrong problem

The hallucination isn’t a retrieval-quality problem

This is the one I almost fell into. “Switch engines for better accuracy” assumes ranking quality is what’s failing. Two of my worst symptoms weren’t ranking problems at all:

The wrong dog’s name. My original instance answered correctly. The second instance, behind the router, answered wrong using the same model. That points to a misconfigured or nearly empty workspace, or a mismatched embedding model, on the second instance. Moving to a new RAG engine would hide that problem, not fix it.
“Outline all my failed projects.” That question asks for a list across the whole corpus. Standard top-k RAG fails at this no matter which engine you use, because the model only ever sees a few retrieved chunks. AnythingLLM’s own documentation gives almost this exact example as something not to expect.

The fix is structural: verify each instance’s workspace and embedder, and add metadata (for example, a status: failed frontmatter field or an index note) so listing questions have something to retrieve.

Scenario 4 · Accuracy features that cost more than they return

Small corpus, speed-first, sharing a box with the model

Reranking is the most-cited upgrade, and the evidence is more mixed than people admit. RAGFlow’s own team once removed its built-in rerankers, saying they barely improved retrieval while adding a lot of latency. Common guidance treats reranking as optional for small knowledge bases of under about a hundred documents.

The resource cost is real too. One bug report describes a handful of parallel PDF embedding jobs pushing RAGFlow to roughly 20 GB of RAM that wasn’t released afterward. If you run a fast local model and care about response time, putting that stack on the same machine as inference works against you. Give RAGFlow its own VM, and turn features on one at a time with a reason for each.

The evidence: what actually improves retrieval

The best controlled data I found is Anthropic’s Contextual Retrieval study. The useful part isn’t the headline number. It’s that the gains come in steps, with each technique adding its own share:

5.7%Baseline top-20 retrieval failure rate

3.7%+ context added to each chunk before embedding (~35% fewer failures)

2.9%+ keyword (BM25) search alongside vectors (~49%)

1.9%+ reranking (~67%)

Read these numbers honestlyThey come from Anthropic’s benchmark, and the full gain includes LLM-generated context added to every chunk. Plain hybrid search on its own gets you less than the full 49%. The direction is consistent, though: keyword search catches what embeddings miss, especially exact identifiers like hostnames, IP addresses, product names, error codes and people’s names.

That last point decided it for me. A homelab and work vault is full of exact identifiers. Vector-only search is weakest at exactly the questions I ask most.

Verdict

RAGFlow is the better retrieval layer, for a different reason than most people give.For a Markdown-heavy personal knowledge base, the reason isn’t DeepDoc. It’s hybrid keyword + vector search, which AnythingLLM doesn’t offer, and a retrieval API that lets any front end use the same knowledge base. AnythingLLM still earns its place as the chat interface. Split the roles: RAGFlow retrieves, AnythingLLM talks.

Two more reasons to decouple:

Front-end independence. RAGFlow returns raw chunks over its API and MCP, so a chat UI, an agent framework or a Telegram bot can all query the same source.
Sync. AnythingLLM’s live document sync is still a beta preview. In Docker it doesn’t watch uploaded files and can’t watch a whole folder, and the docs themselves warn about possible database corruption. An API-based push from Obsidian is more reliable and works with either engine.

Choosing your path

Option	What you do	Tradeoff
Best	Stand up RAGFlow on its own VM, test it with your real data, and hold off on permanent integrations until v1.0 is stable. Then build your sync and agent wiring against v1 only.	A few weeks of patience. No rework later.
Safer	Run both side by side and measure (method below). Keep AnythingLLM in production until RAGFlow proves itself on your questions.	A few evenings of testing. The decision is based on data, not reviews.
Budget	Stay on AnythingLLM and fix the cheap things: raise chunk overlap well above the 20-character default, check each instance’s workspace and embedder, and add metadata for listing questions.	Still vector-only, so exact-identifier recall stays weak.
Fastest	Point your existing tools at RAGFlow today through its Python SDK.	Usable now. Plan to retest when v1.0 lands.

Test before you switch

This is the step I skipped, and it would have saved me the most time. Before changing anything, build a small set of questions you already know the answers to:

Write 30 questions whose answers you know from your notes: about 10 exact-identifier questions (“what IP is the router on?”), 10 conceptual ones, and 10 that need facts from more than one note.
Run each question through every system you’re considering.
Score two things: did the right note show up in the top 5 retrieved chunks, and was the final answer correct. Keeping those separate tells you whether a failure is a retrieval problem or a model problem.
Rerun the same set after every config change. If a change doesn’t move the score, roll it back.

If the new engine doesn’t clearly win on the exact-identifier questions, it isn’t worth what it costs to run.

What I’d tell anyone running a local RAG stack

Hallucination in RAG is usually a retrieval symptom. Check what chunks the model was handed before you touch the prompt.
Change one layer at a time, and measure. Without a test set you’re guessing, and guessing is where the time goes.
Match the tool to your corpus, not to the reviews. Advice built for scanned PDFs doesn’t automatically apply to clean Markdown.
Separate retrieval from chat. When the knowledge base doesn’t belong to any one front end, you can swap either side without starting over.
Watch the release cycle before you build integrations. Pinning your scripts to an API that’s about to be rewritten is the most avoidable kind of technical debt.

I lost real time learning this the hard way. If this post saves you a few weekends of rewriting system prompts for a problem that was never in the prompt, it was worth writing.