I’ll try explaining using an analogy (though I can go nerd mode if that’s better? Let me know; I’m assuming an intelligent lay audience for this but if you want nerd-core, my body is ready lol).

PS: Sorry if scattered - am dictating using my phone (on holiday / laptop broke).

---

Hallucinations get minimized the same way a teacher might minimise a student from confidently bullshitting on their book reports: you control context (what they’re allowed to talk about), when they’re allowed to improvise, and you make them show their work when it matters by doing a class presentation.

Broadly speaking, that involves using RAG and GAG (of your own documents) as “ground truth”, setting temperature low (so LLM has no flights of fancy) and adding verifier passes / critic assessment by second model.

Additionally, a lot of hallucinations come from the model half-remembering something that isn’t in front of it and then “improvising”.

To minimise that, I coded a little python tool that forces the llm to store facts verbatim (triggered by using !!) into a JSON (text) file, so that when you ask it something it recalls it exactly as a sort of rolling memory. The basis of that is from something I made earlier for OWUI

https://openwebui.com/posts/total_recall_4a918b04

So what I have in place is this -

I use / orchestrate a couple of different models, each one tuned for a specific behaviour. They work together to produce an answer.

My python router then invokes the correct model for the task at hand based on simple rules (is the question over 300 words? Does it have images? Does it involve facts and figures or is it brain storming/venting/shooting the shit?)

The models I use are

• Qwen 3-4B 2507 Instruct (usual main brain)
• Phi-4-mini (critic)
• Nanbeige 3B (2nd main brain when invoked / shit shooter)
• You-tu LLM (coding stuff)
• Qwen3-VL-4b (visual processing)
• Qwen3-8b (document summariser)
• Qwen3-1.7b (court jester that when invoked rewrites “main brain” output with contextually appropriate Futurama, Simpsons, Firefly etc quotes. With blackjack. And hookers!).

To give a workflow example - you ask a question.

The python router decides where it needs to go to. Let’s suppose its a technical look up / thinking about something in my documents.

The “main brain” generates an answer using whatever grounded stuff you’ve given it access to (in Qdrant database and JSON text file). If no stored info, it notes that explicitly and proceeds to next step (I always want to know where it’s pulling it’s into from, so I make it cite its references).

That draft gets handed to a separate “critic” whose entire job is to poke holes in it. (I use very specific system prompt for both models so they stay on track).

Then the main brain comes back for a final pass where it fixes the mistakes, reconciles the critique, and gives you the cleaned‑up answer.

It’s also allowed to say “I’m not sure; I need XYZ for extra context. Please provide”.

It’s basically: propose → attack → improve.

Additionally, I use a deterministic memory system (basically just a python script that writes to a JSON / text file that the LLM writes exactly into and then retrives exactly out from), without editorialising facts of a conversation in progress.

Facts stored get recalled exactly without llm massage or rewrite.

Urgh, I hope that came out OK. I’ve never had to verbally rubber-duck (explain) it to my phone before :)

---

TL;DR

Hallucinations minimised by -

1. Careful fact scraping and curation (using Qdrant database, markdown text summaries and rolling JSON plain text facts file)

2. Python router that decides which LLM (or more accurately, SLM, given I only have 8GB VRAM) answers what, based on simple rules (eg: coding questions go to coder, science questions go to science etc)

3. Keeping important facts outside of the LLM, that it needs to reference directly (RAG, GAG, JSON rolling summary).

4. Setting model temperatures so that responses are as deterministic as possible (no flowery language or fancy reinterpretations; just the facts, ma’am).

5. Letting the model say “I don’t know, based on context. Here’s my best guess. Give me XYZ if you want better answer”.

Basic flow:

ask question --> router calls model/s --> “main brain” polls stored info, thinks and writes draft --> get criticized by separate “critic” --> “main brain” gets critic output, responds to that, and produces final version.

That reduces “sounds right” answers that are actually wrong. All the seams are exposed for inspection.

Yes :)

Yep! I will mirror it here -

https://github.com/BobbyLLM

(Its empty rn / place holder only).

I had a bunch of prelim write-ups on r/LocalLLM and r/LocalLlama and r/homelab but they’re in the shadowrealm now due to reddit ban (fuck reddit)

I will also post it on @homelabs and @privacy here; I think my MoA design is worthwhile enough to maybe even merit a post on HackerNews…but I want to cross all T’s and dot all I’s before I get into that bar fight lol.