Projects /
AI Stack
A private AI platform on hardware in the lab: local models, retrieval over my own notes, agents and research workflows, all behind one gateway.
The drawing
This is the public version of the system schematic, with addresses and internal risk notes removed. It’s a living document — I revise it every time the stack changes.
How it fits together
Main inferenceDexter — Ryzen AI 9 HX 370, 96 GB RAM, RTX 5070 12 GB. LM Studio serves openai/gpt-oss-20b with 64K context. It’s the keeper node: all private data lives here.
Vision + fallbackGadget — Ryzen 9 8945HX, RTX 5060 8 GB, running Qwen3-VL 8B for images and OCR, and answering when Dexter can’t.
GatewayLiteLLM with Postgres and Presidio. Every request from a client, agent or workflow enters here; message logging is off.
Front doorAnythingLLM with persona workspaces (a Chief-of-Staff, a deep-research analyst, a restricted workspace) and retrieval over my Obsidian vault.
Automationn8n flows for a research library, a morning brief and a report outbox; CrewAI for multi-agent tasks.
ResearchVane (formerly Perplexica) with SearXNG, backed by a ~300 GB private YaCy index so most searches never leave the LAN.
PlatformUnprivileged Ubuntu 24.04 containers on a 4-node Proxmox cluster, iSCSI storage on a QNAP array, backups to Proxmox Backup Server.
AccessRemote use over Tailscale, plus a Telegram bot for quick questions from the phone.
Rules the stack lives by
- Private data stays on hardware I own. The only traffic that leaves the lab is search queries.
- One way in. Clients, agents and workflows all go through the gateway, so keys, routing and guardrails live in one place.
- No personal information goes to cloud AI. If a cloud model is ever wired back in, it sits behind PII masking that fails closed.
- Work and patient data never enter this stack. Full stop.
- Fast beats big. The default model is the one that answers quickly; bigger models are an optional slow lane.
Field notes
Three rebuilds later: what stuck
Memory layers that poisoned themselves, a runtime swap that fixed ‘model’ crashes, and why one GPU can’t do everything. AI Stack
Why my AI stack lives at home
Two GPU boxes, one gateway, and a few hard rules about where data is allowed to go. AI Stack