AI Node

Why Local

Every prompt you send to a cloud LLM is logged, analyzed, and potentially used for training. When you’re asking about architecture decisions or reviewing internal code, that’s not just a privacy concern — it’s a real one.

Modern GPUs are capable enough and models like Llama have made local inference practical. The setup isn’t complicated, and the freedom is worth the tradeoffs.

How It Works

The inference runs on a separate machine with a CUDA GPU — llama.cpp server with quantized GGUF models, mostly Q4_K_M. Good enough for reasoning and code review, not great for anything requiring factual precision. That’s not the use case though.

Open WebUI runs on my main homelab box and points to the llama.cpp server over the network. It’s the interface — clean, familiar, no vendor lock-in.

LocalAI runs as a second option on the main box, capped to one model at a time. It gives me access to a wider range of models without the memory overhead of keeping them all loaded.

CouchDB handles Obsidian sync — my notes stay in sync across devices and feed directly into the Hermes agent context. Nothing goes through a cloud sync service.

How I Actually Use It

Not as a chat toy — as a working tool.

When I’m at a desk, I run OpenCode on my laptop pointing to the local model. It reads code, suggests refactors, and helps me navigate unfamiliar codebases. Everything stays on the network.

When I’m out, my Hermes agent (Sage) runs on the homelab and I talk to it through Matrix. It has tools — terminal access, file reading, web search — so it’s not just a text completion engine. It can actually do things.

What Works

• Code review and refactoring suggestions
• Drafting technical documents and emails
• Explaining unfamiliar codebases
• Brainstorming architecture approaches

What Doesn’t

• Anything requiring factual accuracy without verification
• Long-context tasks beyond the model’s window
• Anything you’d trust without reading the output first

The local-first constraint means I can experiment freely. If a model hallucinates or gives bad advice, it doesn’t leak to anyone. That freedom is the whole point.