ai

AI & LLM on Your Homelab — A Tutorial

llama.cpp Open WebUI Caddy Docker

This is a step-by-step guide to running your own AI inference stack at home. Not cloud APIs, not vendor lock-in — your own models, your own hardware, your own data.

Everything stays on your network.

Why Local First

Every prompt you send to a cloud LLM gets logged, analyzed, and potentially used for training. When you’re asking about architecture decisions or reviewing internal code, that’s a real privacy concern, not a theoretical one.

Modern GPUs are capable enough and open models have made local inference practical. The setup isn’t complicated, and the freedom is worth the tradeoffs.

What You’ll Build

A local inference server with a clean web interface, reverse proxy with TLS, and optional Matrix integration for remote access. All services communicate via DNS names — no port numbers to remember.

Core components:

  • llama.cpp — inference server with CUDA acceleration
  • Open WebUI — web frontend for chatting with models
  • Caddy — reverse proxy with automatic TLS via Cloudflare DNS
  • Tuwunel — Matrix homeserver (optional, for federation)
  • Hermes (Sage) — AI agent with tool access (optional)

Client tools (bonus):

  • OpenCode — IDE integration from your laptop
  • pi — lightweight agent harness

Prerequisites

  • A machine with a CUDA GPU (NVIDIA RTX 3090 or similar)
  • Docker + Docker Compose
  • Cloudflare account with DNS management
  • Domain name(s) for services
  • Basic comfort with the terminal

Step 1: llama.cpp Server

The inference engine. Runs quantized GGUF models on your GPU.

Create docker-compose.yaml:

services:
  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: llamacpp
    restart: unless-stopped
    networks:
      - cadnet
    volumes:
      - /path/to/ai-models:/models
    deploy:
      resources:
        limits:
          memory: 8G
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    command: >
      -m /models/YOUR_MODEL.gguf -c 8192 --host 0.0.0.0 --port 8080
      --n-gpu-layers 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0
      -b 256 -ub 256 --no-mmap -np 1
    environment:
      - MODEL_FILE=YOUR_MODEL.gguf

networks:
  cadnet:
    external: true

Notes:

  • Replace YOUR_MODEL.gguf with your actual model filename
  • Context size (-c) depends on your GPU VRAM — 8192 is safe for most 3090-class cards
  • Q4_K_M quantization gives good quality/speed tradeoff
  • Model files go in whatever directory you set as /path/to/ai-models

Getting models: Download GGUF models from HuggingFace. Look for quantized versions from trusted sources like TheBloke or Bartowski. For a starting point, models in the 7B-13B range work well on consumer hardware.

Step 2: Open WebUI

A clean web interface that connects to your llama.cpp server. No Ollama needed when pointing to a remote API.

Create docker-compose.yaml:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    networks:
      - cadnet
    volumes:
      - ./open-webui:/app/backend/data
    environment:
      - 'OLLAMA_BASE_URL=http://llamacpp:8080'
      - 'WEBUI_SECRET_KEY='
    restart: unless-stopped

networks:
  cadnet:
    external: true

Key configuration:

  • OLLAMA_BASE_URL points to your llama.cpp container on the Docker network
  • Open WebUI thinks it’s talking to Ollama, but llama.cpp serves a compatible API
  • Data persists in the ./open-webui volume

Step 3: Caddy Reverse Proxy

Caddy handles TLS automatically via Cloudflare DNS. One config, all services covered.

Docker Compose

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    environment:
      - CF_TOKEN=${CF_TOKEN}
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./conf:/etc/caddy
      - ./site:/srv
      - ./caddy_data:/data
      - ./caddy_config:/config
    networks:
      - cadnet

networks:
  cadnet:
    external: true

Set CF_TOKEN in your environment — this is a Cloudflare API token with DNS zone edit permissions.

Caddyfile

{
    acme_dns cloudflare {env.CF_TOKEN}
}

(llamacpp) {
    import cfdns
    encode gzip
    header Strict-Transport-Security max-age=63072000
    reverse_proxy llamacpp:8080
}

(openwebui) {
    import cfdns
    encode gzip
    header Strict-Transport-Security max-age=63072000
    reverse_proxy open-webui:8080
}

llm.yourdomain.com {
    import llamacpp
}

chat.yourdomain.com {
    import openwebui
}

Replace yourdomain.com with your actual domain. Pick whatever subdomain names make sense for you.

How it works:

  • Caddy automatically gets TLS certificates via Cloudflare DNS challenge
  • No manual cert management
  • Each service gets its own DNS name
  • Docker network name (llamacpp, open-webui) resolves automatically

Step 4: Optional — Matrix Federation

If you want to chat with your models over Matrix (federated, encrypted, your own server), Tuwunel is a lightweight Rust-based homeserver.

Create docker-compose.yaml:

services:
  homeserver:
    image: jevolk/tuwunel:latest
    restart: unless-stopped
    container_name: twunnel
    networks:
      - cadnet
    volumes:
      - ./db:/var/lib/tuwunel
      - ./tuwunel.toml:/etc/tuwunel.toml
    environment:
      TUWUNEL_SERVER_NAME: your-matrix-domain.com
      TUWUNEL_DATABASE_PATH: /var/lib/tuwunel
      TUWUNEL_PORT: 6167
      TUWUNEL_ALLOW_REGISTRATION: 'false'
      TUWUNEL_ALLOW_FEDERATION: 'true'
      TUWUNEL_TRUSTED_SERVERS: '[\"matrix.org\"]'
      TUWUNEL_ADDRESS: 0.0.0.0

networks:
  cadnet:
    external: true

Add to your Caddyfile:

your-matrix-domain.com {
    import cfdns
    encode gzip
    header Strict-Transport-Security max-age=63072000
    reverse_proxy twunnel:6167
}

Why Tuwunel? It’s not Synapse with all its memory overhead. Written in Rust, uses SQLite, no separate database server. Federation works with the wider Matrix network.

Step 5: Optional — Hermes Agent

The Sage agent runs on your homelab and connects through Matrix. It has tools — terminal access, file reading, web search — so it’s not just a text completion engine.

Create docker-compose.yaml:

networks:
  cadnet:
    external: true

services:
  hermes:
    image: nousresearch/hermes-agent:latest
    container_name: sage
    restart: unless-stopped
    command: gateway run
    networks:
      - cadnet
    volumes:
      - ./hermes-data:/opt/data
      - /path/to/host-docker:/opt/data/host-docker:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - HERMES_DASHBOARD=1
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: "2.0"

Add to Caddyfile:

sage.yourdomain.com {
    import cfdns
    encode gzip
    header Strict-Transport-Security max-age=63072000
    reverse_proxy sage:9119
}

What it does: When you’re out, you talk to it through Matrix. It can check server status, run commands, or look up information. Not a chat bot — an agent that can actually do things.

Architecture Overview

[Your Domain]
├── llm.yourdomain.com    → Caddy → llama.cpp (GPU box)
├── chat.yourdomain.com   → Caddy → Open WebUI (main box)
├── your-matrix-domain.com → Caddy → Tuwunel (main box)
└── sage.yourdomain.com   → Caddy → Hermes agent (main box)

All on Docker network 'cadnet'
All TLS via Cloudflare DNS
All DNS names, no port numbers

Client Tools (Bonus)

OpenCode

Run on your laptop, point to your local inference server. It reads code, suggests refactors, and helps navigate unfamiliar codebases. Everything stays on your network.

# Point OpenCode to your inference server
export OPENCODE_API_BASE=https://llm.yourdomain.com

pi

Similar concept — lightweight agent harness that runs on your laptop and connects to your homelab inference.

What Works Well

  • Code review and refactoring suggestions
  • Drafting technical documents and emails
  • Explaining unfamiliar codebases
  • Brainstorming architecture approaches
  • Having a working tool, not a chat toy

What Doesn’t

  • Anything requiring factual accuracy without verification
  • Long-context tasks beyond the model’s window
  • Anything you’d trust without reading the output first

What I’d Change

Model selection matters more than I expected. Start with a smaller model and work up — you’ll learn what your GPU can handle before committing to larger ones. Context size is a real constraint; 8K is fine for most tasks, but if you need more, you’ll need more VRAM.

The local-first constraint means you can experiment freely. If a model hallucinates or gives bad advice, it doesn’t leak to anyone. That freedom is the whole point.

Next Steps

Once this is running, you have a few options:

  • Add more models and swap between them
  • Connect additional frontends (mobile apps, desktop clients)
  • Set up the Hermes agent for automated tasks
  • Join the Matrix federation and chat with other homeservers

Pick what fits your use case. The stack is yours — change it however you want.

contact

Pick a channel.

mail hi@meherchaitanya.com linkedin meherchaitanya matrix @meher:hanumara.online