#
Why Ditch Cloud Inference Privacy isn’t a checkbox. it’s a boundary you control when your model never touches a foreign network. Every request stays inside your VLAN, hitting only your GPUs and never an OpenAI load balancer recovering from last year’s breach where employee Slack dumps were scraped into training sets. That $20/month subscription plus per-token usage for GPT‑4 flexes harder than most realize when you multiply by ten users burning chat completions daily on proprietary codebases or HIPAA-bound notes (e.g., patient records containing diagnoses and medications). Self-hosted Llama‑3 with vLLM on my Threadripper PRO nodes runs multiple concurrent streams at low latency with zero egress cost. No audit trail leaks. Your log retention policy is grep + rm, not a third-party SLA promising they deleted your conversation after thirty days. and then showing up in a class-action filing months later because their S3 bucket was misconfigured for five weeks. I migrated from ChatGPT Plus in January after watching a colleague’s entire project pipeline surface verbatim in a “model improvement” transparency report published earlier than planned. names, internal tool names, response thresholds all unredacted. On my K3s cluster hosting many containers across several RTX PRO cards (each with 96GB VRAM), I run Mistral Large alongside Llama‑70B quantized to Q4_K_M such that both fit within a couple of cards simultaneously while serving WebUI requests through Traef.
Cost vs Privacy Tradeoff That unredacted report cost my colleague a significant amount in lost contracts. clients saw internal tool names and thresholds scraped into training data without consent.each draws about 350W under load.Break-even lands within a year compared to cloud billing. The privacy calculus is starker. Samsung employees leaked proprietary source code through ChatGPT last April. the company banned the service internally afterward but the damage was done. Slack’s opt-out AI training policy surfaced last year showing user messages fed into model improvement pipelines without explicit consent. Self-hosted Open WebUI logs everything locally via Ollama; no packet leaves your wireguard VPN tunnel unless you explicitly enable internet routing.
Hardware Budgets Before Anything That zero-trust mesh collapses if your model never loads. I’ve watched engineers spend hours configuring Traefik routes only to discover their GPU has no room for even a quantized Llama variant. First thing: audit your VRAM with nvidia-smi and check ollama list trunk size for each quantization level you plan to run. Q4_K_M chews roughly four bytes per parameter plus cache overhead. A tiny Gemma 2B fits inside any modern GPU with six gigabytes free after system draw. But jump to Llama2. 13B at Q5_K_M and you need ten gigabytes dedicated solely to weights before context grows past four thousand tokens. CPU-only setups feel like dial-up reasoning against local GPU speeds. A single prompt on a high-core-count AMD Ryzen takes a long time, often minutes per generation depending on prompt length and quantization depth. usable for batch document analysis but agonizing during interactive chat sessions where sub-three-second replies matter. Docker deployment sidesteps most compatibility nightmares if you grab the official engine installer before doing anything else (curl -fsSL https://get.docker.com | sh). After that it’s two pulls: ollama/ollama for inference runtime plus ghcr.io/open-webui/open-webui for the interface layer overlay that wraps prompts and history into a familiar ChatGPT-style layout without phone-home telemetry. high IOPS ensures model loading stays quick even when swapping between seven billion and thirteen billion parameter variants.
Containers Up and Running That storage setup isn’t theoretical. it pays off immediately when you fire up both services simultaneously. I use a single docker-compose.yml binding port 3000 on host to port 8080 inside Open WebUI container, plus a bind mount from /nvme/models to /root/.ollama/models so Ollama sees my pre-downloaded weights instantly. First launch takes some time (roughly forty seconds) as the interface initializes its SQLite database and loads default embeddings model (nomic-embed-text, about 274MB). No telemetry pings leave the network. I verified with tcpdump on eth0 during startup; zero outbound connections besides localhost DNS lookups. The critical environment variable is OLLAMA_HOST=http://localhost:11434 so Open WebUI knows where inference lives. Without that explicit hostname, it defaults to external API calls. Model selection appears in the dropdown once Ollama finishes scanning its library hash (a few seconds per seven billion parameter variant). I keep four quantized checkpoints on disk: Q4_K_M versions of Gemma 2B, Llama 8B, Mistral Nemo Mini, and DeepSeek Coder V2 Lite.
Lock Image Tags Now
That four-model library—spanning Llama 2 7B through Mixtral 8x7B—saved me significant debugging time compared to running latest across all services. Without pinned tags, I once woke up to a broken stack because Ollama’s latest silently upgraded from v0.1.34 to v0.1.35 overnight and changed its HTTP response format for streaming tokens. I draft every Open WebUI deployment with explicit tags rather than vague latest references:. yaml
services: ollama: image: ollama/ollama:0.1.35 restart: unless-stopped volumes: - ./ollama-data:/root/.ollama environment: - OLLAMA_KEEP_ALIVE=24h For even tighter control, replace the tag with the full SHA256 digest (`image: ollama/ollama@sha256:a3c9b...`) so no registry mutation can ever substitute your version. That `WEBUI_SECRET_KEY` variable is non-negotiable for persistent sessions across restarts; without it every container recycle invalidates all active chats and user tokens. a nightmare when you have many ongoing conversations lost overnight. Generate that key with one command before launching:.bash
openssl rand -base64 42 > .env_secret_key
``` Then source it inside your .env file as WEBUI_SECRET_KEY=$(cat .env_secret_key). Forty-two base64 characters gives you roughly 252 bits of entropy. overkill for most setups but trivial to generate once and forget. The real pain killer comes from the `OLLAMA
Health Check Before Exposing. That OLLAMA_KEEP_ALIVE value works best after confirming your stack actually boots cleanly. I always hit the /health endpoint locally first. no reverse proxy, no domain name in play yet. A simple curl http://localhost:11434/api/health returns either {"status":"ok"} or nothing at all. If it’s silent, you’ve got a port conflict or missing model directory bind mount. Version lock beats “latest” every time. Several builds broken overnight taught me that lesson hard in production. Pin both images explicitly: ghcr.io/open-webui/open-webui:main isn’t safe enough. use a specific digest or release tag like v0.4.x. For Ollama I pull ollama/ollama:0.3.x. Write those exact strings into your compose YAML so CI never surprises you at 2 AM. Secrets generation is two commands away. Run this in your shell before starting anything:. ```bash
openssl rand -hex 64 > .webui_secret_key
export WEBUI_SECRET_KEY=$(cat .webui_secret_key)
That 128‑character hex string becomes the internal credential Open WebUI use to sign sessions and JWTs. Without it, login resets every container restart. ask me how I know after debugging for a while last month. **Bind ports deliberately for your reverse proxy.** I map `3000:8080` on the webui service and `11434:11434` on ollama. but only if Nginx or Caddy sits in front handling TLS termination and rate limiting. Expose port 3000 locally during testing; once confirmed alive behind Caddy’s Let’s Encrypt flow, switch to internal-only bindings like `127.0.0.1:11434`. My final compose snippet looks roughly like this abbreviated excerpt:.yaml
services: ollama: image: ollama/ollama:0.3.x ports: - “127.0.
Piping Models Into The UI But binding Ollama to 127.0.0.1 creates an interesting problem. how does Open WebUI talk to it. Both live inside the same Docker network defined by the compose file’s networks: block; I use a custom bridge named llm-net. Inside that overlay each container resolves its neighbor by service name (ollama, open-webui), so the web UI sends requests to http://ollama:11434/v1/chat/completions without ever exposing a real port outside the host. That endpoint follows the OpenAI Chat Completions schema exactly. identical JSON structure down to the messages, model, and stream fields. That compatibility means you can plug in any backend that speaks that shape: local vLLM instances running on separate hardware. Self-hosted TGI containers on another server in your rack, even remote services like Together AI or Groq if you trust their latency. ```bash
curl -N http://ollama:11434/v1/chat/completions
-H “Content-Type: application/json” \ -d ‘{ “model”: “mistral”,
“messages”: [{“role”:”user”,”content”:”Count to five”}], “stream”: true }’
``` If everything is wired correctly you’ll see raw Server-Sent Events pouring back. each line starting with data: {"choices":[{"delta":{"content":"..."}}]} followed by a double newline and finally a data: [DONE]. Missing those SSE headers usually means something upstream (Caddy reverse proxy stripping buffering) or downstream (Open WebUI expecting non-streaming) is breaking the chain. I configure additional backends inside Open WebUI’s admin panel under “Connections → OpenAI-Compatible Endpoint”. Each entry needs three things: a name label (e.g., “Home vLLM”), the full URL (http://10.99.0.15:8000/v1), and an API key if the target requires one. otherwise a placeholder like “sk-no-key” works fine for unauthenticated local models. One mistake I made early on was forgetting to pin which models should appear in the dropdown selector under “Models → Visible Models”. Without that explicit list the UI tries to query /v1/models on every configured endpoint at startup, which can timeout against slow local servers or return hundreds of provider options if you connected OpenAI directly instead of an Ollama proxy. multiple GPUs across several hosts means multiple Ollama agents are typical here. Expose only what users actually need; hide everything else behind explicit enumeration rather than wildcard loading even when using single-server setups on less exotic hardware than mine.