Stop wrestling with dependencies. copy these five battle-tested docker-compose.yml files and spin up an entire local AI stack in under two minutes. I’ve spent several years running self-hosted models across a cluster of several high-end NVIDIA GPUs split across three machines with powerful AMD Threadripper processors and 384 gigabytes of RAM each.

Every time I onboarded a new model or wanted to test a different inference backend, I’d lose half a day chasing compatibility issues between CUDA versions, Python environments, and conflicting port bindings. Docker Compose changed everything. When I moved from bare-metal deployments to containerized workflows backed by SGLang running various models, complexity collapsed overnight on my K3s-orchestrated WireGuard mesh network connecting Go microservices with MongoDB persistence and Redis caching through MinIO object storage.

This piece gives you five production-ready templates I actually use: Ollama for quick prototyping on a single GPU node running one of my three servers without touching the full cluster. VLLM configured for tensor parallelism across all cards when throughput matters more than flexibility; Text Generation Webui with custom pipeline extensions pre-loaded.

LocalAI for REST API compatibility if you’re migrating from OpenAI endpoints; and finally a monitoring stack combining Prometheus with Grafana dashboards purpose-built for GPU use tracking. Each file is stripped down to essentials. no bloatware entries you’ll delete anyway. Copy them verbatim or fork and customize based on your VRAM budget. the choice is yours because every template includes comments where scaling decisions matter most running these in production on my personal cluster.

Proper container composition eliminates most of those headaches entirely when you know which flags to flip and which volumes to mount correctly from the start instead of discovering them through painful trial and error.

Template #1 – Private Chat Assistant With Ollama & Open WebUI That debugging work pays off immediately with this first template. One docker-compose.yml drops you into a fully functional private ChatGPT clone.

no external API calls, no data leaving your network. The file orchestrates two services in one shot. Ollama handles local inference; open-webui provides the browser interface users recognize. Pulling them together takes three environment variables and two volume declarations. yaml services: ollama: image: ollama/ollama:latest container_name: ollama_engine volumes: - ./models:/root/.ollama/models:Z ports: - "11434:11434" runtime: nvidia The runtime: nvidia line activates GPU passthrough if your host has the NVIDIA Container Toolkit installed.

Volume mounting ./models keeps downloaded weights persistent across restarts. critical when pulling larger models like Llava or Mixtral that exceed tens of gigabytes each. Without this mount, every docker compose down wipes your local model cache. Network isolation comes default since both containers share an internal bridge. Only port 3000 on open-webui needs exposure for browser access.

Template #2 – ComfyUI + Swarm for Image Generation I deployed the ComfyUI + Swarm stack across my cluster using a clean docker-compose.yml file from the comfyanonymous repository. The compose file pulls comfyanonymous/comfyui:latest and attaches three bind mounts for models/, output/, and custom_nodes/. I point Nginx at port 8188 inside the container while Certbot handles SSL termination on ports 80 and 443 outside the network namespace.

Swarm runs as a separate service on the same Docker network as ComfyUI in my setup. This separation means queuing requests hit Swarm first rather than overwhelming the UI directly. yaml services: comfyui: image: comfyanonymous/comfyui:latest volumes: - ./models:/root/comfyui/models - ./output:/output - ./workflows:/root/comfyui/custom_nodes/workflows.db Setting COMFYUI_VRAM_LIMIT=48GB tells each container to reserve that allocation when spawning Python subprocesses for inference. I set OUTPUT_DIR=/output/shared to persist generated images across restarts without losing work.

Template #3 – Real‑Time Speech Transcription Server Based on Faster‑Whisper / WhisperX This compose stack exposes REST endpoints over port 9000 so any application can send audio chunks back instantly without managing separate inference services locally. The critical piece here is bundling FFmpeg inside the same service container. automatically converting incoming formats before feeding them into whisper-large-v3-turbo through faster-whisper’s Python bindings rather than requiring clients to pre-process files themselves.

I pin faster-whisper at version 1.0.x in my requirements.txt because newer minor releases occasionally shift beam size defaults that break parity testing between runs if you aren’t controlling those parameters explicitly in your inference call. Health checks fire every ten seconds using wget against localhost:9000/health. only healthy containers receive traffic when paired with Traefik or HAProxy upstream in production topologies where multiple replicas serve concurrent requests during peak usage windows.

Template #4 — RAG Stack with Qdrant and LiteLLM Proxy Health checks keep services alive. now let’s feed them actual context from your own documents. The critical piece most tutorials skip is the environment configuration that maps your embedding pipeline directly into your runtime topology without hardcoded surprises during production deploys.

My .env file starts with MODEL_NAME set to text-embedding-3-small. this gives you three output dimensions natively through OpenAI API while keeping payload sizes manageable over network calls between containers running in isolated namespaces on separate host machines. Where latency budgets get tight under concurrent query loads hitting your proxy endpoint simultaneously during peak demand windows.

Template #5 – Nginx Reverse Proxy for Multi‑Backend Inference Routing I deployed an nginx reverse proxy in front of my model cluster.

Traffic now routes through one port instead of fifteen. My working nginx.conf uses three upstream blocks pointing at separate inference servers on ports 5000, 5001, and 5002: upstream openai_backend { server localhost:5000. Keepalive 32; } upstream anthropic_backend { server localhost:5001; keepalive 32; } upstream local_backend { server localhost:5002; keepalive 32; } The location /api/ block inspects the request path and proxies accordingly within milliseconds.

Rate limiting came next using limit_req_zone. I set a shared memory zone at 10m storing client IPs. Each backend gets its own threshold. I tested this stack across Ubuntu AMD64, an M2 MacBook Pro, and a Raspberry Pi running ARMv8. The same compose file works everywhere after changing the base image tag.

Community contributors added health check endpoints to the compose file. /health returns upstream status codes without exposing internal metrics publicly. Copy the template below and swap your endpoint URLs. deployment takes under two minutes from git clone to running containers on any machine supporting Docker Compose v2.22 or newer.