I’ve burned more GPU racks than I care to count. The first one went down because of a kernel module mismatch. It took me three days to realize nvidia-smi wasn’t even seeing the card. By rack number four, I had a standing cron job that checked CUDA compatibility before model loading even started. Most teams spend weeks wrestling with driver hell, environment drift, and OOMs that silently kill inference pipelines at 2 AM.

We didn’t set out to build the ultimate LLM deployment stack. We got tired of waking up to dead inference endpoints. Our approach was simple: iterate fast, fail hard, document everything. After twelve months of running our own GPU cluster for production inference (peaking at around 8,000 requests per day across five models), we assembled something that works without constant firefighting.

This guide distills every hard-won lesson: how we load-balance across cards using vLLM with continuous batching (not naive round-robin), why we scrapped Kubernetes for raw systemd units on bare metal (halved our p99 latency). And the exact alerting chain that catches OOMs before they cascade into full reboots.

You don’t need hyperscaler budgets or a dedicated SRE team. You need systematic caching policies, aggressive concurrency limits, and someone who has already broken every component so you don’t have to rebuild from scratch mid-crisis. This is the stack we run today in production: no marketing fluff, no vendor pitches. the benchmarks that survived actual traffic.

Know Your Model Before It Knows You That stack assumes you picked the right model.

They grab a 7B because it sounds small. A 13B sounds bigger and therefore better. Parameter count is a liar. KV-cache overhead flips that logic entirely when your context window hits 32K tokens. I’ve watched a 7B collapse under its own attention mechanism while a well-optimized 13B hummed along at the same effective throughput.

Quantization is where the real math lives. Q4_K_M shrinks a 7B from 14GB to roughly 4GB while retaining output quality, according to real-world benchmarks compiled across GPU backends. That’s the difference between needing one consumer card or four enterprise cards. But slap INT8 on every layer without checking your workload, and you’ll watch inference latency spike rather than drop. activation quantization can fight back against certain attention patterns.

FP16 remains king for batch sizes above eight, assuming VRAM permits. AWQ wins for single-stream latency when you’re serving one user at a time through an API gateway like FastAPI or vLLM. GPTQ works well on older GPU architectures but suffers from calibration dataset sensitivity; train on Wikipedia, serve on legal contracts, and watch perplexity drift upward by several points. Model selection dictates everything downstream: throughput targets, hardware provisioning, even your retry logic.

Pick wrong here and no amount of infrastructure wizardry saves you.

Quantization Is Not Optional That model you downloaded won’t fit A 70B parameter model in FP16 demands 140GB.

That’s more than most server GPUs combined. Quantization shrinks precision while keeping the intelligence. Q4_K_M drops from 16 bits to roughly 4.5 bits per weight. It halves your VRAM bill with minimal quality loss measured by perplexity benchmarks. I’ve run Mixtral 8x7B on a single GPU this way when it otherwise needed two.

The math is brutal: params × bytes_per_param = VRAM. A 7B model at FP32 (4 bytes) needs 28GB. At Q8 (1 byte) it’s just 7GB. At Q4_K_M (~0.5 bytes) you get about 3.5GB plus some overhead for KV cache and activations. Which quantization should you pick?

Run the arithmetic against your hardware ceiling first, not against a wishlist of features. Precision Bits/Param Quality Cost Use Case   FP16/BF16 16 Baseline Training or highest-quality inference   Q8_0 8 Minimal loss Safe choice for production without noticeable degradation   Q4_K_M ~4.5 Small loss Sweet spot: huge VRAM savings.
Barely detectable quality drop GGUF format makes this trivial: just swap the model file and restart. No recompilation or conversion pipeline nightmares are needed. Ignore anyone telling you to quantize after deployment.

Test your chosen bit-width in development against your actual prompt distribution. Then measure the output quality yourself with something like lm-evaluation-use. The difference between perfect reproduction and acceptable drift varies wildly by task; classification tolerates compression than code generation does.

The Quantization Tradeoff FP16 eats 16GB for a 8B model

Q4_K_M squeezes into 5.3GB. That’s the difference between an A10G with breathing room and a T4 running out of memory mid-conversation. The Ionio.ai benchmarks on Qwen2.5 and DeepSeek show why format matters: GPTQ-INT8 keeps close to baseline accuracy on coding tasks, while Q5_K_M holds its own for agent workflows. AWQ’s reordering trick buys another few percentage points on arithmetic reasoning without changing the bit count.

Batch size kills your math faster than model size does. With Llama‑3‑8B at Q4_K_M, a batch of 32 sequences at 4096 tokens each demands roughly (batch × seq_len × layers × hidden) / (bytes_per_bit). plug your own numbers in there. The formula saves you from guessing: (batch_size × max_seq_len × num_hidden_layers × hidden_size × bytes_per_weight). For most setups, that means FP16 fails before inference even starts.

The perplexity chart tells a clearer story than any marketing slide. I ran evaluation use comparisons across five quantization schemes last quarter. GPTQ-INT8 matched FP16 closely on MMLU but cost nothing in runtime overhead. Q3_K_M cut memory by half and still performed well on HumanEval. close enough for prototyping, dangerous for production financial analysis where every token matters.

Pick your bit width based on task risk, not GPU envy. Classification pipelines tolerate aggressive compression because they only need broad semantic boundaries. Code generation demands stricter preservation; one wrong weight can shift an entire function body into incorrect logic. Test both schemes with lm-evaluation-use against your actual prompt distribution before deploying anything to staging.

Traffic Is Never Gradual Ten concurrent users is not a stress test.

The single-node fallacy kills more deployments than bad models. Most GPU servers look capable on paper: plenty of VRAM, fast interconnects, low latency. Then multiple users hit POST /generate simultaneously, and decode latency spikes. Here’s what happens inside CUDA streams. When both compete for the same GPU. and they will, because inference frameworks default to greedy scheduling. the prefill phase starves decode tokens.

Your first user gets a response quickly. The eleventh waits much longer while the server batches their prompt into an already-running stream. NVIDIA MPS isolates tasks but caps at 48 concurrent CUDA streams per GPU. I’ve watched monitoring dashboards show flat use while response times tripled. no red metrics, just angry users. Device partitioning solves this differently.

Instead of sharing one GPU across all request phases, split your hardware: one device dedicated to prefill (high memory bandwidth), another handling pure decode (compute throughput). Llama.cpp’s --parallel-requests flag approximates this on a single node by reserving CUDA contexts per phase. Kubernetes makes the split harder than it should be. Standard device plugins expose GPUs as opaque resources; you cannot express “I need half for prefill.

And half for decode.” work around this with node affinity labels and custom schedulers that pin pods to specific device indices via NVIDIA_VISIBLE_DEVICES env vars.

Does it hold under real load. Only if you cap concurrency at your device count minus one headroom slot. the rule I’ve broken three times with predictable failure each time?

#

Time-to-First-Token Gets Weird Cold-start latency killed my first production deployment

A new model pod took a long time to load the weights before serving its first request. Users hitting a scale-up event waited longer than they would for a physical RMA. I fixed it by pinning two base replicas per model region. This is wasteful on paper. One idle GPU costs me money every minute it sits empty.

But the math changes when you factor in user retention. Most autoscaling systems measure average CPU or memory use. That’s fine for web apps. For LLMs, the right metric is queue depth. How many requests are waiting for a compute slot at any moment?

Set your scale-up threshold at one queued request per active replica. Not two, not five. When you’re batching tokens, a single queued request signals that your current batch window is filling faster than your inference can drain it. Scale-down needs guardrails too. Never drop below two replicas per model, and wait at least five minutes after the queue drains to even consider removing a pod.

I’ve watched Kubernetes tear down all but one instance during a lull, leaving the next burst to cold-start through a long delay. The orchestrator matters more than most teams admit. GMI Cloud’s platform gives me granular control over these thresholds without writing custom HPA manifests. Their autoscaler watches token throughput, not generic CPU load. exactly what I’d build if I had three months of dev time. You don’t need Google-scale infrastructure to survive traffic spikes.

You need honest metrics and conservative minimums.

The Cold-Start Cascade That Killed Our P99 Queue_depth=12 was the trigger.

A single metric I’d ignored for months because the dashboard showed green. One number crossing a threshold I hadn’t defined. P99 response times went from low to high in under three minutes. The sequence is predictable once you’ve lived it. Request volume exceeds worker capacity, so pending batches pile up in the inference queue.

Each new request waits longer for an available slot. Those slots are processing slower because memory pressure is rising. Your GPU starts swapping context windows to system RAM, which drops throughput significantly per request that hits swap. Grafana showed the cascade clearly. after we knew what to look.

The trace spans from that incident show auth middleware at normal latency, then a long gap where nothing appears in the trace because the batch scheduler blocked on a memory allocation call that never returned quickly.

We added three panels to our monitoring setup after that incident: queue_depth as a time-series overlay on every deployment’s dashboard, pending_batch_ratio with a red threshold at a high value, and utilization_slope computed over a sliding window. The slope panel catches problems before depth spikes. when use climbs faster than a certain rate per minute, something upstream is degrading. OneUTM’s infrastructure dashboards now feed these signals into alert conditions directly.

Inference Serving Architecture Done Right A raw model file is dead weight without a proper serving layer.

I learned this the hard way after watching an 8B Llama 3 deployment fall over at a modest number of concurrent requests. The problem wasn’t the GPU. it was my thread pool. I start every inference server with vLLM now. It handles continuous batching natively. You don’t need to glue together custom scheduling logic with this approach.

Switching from raw Hugging Face Transformers to vLLM bumped my throughput significantly on a single . The request pipeline matters more than most teams admit. My setup routes through NGINX first for rate limiting, capping at a reasonable rate per model. Then a FastAPI middleware layer validates input length before it ever touches GPU memory. A single malicious 32K token prompt can starve an entire batch if you don’t check upfront.

Cold start latency kills user trust faster than any model hallucination does. I preload my top two models into GPU memory on deploy and keep one unallocated slot for swapping smaller models in under a short time. Triton Inference Server handles the model repository logic here, versioning each checkpoint and rolling updates without dropping active connections. Load balancing across replicas requires sticky sessions when handling chat histories.

My setup uses Redis-backed session affinity with a TTL of a reasonable duration per conversation window. Without it, round-robin distribution splatters context windows across four GPUs and everyone sees garbled responses. Dynamic batching demands careful timeout configuration. too aggressive and small requests wait forever, too lax and throughput craters below useful thresholds. I settled on a short batch accumulation window after benchmarking against real traffic patterns from production logs spanning three months.

Monitoring saved my deployment twice last quarter alone. Prometheus scrapes inference latency broken down by percentiles frequently into Grafana dashboards that trigger pager alerts when tail latency crosses a threshold sustained for several consecutive samples.

Observability Beyond Dashboards Metrics told me what was breaking But I needed to know why.

I wired OpenTelemetry into the inference pipeline. Spans trace each request through tokenization, model forward pass, and response streaming. One trace revealed a significant delay in our custom tokenizer. It was a regex that ran O(n²) on long prompts. Metrics showed P99 latency spikes but gave no clue about the root cause.

Logs caught the rest. Every hard OOM dumps model weights metadata to Loki. It also stores CUDA allocator state and active request IDs. The structured format lets me query {service="llm-server"} |= "OOM" | json across multiple nodes simultaneously. Found a memory leak in the KV cache handling within a short time of deploying v2.1.

Prometheus metrics surfaces correlation at scale. A sudden increase in tensor_parallel_allreduce_seconds coincided with degraded throughput at a specific time. No code change had landed in many hours. The responsible engineer dug into InfiniBand link training logs and found several HCA links running at reduced speed from firmware autonegotiation errors across multiple host reboots. This was silent degradation undetectable without cross-referencing metrics against hardware telemetry frames collected frequently from switch fabric monitoring streams.

That’s the whole thing. No magic sauce, no proprietary orchestrator, just deliberate defaults and a decade of broken hardware behind them. The single insight that survived every rebuild is this: LLM hosting fails on configuration entropy long before model quality ever matters.

Keep Reading

Every team I’ve talked to has their own version of that first dead GPU rack, the one nobody documented. So my question for you is simple. what happens when your p99 latency drifts by a small amount at an odd hour next Wednesday. If you can’t answer that today, start here. Build the alert chain first, then argue about Kubernetes later.