When your GPUs cost more per hour than most people’s rent, you don’t “hope” the silicon is fine. I built a war room in Grafana with a custom PromQL alert that fires when latency exceeds 1 second. My setup is three servers in a K3s cluster. That’s nine GPUs total, all stitched over a WireGuard mesh?

When MiniMax-M2.7 or DeepSeek-V4-Flash inference goes sideways at 3 AM, I need to know which GPU throttled, which memory controller hit its thermal ceiling, and which SGLang pod OOM’d before my sleep cycle ends. Prometheus scrapes every five seconds across all three nodes. Grafana dashboards surface the delta between “healthy” and “rebuild the world” in real time?

Custom alert rules fire when any GPU’s temperature exceeds 85°C for more than two consecutive scrape intervals. Because one silent thermal throttle on a single card cascades into a cluster-wide throughput collapse. This is production monitoring for silicon that costs more than most people’s rent. Every silent failure costs more.

I use PromQL queries to check if GPU latency exceeds 1 second. This pattern prevents silent failures that already cost you four hours of training time.

Why You Can’t Trust “nvidia-smi” in Production I learned this the hard way. nvidia-smi gives you a photograph. It freezes one frame at Ctrl+R.

That frame looks fine. Your memory sits at 72%. Your power reads 450W. The problem hides between frames. A memory leak creeps 2GB every hour. By hour 36, you’re swapping. nvidia-smi shows 96GB full at hour 38. Your training job crashes at hour 40. Three days of checkpoints vanish. Prometheus changes the game.

It scrapes every 15 seconds. That 15-second window catches the memory climb. It graphs the power draw oscillation. It shows the thermal ramp over Tuesday’s 8-hour inference burst.

The Stack Architecture (Prometheus + Node Exporter + DCGM) I burned 14 days scraping nvidia-smi before switching. DCGM cut my collection latency from 12s to 3s.

Now a PromQL alert fires when GPU temp exceeds 85°C. Three endpoints per node. One for GPU health. XID errors, ECC page faults, power thresholds. One for use curves. One for memory bandwidth saturation. Prometheus scrapes them on a 15-second interval with a 30-second timeout. No driver lock contention. The config lives in /etc/prometheus/file_sd/targets/dcgm.json.

Three entries, one per server IP. File service discovery watches that directory for YAML changes. Add a fourth GPU server tomorrow. Write its hostname into that file. Prometheus picks it at the next reload cycle. Two exporters per node, one for DCGM, one for generic hardware. Separate scrape jobs in prometheus.yml so GPU metrics never block system metrics.

Remote write pushes everything to Thanos with a 6-hour retention on object storage for Prometheus’s local disk. Thanos holds the last 90 days on S3-compatible backend for anomaly hunting across last month’s training runs. The DCGM exporter exposes DCGM_FI_DEV_POWER_USAGE at port 9400 on each node’s static IP range: 192.168.10.[11-13].

The scrape job uses relabel rules to attach node labels per GPU index so every metric carries its physical position in the rack topology. No config reloads when new GPUs arrive. No manual curl probes to validate exporter health before scraping starts. Prometheus logs the exact HTTP error code in prometheus.log under the scrape failure line at line 2847 in the Thanos sidecar’s debug output.

Three lines of YAML in `prometheus.yml defines the entire discovery pipeline:

Grafana Dashboard Design (Not Just Use Gauges) I replaced standard use gauges with Grafana PromQL after day one. My three-tier dashboards track Cluster Overview → Node Health → Single-GPU Deep Dive. Node Health catches thermal throttling before training degrades.

I graph memory junction temps against fan RPM curves. The critical panel most people miss: PCIe replay counter. That rising line means silent packet corruption on your interconnect. Single-GPU Deep Dive panels show three metrics: memory bandwidth stall %, SM occupancy versus compute use, and tensor core active cycles against CUDA core cycles.

High occupancy with low compute means latency-bound kernel launches from 12s to 3s. I found one job hitting 12% PCIe replays on GPU 4 last month from a bad cable reseat. SM occupancy at high but compute use low signals wasted kernel launch cycles. The Kevin Zhou custom panel JSON lives in my GitHub gist linked below.

It queries gpu_pcie_replay_count with rate() over 5-minute windows. My SM occupancy panel uses gpu_sm_occupancy_active_warps / gpu_sm_occupancy_max_warps * gpu_compute_utilization. That ratio reveals wasted cycles instantly. I use Grafana variable templating to switch between model jobs without editing queries per run ID. The $job_id variable filters Prometheus labels from my Go exporter.

One dropdown selects training runs across MongoDB job records. No manual query rewrites. My dashboard starts at 47 panels per node. Add the replay counter first. That single metric saves you one dead training run per quarter.

Alert Rules That Actually Page You at 3 AM (And What Doesn’t) High XID error rate >5/min triggers PagerDuty immediately.

My Prometheus rule fires with the GPU index embedded in the annotation template. The alert includes the exact PCIe slot from nvidia-smi output. Memory temperature hysteresis rules filter 70°C spikes from normal training heat cycles. Thermal throttle doesn’t start at 70°C. My rule ignores anything under 85°C unless it holds for 90+ seconds.

The rate-of-change expression checks dcgm_memory_temp_celsius delta across three consecutive scrape intervals. Power cap violation alerts escalate when non-training workloads sustain >95% TDP for 10+ minutes. Mining detection catches rogue containers before they drain power budgets. My alert uses dcgm_power_usage_watts / dcgm_tdp_limit_watts * 100 with a rate() function over 10-minute windows. Memory temperature rate-of-change PromQL: rate(dcggpu_mem_temp_celsius[2m]) > 15.

That catches thermal runaway without alert fatigue. Every false positive gets a one-week quarantine before deployment. The alert template renders GPU index, PCIe bus ID, and current temp delta in the PagerDuty payload. What never pages me: fan speed fluctuations at 40% versus 60%. What does page me: XID errors exceeding 5 per minute with consecutive failures across multiple scrape intervals.

Memory temperature spikes above 92°C sustained beyond four scrape intervals triggers immediate escalation to senior on-call rotation.

#

Cost Optimization Alerts Most Guides Skip (The Hidden Money Bleed Money Silently. My Prometheus rule triggers on zero compute work when memory sits locked but idle. It checks dcgm_gpu_utilization below 5% with active memory allocation over 80%.

That memory allocator never released tensor buffers from a crashed training job. One job crashed four hours before any human noticed the waste. Thermal cycling alerts catch cooling failures early. My rule fires when fan RPM spikes above 90% while workload remains under 30% for three consecutive per-scrape intervals.

The thermal delta between GPU core and ambient hit 14°C with zero productive work during one test run. Bad cooling design wastes power and silently kills card lifespan across your cluster.

Automating Alert Fatigue Protection Without PagerDuty Spam During Scheduled Maintenance My Grafana OnCall fired 47 alerts before the checkpoint jobs finished at 00:15 UTC.

I solved this with Grafana mute timers annotated via API. My CI/CD pipeline now calls the Grafana API at 23:55 UTC before deploying new training checkpoints. The curl command posts a mute_timer with a 25-minute window targeting the gpu-cluster namespace. The metric change was immediate.

Alert volume dropped from 47 to 3 real alerts per deployment cycle. My PagerDuty integration stopped firing for known maintenance noise. Here’s the exact mute timer annotation payload I use:.

bash curl -X POST "https://grafana.internal/api/annotations" \ -H "Authorization: Bearer $GRAFANA_TOKEN" \ -d '{ "dashboardUID": "gpu-monitor", "panelId": 3, "time": $(date +%s)000, "timeEnd": $(date +%s -d "+25 min")000. "tags": ["maintenance", "checkpoint"] }' I also added a Go middleware in my monitoring agent that checks for active mute timers before firing any alert. If a timer exists, the alert gets suppressed for that specific GPU node. The custom alert rule now includes a mute_if_maintenance condition. It queries MongoDB’s checkpoint_jobs collection every 30 seconds.

If a job is running, the alert threshold shifts from 95% GPU memory to 99% GPU memory. The Go agent reads the mute timer from MongoDB within 2 seconds of job start. I reduced false positives by 93% without disabling alerts entirely. The maintenance window covers exactly when CI/CD deploys new checkpoint jobs.

The real win came from combining both approaches. Mute timers silence alerts during maintenance. Custom rules shift thresholds only during known windows. My team now gets 3 real alerts per deployment instead of 47 false ones. The midnight UTC window costs me exactly one curl call and one MongoDB query every 30 seconds.

Scaling From One Server To Nine Without Burning Out Your SRE Team That setup required maybe 2 hours of weekly maintenance. The alert volume jumped from 12 daily notifications to 247. My first mistake was treating each server like a pet. I had separate dashboards per node. That broke at server 4.

I consolidated into one unified Grafana view in September. My triage time dropped from 45 minutes to 8 minutes. I automated node onboarding in October 2023. The script first runs a check_mongodb_health PromQL query in Grafana. Alert fatigue nearly killed my Friday nights. I tuned every threshold using real data. GPU memory drift triggers at 85% for 5 minutes.

Temperature warnings fire at 87°C for 3 consecutive samples. False alerts dropped from 60% to 7%. My PagerDuty channel went quiet after November. The real win came from batching maintenance windows. I schedule firmware updates on Tuesday at 2 AM UTC. All three servers update within that window. Downtime per server is exactly 4 minutes and 12 seconds.

I wrote a custom alert for MongoDB write latency in December. Any replica set hitting 150ms triggers a direct Slack DM to me and a PagerDuty high-urgency page. That single alert caught three MongoDB connection pool leaks before they hit production. My SRE team is just me and one part-time DevOps engineer.

We spend about 6 hours weekly on monitoring now. That’s less than my solo setup required for that single server. They fail silently, one thermal throttle at a time. That’s the single insight that forced me to build this stack instead of trusting nvidia-smi at 3 AM. No vendor lock-in required.

The real lesson is this: nvidia-smi lies by omission. It shows a snapshot, not a story. My Grafana war room sees the five-second heartbeat of every memory controller. You can build this exact stack tonight. Your GPUs will cost less to babysit tomorrow. What silent failure are you currently missing between frames?