One NVIDIA H100 costs $30,000+ on the open market. This 3-node cluster cost me $7,800 and runs Llama 3.1 70B at 48 tokens per second — matching single-GPU setups that cost four times as much.
I’ve spent six months obsessing over this problem: how do you get serious AI inference horsepower without financing hardware that depreciates 40% in two years? The answer isn’t buying pre-built solutions from vendors marking up 60% margins. It’s building distributed infrastructure from off-the-shelf parts.
I’m going to walk you through exactly how I constructed a three-node cluster using consumer AMD hardware and nine GPUs total, for less than the price of one enterprise card. That figure excludes cables and power upgrades — I’ll cover those too.
Single-GPU buyers face two hard limits simultaneously. A consumer RTX 4090 has 24 GB of VRAM — not enough to load Llama 3.1 70B in FP16 (which needs ~140 GB). Memory bandwidth caps at 1 TB/s on that same card. Neither metric scales affordably when your model outgrows the envelope. Distributed inference breaks through both limits: split the model across nine cards and you get 216 GB of combined VRAM with aggregate bandwidth that scales near-linearly.
The Architecture — How Three Nodes Split the Work
Data parallelism handles fine-tuning well — each GPU trains on a different data shard, gradients sync periodically. Inference is a different beast. You can’t fit a 70B model on one card, so you split it across all nine with tensor parallelism.
My cluster breaks into three roles:
- Node 1 (controller): K3s control plane, Slurm scheduler, MongoDB for job state. 3× RTX 3090 (24 GB each).
- Node 2 (compute): Primary inference worker. 3× RTX 3090.
- Node 3 (compute): Secondary inference worker. 3× RTX 3090.
All three sit on a dedicated 10.10.0.0/24 private subnet, isolated from my home LAN. This separation keeps scheduling overhead off the GPU-bound network path.
Network topology is the part most builders get wrong. I chose Mellanox ConnectX-4 Lx EN adapters ($45 each on eBay) for the inter-node fabric. These NICs deliver 25 GbE per port while drawing under 8W idle. The switch is a MikroTik CRS305-1G-4S+ ($140) — four SFP+ ports for direct fiber runs between machines, zero routing complexity.
I didn’t expect this: batch size changes interconnect utilization more than raw compute does.
- Batch size 1: 48 tokens/sec across two nodes via vLLM tensor parallelism. Respectable, but the interconnect sits 60% idle.
- Batch size 8: 890 tokens/sec aggregate. Per-request latency drops by nearly half compared to sequential processing because the pipeline fills properly.
vLLM’s distributed KV-cache made this work. Each GPU maintains its own portion of the attention key-value cache, but cross-node communication must synchronize these regions before every forward pass completes. Without sufficient bandwidth between nodes, tensor parallelism degrades into a serialization bottleneck — one GPU idles while its neighbor finishes cache transfers. At 25 GbE, I measured ~2.8 GB/s effective throughput between nodes, which keeps synchronization under 3ms per step for Llama 70B.
I chose vLLM over TensorRT-LLM despite slightly lower peak throughput. TensorRT requires ahead-of-time graph compilation — every model change means a 20-minute recompile. vLLM lets me swap quantization levels or switch models in seconds, which matters more during rapid iteration.
Storage — the overlooked variable. Running dataset pipelines against a single SATA SSD creates an I/O bottleneck that manifests as GPU utilization drops during epoch boundaries. Your CUDA kernels sit idle waiting on data loaders. I installed one Samsung PM9A1 NVMe (2 TB, $120) per node for OS and active datasets. Sequential read speeds hit 5.2 GB/s in my benchmarks — enough headroom to keep nine cards fed during prefetch operations.
Cooling. Each chassis runs dual 120mm Noctua NF-A12x25 intake fans pulling ambient air across GPU heatsinks, with two rear 120mm exhaust fans maintaining positive pressure. GPU temps stabilize at 72°C under sustained load — 13°C below throttle threshold.
Software Stack — NCCL Tuning Across Nodes Over Ethernet
Hardware only matters when software lets it breathe.
NCCL installation demands custom topology files from the start. The generic build assumes single-node and autodetects PCI switches incorrectly when you span multiple chassis over Ethernet. I extracted nccl-2.19.3 into /opt/nccl and wrote a topology file at /etc/nccl.conf specifying each node’s NIC MAC address and expected bandwidth ceiling (25 Gbps per link).
Scheduler choice: Slurm vs Ray Serve. I tested both over several weeks:
- Slurm wins for batch training.
srun --gres=gpu:3 --nodes=2allocates GPU counts per task automatically with fair-share accounting. Job scripts live in/opt/slurm/scripts/. - Ray Serve wins for inference. Its
handle.remote()pattern maps cleanly onto HTTP endpoints without wrapper overhead. I run it inside K3s pods for autoscaling.
Both require an explicit hostfile listing all three node IPs before any multi-node job launches.
Critical post-boot checks:
# Verify GPU topology — every card should show PCIe x16, not x8@Shared
nvidia-smi topo -m
# Verify NUMA affinity matches GPU placement
numactl --hardware
NUMA pinning matters if your training script spawns dataloader workers without cpu_affinity constraints matching each server’s socket layout. Cross-socket memory copies bleed bandwidth during every epoch boundary. I caught a 30% throughput regression from this alone.
Enabling NCCL_DEBUG=INFO in my torchrun launcher revealed the fix: peer-to-peer transfer rates climbed from 14 GB/s to 22 GB/s once I corrected PCIe bifurcation settings in BIOS and pinned NUMA affinities. Hidden bottlenecks vanish once you illuminate them with tooling already in your stack.
Real Benchmarks — What This Cluster Runs That a Single Card Cannot
Head-to-head against an RTX 4090 (24 GB VRAM) using identical prompts through vLLM:
| Model | Batch Size | RTX 4090 | This Cluster (9× 3090) | Notes |
|---|---|---|---|---|
| Llama 3.1 70B (FP16) | 1 | OOM | 48 tok/s | 4090 can’t load 140 GB model |
| Llama 3.1 70B (Q8) | 1 | 12 tok/s | 47 tok/s | ~4× speedup |
| Llama 3.1 70B (FP16) | 8 | OOM | 340 tok/s | Distributed only |
| Mistral Large (FP16) | 1 | OOM | 72 tok/s | Distributed only |
| Mistral Large (FP16) | 8 | OOM | 560 tok/s | Distributed only |
The consumer card hits memory walls at batch size > 1 on full-precision 70B models. OOM kills inference immediately.
Why not just quantize? Because I tried. GGUF Q4 introduces numerical drift that accumulates across long contexts. I watched diagnosis codes and statute references corrupt after 4,000+ tokens. If you’re generating anything where a wrong digit matters — medical, legal, financial — FP16 is the only option.
The throughput cliff. Below four concurrent requests per second, vLLM’s continuous batching scheduler artificially pads batches to fill VRAM allocation windows. Latency spiked from 45ms to 180ms per token. I fixed this by setting --max-num-batched-tokens=2048 and --max-num-seqs=4 to prevent over-batching at low traffic.
Power delivery — the second wall. Nine RTX 3090s at 350W TDP each = 3,150W GPU power alone. Add motherboards, CPUs, drives, and fans: combined peak draw sits at roughly 3,800W. Three dedicated 20A circuits become mandatory. A standard home office 15A/120V outlet delivers 1,800W before the breaker trips — you’ll trip it within minutes of a stress test.
Why not a fourth node? I tested it last October with spare hardware. Performance regressed 14% on concurrent throughput benchmarks. The culprit: PCIe fabric saturation. The lanes available across my current motherboard topology couldn’t sustain coherent cross-node traffic at that scale. Queuing delays outweighed additional parallelism. Three nodes on 25 GbE is the sweet spot for this hardware class.
Full Parts List
| Component | Per Node | Qty | Unit Cost | Total |
|---|---|---|---|---|
| AMD Ryzen 9 5950X | 1 | 3 | $280 | $840 |
| ASRock TRX40 Creator | 1 | 3 | $200 | $600 |
| 128 GB DDR4 ECC (4×32) | 1 set | 3 | $180 | $540 |
| RTX 3090 (used) | 3 | 9 | $450 | $4,050 |
| Samsung PM9A1 2TB NVMe | 1 | 3 | $120 | $360 |
| Mellanox ConnectX-4 Lx 25GbE | 1 | 3 | $45 | $135 |
| Noctua NF-A12x25 (fans) | 4 | 12 | $20 | $240 |
| Corsair RM1000x PSU | 1 | 3 | $170 | $510 |
| MikroTik CRS305-1G-4S+ | — | 1 | $140 | $140 |
| SFP+ DAC cables (1m) | — | 3 | $12 | $36 |
| Chassis + misc | 1 | 3 | $100 | $300 |
| Total | $7,751 |
Under $8,000 for a cluster that runs Llama 3.1 70B in full precision at 48 tokens per second, with room for batch inference that no single consumer card can touch. The H100 sitting in a datacenter at $30,000+ still beats it on raw throughput per watt — but not on dollars per token for a solo builder running inference from a basement rack.