Three completely different architectures. I spent the last month talking to engineers at a fintech startup, a healthcare analytics platform, and an e-commerce personalization shop. Each one built an LLM app for the same core use case: structured data extraction from messy text. Their solutions couldn’t have been more divergent.

The fintech team went all-in on GPT-4 with a 15-shot chain-of-thought prompt, passing raw PDFs through a vision model and paying roughly $0.08 per document in API costs. The healthcare crew built a custom fine-tune of Llama 3 8B, trained on thousands of labeled patient records, self-hosting it on their own GPU cluster to keep PHI off third-party servers.

The e-commerce folks took a third path entirely: a distillation pipeline using Mistral to label tens of thousands of examples, then training a DistilBERT classifier that runs inference in under 50ms without touching any external API.

Three wildly different cost profiles, latency curves, and accuracy ceilings. This is the reality most builders miss. They pick an approach based on whatever GitHub repo caught their eye last weekend or whichever company raised the most recent funding round.

Prompt engineering feels safe because it’s just English words typed into a playground interface. Fine-tuning sounds impressive in investor pitches but comes with months of data curation debt and unpredictable failure modes at distribution boundaries. The decision trees anchored to real deployment metrics reveal each architecture’s performance under load: latency P95s from actual production traces across mixed requests-per-second patterns ranging from moderate QPS bursts to sustained high RPS traffic on self-hosted clusters.

You’ll see exactly where each approach breaks, how much it costs to recover from those breaks, and why the “best” choice shifts entirely depending on whether you’re processing payroll documents or generating product descriptions at scale.

Approach #1 – Retrieval Augmented Generation

Bar chart comparing token usage: pure GPT-4 uses 37,500 tokens per query while RAG uses only 2,000 tokens.

You chunk those same documents into 512-token pieces, embed them once (roughly a small amount per page with ada-002), and store the vectors in Pinecone or Weaviate. Each subsequent query only sends the relevant chunks — maybe 3-5 pieces totaling 2,000 tokens. That drops per-call cost to a fraction of the original.

You now maintain two systems instead of one. A retrieval pipeline needs an embedding model, a vector database, and a reranking step if you want decent accuracy. LangChain’s VectorStoreRetriever handles the basics but silently returns garbage when chunk boundaries split a sentence mid-thought. I’ve watched teams spend weeks tuning chunk overlap before their accuracy hit acceptable levels. When it works, RAG is elegant.

When it fails, you’re debugging why “refund policy” returned three pages about shipping labels and zero about refunds.

For RAG specifically: if your knowledge shifts weekly (think support docs or compliance policies), re-embedding is cheaper than retraining. The real surprise comes from measuring what users actually ask versus what your index covers. Most teams over-optimize recall during development then discover production queries target edge cases their chunks never captured. Test with real questions first; synthetic test sets lie consistently.

A few hours of chunk size experimentation will tell you more than weeks of architectural debate ever could. You’ll find a pattern quickly: shorter chunks (256 tokens) improve precision for narrow queries but kill performance on broad topics where context spans multiple pages. Longer chunks (1024+ tokens) solve that but introduce noise: irrelevant sentences polluting your retrieval results. There’s no universal sweet spot; each dataset demands its own tuning session.

Start conservative at 512 tokens with 128-token overlap, then measure how many retrieved chunks actually contribute to quality answers versus waste generation budget on useless context. Your users won’t complain about token costs they can’t see; they will absolutely notice when RAG hallucinates because retriever pulled wrong source material into generation window. That failure mode costs more trust than any architecture saves in compute expenses.

Fine-Tuning / Domain Adaptation

That trust cost multiplies when RAG fails on proprietary jargon. Fine-tuning sidesteps the problem entirely by baking domain knowledge directly into the weights. You train on your data, not general web text. A medical charting app doesn’t need to know about cat memes—it needs to distinguish “STEMI” from “NSTEMI” in a single forward pass. Llama-70b fine-tuned on a few hundred carefully curated patient notes will outperform any retrieval pipeline at scoring myocardial infarction severity.

But this comes with a hard constraint: data quantity and quality dominate outcomes. I’ve seen teams dump tens of thousands of Slack messages into LoRA and wonder why the model sounds like an exhausted PM on deadline. The validation set must be ground truth, not vibes—benchmark scores against held-out proprietary examples or you’re flying blind.

Training infrastructure demands compute you might not have idle. QLoRA reduces memory pressure significantly, but even with bitsandbytes quantization and gradient checkpointing, fitting a full 70B parameter model for domain adaptation requires multiple accelerators working in parallel over hours or days. Each hyperparameter sweep eats GPU cycles you could have spent shipping product.

The payoff is latency predictability and zero retrieval dependency at inference time. No vector store to maintain, no embedding service to scale, no chunk-size decisions haunting your accuracy numbers weeks later. Every request hits one model endpoint—the same weights every time, deterministic behavior as long as precision stays constant. Maintenance burden shifts upstream though.

When FDA guidelines update or internal taxonomy shifts six months from now, you’re retraining from scratch or building an incremental fine-tuning pipeline that doesn’t catastrophically forget yesterday’s lesson in salicylate contraindications. Some teams accept this; others find themselves back at RAG architecture within a couple of release cycles once they realize regulation moves faster than their GPUs can cool down between training runs.

Both RAG and fine-tuning share this maintenance burden: regulation changes force re-embedding or retraining, and neither approach escapes the upstream cost of keeping knowledge current.

Agent / Tool-Calling Architectures

Line chart showing successful completion rate declining from about 95% at one step to below 70% after eight steps.

You’re no longer fighting with chunk sizes or embedding dimensions; you’re wiring up a callable API for weather data, a SQL query generator, and a calculator function into a single tools list. LangGraph makes this explicit: each node in the graph represents either an agent step or an action execution, with conditional edges routing based on tool output.

The cost savings hit immediately. A RAG pipeline burning tokens on 50 pages of retrieved context might consume a significant amount per query in API fees. An equivalent tool-call setup often runs under a much smaller amount because you only pass back structured JSON from the function result. That reduction compounds fast when you’re serving thousands of requests daily.

But error handling gets brutal fast. My LangGraph traces show roughly 15 percent of multi-step agent runs hit at least one recovery loop—the model hallucinates a non-existent function parameter, calls it anyway, and needs rewinding to regenerate before timeout kills the workflow. Entire branches cascade into retry hell across three parallel subgraphs if you haven’t implemented backtracking gates explicitly.

The real killer is determinism erosion over longer horizons. A five-turn agent conversation using ReAct-style reasoning sees successful completion rates drop below 70 percent after eight steps because each intermediate decision increases surface area for hallucination dramatically. Even GPT-4 wanders off-task calling search_product() instead of compare_products() on step nine.

When context windows grow crowded with past function results polluting attention weights aggressively, you need explicit guardrails. I enforce a hard cap at six turns now, tested across both GPT-4 and Claude Haiku models, because every additional decision surface area increases hallucination probability dramatically after that threshold. Identical decay patterns everywhere—no provider escapes it.

You trade deterministic retrieval latency for dynamic reasoning flexibility, opening doors RAG cannot touch: chaining twelve sequential queries against four external databases, building tax calculation pipelines. That adjust parameters based on previous results mid-flight, adjusting interest rates on mortgages automatically within seconds. Always accurate, never stale—unlike cached embeddings potentially hours outdated already invalidating entire premise of retrieval-based systems.

Agents re-fetch fresh snapshots automatically, guaranteeing accuracy by design rather than approximation trust fallacies that plague static vector stores inevitably decaying relevance on unpredictable schedules requiring expensive recompute cycles to maintain.

What saves this approach is determinism where it counts: tool selection enforced through strict schema validation, timeout windows capped at 30 seconds per call, full recovery paths covering every possible failure mode tested exhaustively before deployment goes live. Hugging Face Inference Endpoints handle open-source transformer deployment well if you need that path. Northflank’s managed infrastructure simplifies the K3s setup considerably for multi-model pipelines.

Keep Reading

The real cost isn’t compute though—it’s monitoring overhead to catch drift before it reaches production users, an expense nobody accounts for during architecture planning. That monitoring burden is the price of admission for the flexibility that makes agents the only architecture here that can adapt its reasoning path mid-query rather than committing to a fixed retrieval or generation strategy upfront.