Why We Started With a Monolith

That wrongness started with a deliberate choice. Kevin’s original app was a CRUD storefront. Products, carts, orders, users. Nothing that demanded distributed thinking. Speed to market won the argument. A single Rails monolith on Heroku took three days to ship; the alternatives required more ceremony than the product had features.

The codebase sat at roughly 15,000 lines across forty-some models. One database, one queue, one deploy target. It worked beautifully for two years. The cracks didn’t announce themselves with fanfare. They arrived as slow pagination on an orders index page that once snapped open in 40 milliseconds.

Then came the background job collisions. Sidekiq workers stepped on each other’s writes when inventory counts updated faster than the locks could clear. Every fix made something else worse. Adding indexes bought breathing room until the queries outgrew them again. Caching introduced staleness bugs that were harder to reason about than slow responses ever were.

The real problem wasn’t performance though. It was cohesion priced as complexity: any change to checkout logic required touching four files across three concerns, and testing one flow meant replaying twenty manual scenarios in staging. I knew the industry answer would be microservices eventually. But rewriting everything felt like throwing away hard-won domain knowledge just to trade one set of boundaries for another.

I rebuilt it differently. Same monolith philosophy, but with internal seams sharp enough to sever later. Merriam-Webster defines an “event” as a happening without intent or plan. Mine had neither; it was pure engineering. The rest of this piece covers what broke in unexpected places when I cut those seams.

The First Seam Cuts Deepest

The first break came at 2:47 AM. A payment callback from Stripe timed out, and the entire checkout flow froze because order creation, inventory reservation, and email notification all shared one database transaction. Monoliths fail loudly when coupling is tight. Our OrderService called InventoryService directly. No interface, no event bus, a hard method reference across package boundaries. When inventory threw an exception during peak traffic, the cascading rollback killed every in-flight purchase.

I started by extracting the event publisher from our PostgreSQL-backed job queue. Each module now emits domain events through a message broker running on RabbitMQ instead of making synchronous calls between services. The change took three days to implement but exposed something uncomfortable: my “monolith” was five implicit services duct-taped together with method invocations. The failure trigger wasn’t the code itself. It was the shared transaction boundary that made one slow query in reporting bring down order processing simultaneously.

The 500ms Wall

We crossed it without noticing. What began as a snappy API now sat behind p95 latency numbers that made stakeholders wince during demos. The degradation wasn’t dramatic. It crept up at roughly the pace our service count grew past a dozen internal calls per request. Every feature meant another hop. A typical user action cascaded through five or six services, each adding its own serialization overhead, network round-trip, and retry logic.

By the time we hit 10,000 lines across the codebase, bug frequency started climbing in lockstep with complexity. Correlation isn’t causation, but the pattern was hard to ignore: more modules, more integration points, more ways for a schema change to break something three layers deep. I kept a spreadsheet of incidents. Not because management asked. Because I needed to see the shape of our failures.

The trend line showed defects clustering around service boundaries, not business logic. We weren’t writing bad code; we were writing code that had too many handshake points with other code. A single trace spanned 14 services, each with its own logging format and timezone handling quirks. Tracing tools like Jaeger helped, but they couldn’t fix the underlying problem: our architecture had become a distributed monolith wearing an event-driven costume.

The breaking point came on a Tuesday afternoon. A routine deployment of one service triggered cascading timeouts in three others because of an unversioned message contract we’d all agreed “would never change.” That agreement lasted exactly eleven weeks. The fix felt heretical at first: collapse it back into one deployable unit while keeping the event-driven discipline intact internally. Modular monolith patterns gave us clean boundaries without distributed failure modes. But that’s section six.

First I need to explain what we actually built when we tore down the walls and rebuilt them closer together.

The Bus Emerges from the Walls

That transition felt natural — until our first cross-module event took 40 seconds round-trip through synchronous REST. Two services holding a shared transaction open was the breaking point, not the traffic. We landed on RabbitMQ 3.13 with a dead-letter queue and two exchange types. Topic exchanges handle domain events, while direct exchanges route command-style messages. No Kafka, no Kinesis. We needed something we could run on three modest VMs without a dedicated ops team.

And RabbitMQ fit that bill with its built-in management UI and 12 MB idle footprint.

The pub/sub distinction mattered more than I expected. A message broker guarantees delivery; pub/sub just fires notifications into the void. Our order service publishes order.created to a topic exchange. Both inventory and billing bind their own queues to it independently. Message ordering became the hidden tax.

RabbitMQ preserves order per queue, but that guarantee evaporates once you fan out to multiple consumers. Partitioning by aggregate ID is the only way to keep things sane. We route every event containing orderId through a consistent hash binding. That keeps all events for one order in a single queue.

The retry ladder lives in our dead-letter exchange configuration: three immediate retries at the consumer level, then re-publish with exponential backoff starting at 5 seconds and maxing at 2 minutes after five failed cycles.

Idempotency keys saved us twice in the first week alone. Duplicate deliveries are normal when your consumer crashes mid-acknowledgment. We stamp every event with a UUID and check Redis before processing. That check costs about 1 millisecond of overhead but eliminates entire classes of corrupted state bugs. Synchronous fallbacks still exist where they must. Payment confirmation returns via webhook because our payment processor never learned to speak AMQP.

The Hosting Bill Settled the Debate

That webhook handshake forced the real question: could we run a broker on a modest monthly budget? RabbitMQ on a 1GB DigitalOcean droplet worked, but only just. Every queue consumer competed with the OS for memory. Kafka was never seriously considered; three brokers would’ve blown our entire budget on infrastructure alone. Redis Streams emerged as the pragmatic winner. We already paid for a managed Redis instance through Upstash, and adding streams cost nothing extra.

The consumer groups API felt familiar after years of using Redis lists for job queues, so the learning curve was basically flat. The trade-offs were real. RabbitMQ offers per-message routing that Redis simply cannot match, and its dead-letter exchanges are far more mature than anything in XADD territory. But we needed maybe four queues total, not a topology lab experiment.

Our selection matrix had exactly five rows: cost, operational familiarity, message persistence, consumer group semantics, and failure recovery. Redis won four of them outright. Only durability gave RabbitMQ an edge, and even that faded once we enabled append-only file persistence at 1-second intervals. Kafka never made it past row one. A single-node setup technically fits under budget, but you’re running ZooKeeper or KRaft alongside it on the same box.

That’s not an event bus; that’s a second full-time job. Three months later, the streams have handled roughly 40 million messages without a single lost event. Our latency sits around 5 milliseconds end-to-end within the same region. Fast enough for everything except those payment confirmations still crawling back over HTTP. We spent less on messaging than most teams spend on coffee runs during architecture meetings.

The First Week Broke Things

The savings didn’t survive contact with production. Seven days after flipping traffic to the event bus, duplicate orders hit our Postgres database. Two consumers retried the same Kafka message at 14:32 UTC. Neither had idempotency keys, so both wrote rows. Double shipments followed, then angry customers.

A panicked rollback took 40 minutes while I stared at CloudWatch alarms. The fix was embarrassingly simple. Every producer now attaches a UUIDv7 to the message envelope. Consumers check Redis before processing anything. That single change eliminated the entire failure class.

We also learned that acknowledgment semantics matter more than throughput. Our first implementation used auto-ack on receipt. A crash mid-processing silently lost events that way. Switching to manual acknowledgment with a 30-second visibility timeout cost us roughly 15% raw throughput, but it removed the worst failure mode entirely. The wrk benchmarks told an interesting latency-versus-volume story.

Sync REST calls handled 850 requests per second under load testing on my laptop, with p95 latency around 210ms. The same workload through RabbitMQ with three consumers sustained 1,400 messages per second, though p95 jumped to 340ms due to queueing backpressure. Raw speed wasn’t the point though. The async path stopped blocking on downstream services entirely. Before adoption, a slow payment gateway dragged every order request down with it.

One bad dependency meant whole-system latency spikes that lasted until I manually killed connections from EC2 instances via aws ec2 reboot-instances. After moving off synchronous calls toward queues, a sluggish consumer only backs up its own partition rather than poisoning unrelated traffic. Monthly AWS costs actually dropped once we shed those idle connection pools and redundant retry loops from API Gateway timeouts alone.

Idempotency keys solved correctness; manual acks solved reliability; async boundaries solved blast radius. So refactor phase #2 felt almost boring by comparison.

The Bills Came Due Eventually

That monthly savings didn’t happen overnight. It took three failed rollbacks, two all-nighters, and one very honest postmortem before the architecture earned its keep. The painful part wasn’t the code. It was admitting that my initial event-driven design had a fundamental flaw: I treated distributed transactions like they were still ACID.

Payment settlements and inventory decrements don’t belong in the same saga step when your payment provider’s webhook latency varies by 4 seconds depending on which region routes the request.

Postgres advisory locks saved us where optimistic locking failed. A pg_try_advisory_xact_lock() call around each inventory adjustment forced serialization without holding row locks open during network calls. That one change eliminated the phantom oversell bugs that only appeared at 200 concurrent checkouts. Timing diagrams became our religion.

We printed sequence diagrams for every saga flow and pinned them above the monitor; when a race condition surfaced mid-load-test, we could trace it to a missing compensation handler within minutes instead of spelunking through logs for hours.

Sagas need timeouts everywhere. Our Stripe webhook handler initially waited indefinitely for inventory confirmation; now every step carries a 30-second ceiling, and anything slower triggers a compensation flow that refunds the customer automatically. Event schemas versioned in a shared registry caused more friction than benefit. Breaking changes happened anyway, so we stopped pretending backward compatibility mattered. Consumers got exactly one release cycle to migrate.

Kafka’s consumer groups handled replay cleanly once we stopped using auto-commit offsets. Manually tracking them inside Redis instead gave us surgical control over which messages redelivered after failures, at the cost of approximately 40 extra lines per service. I rebuilt this monolith because events promised freedom from deployment coordination hell. That promise held only after I embraced eventual consistency as a design constraint rather than an exception case requiring apologies during incident reviews.

If I had to distill this whole experiment into one truth, it’s this: coupling is a spectrum, not a switch. I didn’t eliminate my architectural pain; I just traded a slow, predictable bottleneck for a dozen fast, chaotic ones. That trade was worth it. The system breathes now because failure is compartmentalized. A dead-letter queue is a lot easier to stare at than a cascading lock escalation on PostgreSQL.

Keep Reading

The real question you should ask before you start pulling threads out of your own monolith isn’t “will this scale?” It’s “am I prepared to debug my own event ordering at 3 AM with fresh eyes?” Because that’s the job. The infrastructure got smarter, but the debugging got weirder. If that trade sounds fair to you, grab Kafka and start cutting.