My RabbitMQ library looked production-ready. It couldn't survive a single reconnect.

For a couple of years I had a RabbitMQ client library sitting in a private repo. On paper it was impressive: automatic reconnection, channel pooling, a circuit breaker, four rate-limiting strategies, gzip compression, dead letter queues, worker-thread consumers, delayed messages. Over 1,200 lines in the main class alone.

I never quite trusted it enough to publish. Recently I decided to find out whether that instinct was right.

It was. The single most important feature of a message-broker client — surviving a connection drop — had never worked. Not "worked poorly". Never worked. And nothing in the repo could have told me, because there wasn't a single test.

This is the story of the audit, the rebuild, and the release of @pinceladasdaweb/rabbitmq. It's also a collection of lessons I wish I'd internalized years earlier, because none of these bugs were exotic. They were all sitting in plain sight, waiting for a failure path to be exercised for the first time in production.

The audit: what "looks done" actually hides

The library had the shape of a finished product. Rich API, detailed README, twenty example scripts. Here's a sample of what a deep read of the code actually found.

The reconnection that reconnected to nothing. The connection layer dutifully detected drops, retried with exponential backoff, and re-established the TCP connection. Then... nothing. The channel pool — the thing every publish and consume goes through — was only ever created inside connect(), and the reconnection path never called it. After any network blip, every operation would throw Not connected forever, while the connection state proudly reported connected. The recovery code existed; it recreated consumers using channels captured in closures from before the drop — channels that were dead. It had clearly never been executed.

The rate limiter that crashed on its own feature. The block-a-key feature stored blocked keys in a Set... and called this.blocked.set(key, Date.now()) on it. Set has no .set(). Calling blockKey() threw a TypeError, every time, since the day it was written.

The circuit breaker state that didn't exist. The public API exposed getCircuitBreakerState(), which called this.#circuitBreaker.getState() — a method the circuit breaker class simply did not have.

The dead-letter helper with the wrong signature. moveToDeadLetter() called the publish method with four arguments in the wrong order, so the name of the queue was published as the message body, to the wrong exchange.

The library that killed your process. The constructor registered SIGINT, SIGTERM, uncaughtException and unhandledRejection handlers — and called process.exit() in them. A library. Any unhandled rejection anywhere in your application would cause my dependency to shut your process down.

None of these are subtle bugs. They're the kind of thing the first test you ever write catches. Which brings me to the first lesson.

Lesson 1: features are cheap; invariants are expensive. An API surface can look complete while every failure path is fiction. The happy path is maybe 20% of an infrastructure library — the rest is what happens during the bad minutes, and that's exactly the part that "it runs on my machine" never exercises.

If you claim resilience, your CI has to actually break things

The rebuild started with the reliability core: reconnection has to restore the channel pool, re-assert topology, and recreate every consumer on fresh channels. But I'd been burned by "code that looks like it recovers", so the real work was making the claim falsifiable.

The integration suite now runs against a real RabbitMQ (Docker locally, a service container in GitHub Actions) and includes this test — the one I care most about in the entire repo:

test('recovers channel pool and consumers after a forced connection drop', async () => {
  const reconnected = new Promise(resolve => rabbitMQ.once('reconnected', resolve))

  // Ask the broker to close every AMQP connection, simulating a real outage.
  await forceCloseAllConnections() // DELETE /api/connections/* via the management API

  await reconnected

  const countBefore = received.length
  await rabbitMQ.publish(ROUTING_KEY, { after: 'reconnect' })

  await waitFor(() => received.length > countBefore)
  assert.equal(rabbitMQ.getClusterStatus().connectionState, 'connected')
})

No mocks. The broker genuinely kills the connection; the library genuinely has to come back: pool rebuilt, exchange re-asserted, consumer re-subscribed on a new channel, message flowing again. This runs on every pull request.

Using the management HTTP API for the kill switch (instead of docker restart) was a small trick that paid off: it works identically on a laptop and inside a CI service container, and it's fast.

Lesson 2: a resilience feature without a test that injects the failure is a rumor. The test that kills your broker connection is worth more than the thousand lines of recovery code it validates.

The adversarial review: 42 candidates, 10 confirmed bugs

With the core rebuilt and a test harness in place, I ran a structured review of the source: multiple independent passes over the code, each hunting from a different angle — line-by-line, lifecycle invariants, cross-module contracts, plus efficiency and design-depth sweeps. Every candidate finding then got adversarially verified against the code before I accepted it.

Ten confirmed bugs came out. Three of my favorites, because each one represents a category:

1. The option that silently discarded messages.

async retryOperation (operation, maxRetries = 3, delay = 1000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    // ...
  }
}

Spot it? Pass { maxRetries: 0 } — a perfectly reasonable "try once, don't retry" intent — and the loop body never runs. The operation is never attempted, channel.publish is never called, and the promise resolves successfully. The circuit breaker even records a success. Silent message loss, triggered by a documented option. The fix is one line (Math.max(1, maxRetries)), plus a regression test that pins the semantics forever.

2. The poison message that ate a consumer. A message with a corrupt gzip body would fail to decode, get nacked with requeue: true, be redelivered instantly, fail again... forever. With prefetch: 1, one bad message parked an entire queue while burning 100% CPU. Undecodable messages now go to the dead letter queue — failure handling should drain problems, not recirculate them.

3. The unroutable dead-letter that vanished. Moving a message to a DLQ published it without the mandatory flag. If the dead-letter routing had no binding — one topology mistake away — the broker would confirm the publish and silently drop the message. The method resolved, logged success, and the message ceased to exist. It now publishes with mandatory: true, listens for basic.return, and rejects loudly when the DLQ isn't routable.

There were seven more: a disconnect()/connect() cycle that permanently disabled auto-reconnection, consumer callback failures tripping the circuit breaker that gates publishing, a fire-and-forget publish that could hang forever awaiting 'drain' on a dead channel, duplicate consumers spawned by racing recovery paths...

Lesson 3: bugs cluster in the seams. Almost every confirmed finding lived at an intersection — between two recovery mechanisms, between an option's edge value and a default, between "the callback succeeded" and "the ack was delivered". Reviews that only read functions in isolation don't find these. You have to trace the interactions.

Your test suite also protects you from everyone else

Two things happened during this project that had nothing to do with my code, and the test suite caught both.

The dependency major-bump. Mid-rebuild, I upgraded amqplib from 0.10 to 2.0 — a jump across two majors of the library this whole thing wraps. The entire integration suite (confirms, compression round-trips, forced reconnections, delayed messages) passed against the new version in one afternoon. Without the suite, that upgrade would have been an act of faith deferred indefinitely.

The floating Docker tag. My CI used rabbitmq:4-management-alpine. One day that tag started resolving to RabbitMQ 4.3, which rejected the delayed-message plugin version I bundled (built for 4.0.x). CI went red before any user ever saw the problem. The fix taught me to pin brokers and plugins as a matched pair (rabbitmq:4.2-management-alpine + plugin 4.2.0) — a floating major tag in CI is a time bomb with a pleasant default.

Lesson 4: automated tests are not just about your bugs. They're your early-warning system for upstream majors, ecosystem drift, and infrastructure that changes underneath you.

Dependencies are promises

At the start, the library shipped four runtime dependencies, including pino and pino-pretty — a pretty-printer for development logs, in every production install.

It now ships two: amqplib and a small cache. The logger question is the interesting one. The library used to create its own pino instance at import time — its own format, its own destination, disconnected from whatever the host application does. But an infrastructure library shouldn't own a logging pipeline; it should write to whatever you give it:

import pino from 'pino'

const rabbit = new RabbitMQ({
  endpoints: ['localhost:5672'],
  logger: pino({ name: 'rabbitmq' }) // any pino/winston/bunyan instance works
})

If you inject nothing, a 25-line dependency-free console logger provides timestamped, leveled output. Nobody pays for a logging stack they already have.

The same thinking applied to process lifecycle: the library no longer touches your signal handlers. Graceful shutdown is one explicit call — rabbit.enableGracefulShutdown() — and it's opt-in, because your process belongs to you, not to your dependencies.

A small release-engineering war story

The publish pipeline (GitHub Actions → npm on every merge to main) had its own baptism by fire. Two merges landed on main in quick succession; two release runs raced each other. The first published 1.0.1 to npm, then failed pushing the git tag back (the branch had moved). The second run recomputed "next version = 1.0.1" from the now-stale package.json — and npm refused to publish over an existing version. Deadlock: every subsequent release run would fail the same way.

Two structural fixes, both worth stealing:

1. The registry is the source of truth. The workflow now asks npm what the latest published version is (npm view <pkg> version) and bumps from that, never from package.json. Whatever half-finished state a previous run left behind, the next run self-corrects.
2. Serialize your releases. A concurrency: { group: publish } block means rapid merges queue instead of racing.

Lesson 5: release automation fails in the middle, not at the edges. Design every step to be safe to re-run, and derive state from systems of record, not from files a previous failed run may or may not have updated.

Where it landed

The library is now public: @pinceladasdaweb/rabbitmq.

npm install @pinceladasdaweb/rabbitmq

import RabbitMQ from '@pinceladasdaweb/rabbitmq'

const rabbit = new RabbitMQ({
  endpoints: ['localhost:5672'],
  exchange: { name: 'orders', type: 'direct' }
})

await rabbit.connect()

await rabbit.subscribe('order-processing', async (content, message) => {
  await handleOrder(content) // auto-ack on success, nack → DLQ on throw
})

await rabbit.publish('order.created', { id: 42 }) // publisher-confirmed

rabbit.on('reconnected', () => console.log('survived an outage, consumers restored'))

What's in the box: automatic reconnection with full state recovery (pool, topology, consumers), publisher confirms on everything, per-key rate limiting (token bucket, leaky bucket, fixed and sliding windows), a circuit breaker that's isolated from consumer failures, transparent gzip compression, first-class dead-letter support (createQueue, moveToDeadLetter, processDeadLetterQueue), delayed messages via the official plugin, worker-thread parallel consumers with automatic respawn, dependency-ordered sequential processing, consumer lifecycle management (unsubscribe, consumerCancelled/consumerRecovered/consumerLost events), and TypeScript declarations checked in strict mode on CI.

Guarding all of it: 87 automated tests — 72 unit, 15 integration against a real broker including the forced-outage scenario — running on Node 22 and 24 on every pull request, plus 22 runnable examples that double as living documentation.

And in the spirit of the whole post, the honest part: this library has excellent test coverage and zero production mileage under other people's workloads. Those are different kinds of confidence, and only one of them can be built alone. If you run RabbitMQ in Node and any of this resonates, I'd genuinely love an issue, a hard question, or a war story from your queue topology: github.com/pinceladasdaweb/rabbitmq.

The library I was too embarrassed to publish taught me more in this rebuild than it did in all the years it sat in a folder looking finished. Turns out "looking finished" was the bug.