The wildcard cert saga — eight days from expiry, three failed attempts, one flag

Cert-expiry monitoring is one of those systems that runs quietly for months and then, suddenly, starts asking for your attention with growing volume. On the Akash provider running out of the Sydney rack, the daily-check script had been raising the same warning for a week — quiet drumbeat, background noise until it wasn't. The wildcard cert covering the provider's public ingress had eight days to live.

Eight days is not a crisis. It's not comfortable either. The comfortable window for renewing a wildcard cert that fronts real tenant workloads is measured in weeks, not days, and I did not want to be doing this work at expiry-minus-hours with tenants noticing. So it moved to the top of the queue on 29 July, expiry set at 6 August.

The first instinct was the wrong one. cert-manager is normally auto-magical. It watches its Certificate resources, notices when they're within the renewal window, kicks off a fresh order against Let's Encrypt, and the operator finds out about it via absence — no notification because nothing broke. When the automation goes quiet, the reflex is to assume it just needs a nudge: restart the pod, re-issue the CertificateRequest, watch the reconciler pick things up again.

That reflex was wrong for reasons the first hour of investigation would surface. But the deeper story is what surfaced after that. Three failed renewal attempts. Root causes distributed across four different components — DNS provider, cert-manager, Cloudflare, acme.sh. Rate-limits triggered by the diagnosis itself. And ultimately a fix that landed as a single flag on an already-working tool, once the problem was framed correctly.

This piece is that story. The 8-day countdown, the three attempts that didn't work, the reframing that ended the incident in about half an hour of active work after days of distributed-bug hunting, and the four lessons that stuck.

// the first hour: what killed the renewal cron

Before any renewal attempt, the first hour went to config review. If cert-manager was supposed to auto-renew and hadn't, the question wasn't "what's wrong with the renewal now" — it was "what's been wrong for weeks that stopped the renewal from happening at all."

The acme.sh account configuration held the answer. Credentials were CF_Key + CF_Email — Cloudflare's Global API Key model. Those credentials had degraded, silently rejected for reasons not fully diagnosed at the time. The renewal cron had been executing on schedule the entire time. It had been failing at authentication the entire time. Neither of us — cron nor operator — had noticed. The immediate fix was rebuilding on scoped API Tokens rather than forensically reconstructing the failure cause — an auth-mode swap that doubled as overdue credential rotation.

Discovery mechanism matters. This wasn't found via log review. acme.sh's cron logs did contain terse authentication errors, but they were the kind of error that reads as normal noise if you're not already suspicious. The smoking gun was the config file itself — one line, CF_Key=..., unambiguous once you know Global Keys are being deprecated.

Auth was the technical trigger. The renewal cron dying was the symptom. The structural problem was that nobody was watching whether the cron actually succeeded. The cron was scheduled. The cron ran. The exit code was ignored. The distinction between "the cron executed" and "the cron did what it was supposed to do" was the gap monitoring should have caught, and didn't.

This is the class of failure that turns a routine credential rotation — normally a fifteen-minute job you do during a Cloudflare account maintenance session — into an 8-day fire drill. The credential rotation itself was trivial. The consequence of the credential rotation having happened silently, weeks earlier, without any monitoring signal that it had happened, is what pushed the deadline.

Cron exit codes without monitoring aren't automation. They're theatre.

// combining fixes: credential rotation plus architecture change

The natural response would have been to rotate to a scoped API Token, restart the renewal cron, and consider the incident closed. That's not what happened. The credential rotation got combined with a larger architecture change — one that had been on the "should probably do this eventually" list for months.

The reasoning was structural. Cloudflare's push away from Global API Keys wasn't arbitrary. Global Keys granted full-account write access with no audit trail — one credential capable of every action across every zone on the account, with no way to scope permissions or track which automation had done what. Scoped API Tokens fixed both problems: zone-limited, purpose-specific, individually revocable.

That same reasoning also applied to the DNS-01 challenge path itself. The existing wildcard cert renewal used acme.sh's default TXT placement — write records to the domain's authoritative zone at Cloudflare, remove them after validation. That mode required Global-API-Key-equivalent write access to the entire zone. A safer architecture would isolate the challenge writes to a dedicated CNAME-delegated location — SiteGround holding the primary nameservers, _acme-challenge CNAMEs delegating just the ACME work to Cloudflare, and API Tokens scoped to only that delegated zone.

Doing credential rotation and DNS architecture change as one combined fix meant less integration risk, less cleanup, and fewer moving parts to reason about afterwards. Two-in-one is usually a false economy — but here, the two changes were driven by the same underlying reasoning, and the fix stack for both landed in the same session.

// first failed attempt: cert-manager's zone-walking bug

With API Tokens in place and the CNAME delegation planned, the first renewal attempt went through cert-manager. And this wasn't a single-vector change. Three vectors were shifting at the same time.

The tool was changing: cert-manager as the new standard for the Kubernetes stack, replacing acme.sh. The CA was changing: Let's Encrypt as the new issuer, replacing the incumbent ZeroSSL. The challenge architecture was changing: CNAME delegation replacing acme.sh's default TXT placement.

Each had its own accumulated "should probably do this eventually" reasoning, and two of them were entangled. Wildcard issuance needs DNS-01 regardless of CA — the ZeroSSL setup was already doing that. What made CNAME delegation a prerequisite rather than a nicety was the credential model: default TXT placement required write access to the whole zone, so scoping the token to anything narrower meant moving the challenge records somewhere that existed only for that purpose.

Whether that was a wise bundling decision is worth naming: it was probably too many changes at once. Multi-vector migrations amplify the diagnostic difficulty when they fail, because you don't immediately know which vector is the problem. That was the lesson not yet learned going into the first attempt.

The attempt failed. But not in a way that produced an obvious error.

The failure signature was subtle. cert-manager wasn't returning an obvious error. The CertificateRequest sat in a Pending state — no failure event, no red status, nothing that would light up a dashboard. cert-manager's Cloudflare provider logs showed it querying zones, matching them, and then attempting to place the TXT record for the ACME challenge at a location. And Let's Encrypt's validator was looking for the TXT record at a different location. The challenge never validated. But nothing about the failure looked like a failure — every component was doing what its logs said it was doing.

It wasn't "this is broken." It was "this looks like it's working but the challenge never validates."

That's the class of failure that eats hours. When every log line reads as healthy activity, the diagnostic instinct — "find the error message and follow it" — has nowhere to go. There is no error message.

The trace came from reading cert-manager's Cloudflare provider source directly, understanding its zone-detection logic, and then cross-referencing that logic against the actual DNS zone contents of the affected domain. cert-manager's provider walks the DNS hierarchy from the record name up toward the root, looking for the "authoritative zone" — the zone where it should write the TXT record. That walk followed a CNAME chain. And the CNAME chain led to .snapshots.

.snapshots is a SiteGround backup-access hostname. It's part of the shared-hosting furniture — installed by the hosting provider for their own backup tooling, invisible to you because you didn't set it up and never think about it. If you're not a SiteGround-hosted domain operator you'd never look at it. If you are, you'd still never look at it, because it has nothing to do with anything you actively manage.

Except now it did. cert-manager's zone-walker followed the .snapshots CNAME, decided that was the authoritative zone for the ACME challenge, and placed the TXT record somewhere Let's Encrypt would never look. Every piece was behaving as documented. The whole still didn't work.

The teaching moment isn't about .snapshots specifically. It's about the class of failure where every component reads healthy and the composition fails silently. That class of failure needs a different diagnostic instinct than error-reading. It needs a diagnostic instinct closer to what network engineers use for packet capture — read the actual traffic between components, don't trust either component's log about what it thinks it's doing.

// second failed attempt: the retry loop became the story

With the .snapshots CNAME identified as the zone-walking trap, the second attempt tried a workaround: manually specify the zone cert-manager should target, bypassing the auto-detection. The attempt failed in a very different way.

cert-manager's default behaviour when a challenge fails is to back off and retry. Backoff intervals lengthen with each retry, but the retries never stop until the CertificateRequest is either manually paused or succeeds. That default is correct for transient failures — network hiccups, brief provider outages, momentary rate-limits. It's harmful for persistent failures on already-loaded infrastructure, where the retry adds pressure without being the sole cause of the problem.

The second attempt ran for about twenty minutes before I noticed it had shifted into a distinct failure mode. cert-manager's default backoff made each retry look reasonable in isolation — a single request every few minutes, exponentially spaced. Then Cloudflare 429 responses started appearing in the cert-manager pod logs — rate-limit rejections.

Here the mechanism deserves care. Cloudflare's global rate limit is 1,200 requests per five-minute window, counted account-wide across dashboard sessions, API tokens, and legacy keys combined. When the limit breaks, everything talking to that account gets blocked for the remainder of the window. cert-manager's backoff-driven retries alone weren't generating anything near 1,200 requests in five minutes — but the retry loop was probably enough additional load, on an account already carrying baseline API traffic from other automation, to push the account over the threshold.

Retries were the last straw, not the sole cause. Exact contribution unknown — request-rate telemetry from the incident window wasn't captured, and I can't reconstruct precisely how much of the budget the retry loop was consuming versus what was already in flight.

That makes the lesson more useful, not less. Account-wide rate-limit budgets mean isolated request rates aren't the whole picture. A retry loop that would be harmless on a quiet account can be the tipping point on a busy one. The operator awareness that matters: not "how many requests does this thing generate" but "how many requests does this thing generate on top of what's already there."

The 429s in the cert-manager logs meant other operations against the same account would have been affected during that window too — that's how account-wide limits work by definition. Whether specific other automation on the account actually noticed depends on whether it was making requests during the limited window; I didn't directly correlate specific other-tool failures during the incident.

The retry loop had become part of the problem.

Not the primary cert renewal being wrong — that was still the thing being diagnosed. But the diagnosis, in the form of cert-manager retrying against a persistently failing challenge, was contributing to a rate-limit condition that affected the whole account. Diagnostic activity was making things worse in ways I couldn't fully see.

The immediate fix was to pause the CertificateRequest — one kubectl annotate to stop cert-manager retrying — and wait for the rate-limit window to clear before touching Cloudflare again. That's the mechanical fix. The operator lesson is broader.

Retry loops without circuit breakers can amplify account-wide rate-limit conditions. cert-manager's default retry behaviour is fine for the failure modes it was designed for. It is not fine for the failure mode where the retry adds pressure to a rate-limit budget that's already tight for reasons outside the tool's visibility.

Circuit breakers — "stop retrying after N consecutive failures of the same type" — exist as a concept in most well-designed retry systems. cert-manager's Cloudflare provider doesn't have one aggressive enough to matter in this situation. Knowing that in advance would have saved twenty minutes of adding pressure to an already-loaded account.

// the pivot that wasn't really a pivot

At this point the shape of the incident changed. cert-manager's Cloudflare provider bug — the zone-walking through .snapshots — wasn't going to be a fix I could apply in-session. It was a bug in the provider code, and correcting it would need an upstream patch, testing, release. That timeline was days minimum, and the cert had single-digit days until expiry.

The path forward was cert-manager off the critical path. Which meant collapsing the three-vector migration down to two.

Framing that as a "pivot" isn't quite right. Two of the three vectors — Let's Encrypt as CA, CNAME delegation as challenge architecture — were staying. The tool vector was reverting. cert-manager was coming off; acme.sh was going back on. But acme.sh wasn't going to be doing what it had been doing before. It had been talking to ZeroSSL with default TXT placement. Now it would talk to Let's Encrypt with CNAME delegation.

Same tool. New work.

The mechanical work was smaller than that reframe implies. acme.sh was already installed. Its cron entry was already running — dead, from the failed authentication weeks earlier, but present. The credentials just needed rotation from Global API Key to API Token, already planned as part of the combined fix. The CA switch was one flag on the renewal command. And the challenge path needed pointing at the right place via --challenge-alias.

Or rather, needed to be pointed at the right place via --challenge-alias. The third obstacle was already waiting.

// third obstacle: the --challenge-alias discoverability gap

The third obstacle isn't quite fair to call a "failed attempt." It was more that the tooling had exactly the feature the situation needed, and the documentation didn't foreground it that way.

acme.sh supports DNS Alias Mode via the --challenge-alias flag. When you use CNAME delegation to redirect ACME challenges to a different zone — exactly the architecture the combined fix required — --challenge-alias tells acme.sh where to write the TXT record. Without it, acme.sh writes to the default location. With it, acme.sh writes to the aliased location. The flag is documented. The functionality works cleanly once configured.

But finding it required already knowing that's what you were looking for.

The acme.sh DNS Alias Mode wiki page describes the mechanism — "place TXT records at an aliased location via CNAME delegation." It does not describe the mechanism as "the solution to using base-zone NS delegation with a non-API DNS provider." Someone reading the DNS Alias Mode page top-to-bottom, without already knowing they need CNAME delegation for their specific problem, would come away understanding the mechanism but not connecting it to their situation.

I found it via a specific search for the problem, not by reading through the docs comprehensively. If I hadn't already been sitting with "I need to write TXT records at a location other than the default because SiteGround controls my authoritative nameservers and doesn't expose a DNS API," the wiki page would not have surfaced as relevant.

That's a documentation gap worth naming. It's not the same class of gap as missing documentation or wrong documentation. The docs exist. The docs are correct. The docs describe the mechanism accurately. What's missing is the connection between the mechanism and the class of problem it solves. Discoverability is a distinct axis from correctness, and the acme.sh DNS Alias Mode page is a case where correctness is high and discoverability is low.

Naming that gap publicly is more useful than pretending the answer was always obvious.

// the turning point

The turning point in the incident wasn't a moment of insight. It was a moment of reframing.

After the third obstacle, I sat down with the acme.sh docs again. Second read-through. Different question this time. The first read had asked "how do I do DNS-01 challenge with acme.sh?" — that had produced generic DNS-01 documentation, general-purpose, not obviously connected to the specific architecture I now had in place. The second read asked "what's the flag for the setup I've just configured?" — which is a much smaller, much more specific question.

The problem was framed correctly this time. So the answer was findable.

--challenge-alias surfaced immediately once I knew to look for the flag that told acme.sh "the TXT records go to a different zone than the domain's default authoritative one." That's the whole feature description in one line, and it maps directly to what the CNAME delegation was doing at the DNS layer.

What made this a turning point rather than "another thing to try" wasn't the flag itself. It was the shape of the fix collapsing at that moment.

The cert-manager path had been: diagnose the distributed bug across the zone-walker and the DNS layer, wait for the upstream patch, deploy the patched version, hope no new failures surface. Multi-day, multi-component, uncertain.

The acme.sh path was: add two CNAMEs at SiteGround, add one flag to an already-working cron entry, restart the cron, done. Single-digit hours, two-component, deterministic.

When the plan collapses from "diagnose distributed bug across three components" to "add a flag to an already-working tool," you know before you run it whether it's going to work. That's the anti-heroic version of the story. Not a moment of insight. A moment of reframing that made the answer obvious.

// the fix that landed

The mechanical fix was smaller than the diagnosis suggested it would be.

Two CNAMEs at SiteGround. _acme-challenge for the base domain pointing to a Cloudflare-hosted alias zone. The wildcard-challenge variant pointing to the same alias zone. SiteGround's DNS management supports CNAME creation via their control panel; no API required.

At Cloudflare, a dedicated zone for the alias, with a scoped API Token that had write access only to that zone. Not the domain's actual authoritative zone. Not the full account. Just the ACME challenge zone.

At acme.sh, the credential rotation from Global API Key to API Token in the account configuration. Two additions to the renewal command in the cron: --server letsencrypt to explicitly target Let's Encrypt (rather than acme.sh's ZeroSSL default), and --challenge-alias pointing at the delegated location. That's the whole acme.sh-side change.

At the Kubernetes layer, no change was needed. The renewed certificate would be dropped into the same Kubernetes secret acme.sh had been maintaining all along. The Gateway resource watching that secret would pick up the update automatically. Which is where NGINX Gateway Fabric earned its keep in this incident — but that's the next section.

The renewal ran clean the first time under the new configuration. acme.sh requested the challenge from Let's Encrypt, wrote the TXT record to the alias zone at Cloudflare via the scoped API Token, Let's Encrypt validated against the correct location (because CNAME delegation resolved correctly from Let's Encrypt's side), and the fresh wildcard cert issued. Total active work from "the flag surfaced" to "renewed cert deployed" was about half an hour — most of that spent waiting for SiteGround's NS delegation to propagate.

Two DNS records. Two flags. One credential rotation. And two vectors of the original migration (Let's Encrypt as CA, CNAME delegation as challenge architecture) preserved cleanly — cert-manager came off, but the harder-to-reverse changes stayed.

// NGINX Gateway Fabric added one thing for free

One small win worth naming, because it saved a step during the incident that would have otherwise been manual.

The certificate that acme.sh renews lives in a Kubernetes secret. The ingress layer that serves the certificate to inbound HTTPS traffic — in this stack, NGINX Gateway Fabric — reads that secret and terminates TLS with it.

Standard nginx-ingress-controller requires that when the secret changes, the ingress-controller pods either restart or be signalled to reload their configuration. Some deployments handle this via a sidecar that watches Kubernetes secrets and triggers reloads. Others handle it via periodic reload jobs. Some handle it by not handling it, and letting operators reload manually when they remember. All of those approaches add moving parts.

NGINX Gateway Fabric handles secret updates automatically. When the wildcard cert secret updated with the new acme.sh-issued certificate, the Gateway picked it up without any manual reload, without a sidecar, without a periodic job. TLS termination started using the new cert. No operator action required.

Not the story of this incident. But worth naming as a small win: the ingress layer was one of the few things in this whole stack that didn't need touching during the recovery. When something goes right during an incident, it's worth naming — especially when the same functionality in a sibling tool would have added another manual step under time pressure.

// what I'd make instinctive

Four lessons went into the runbook after this.

Cron exit codes need monitoring, not just cron execution. The renewal cron was scheduled correctly. It ran on schedule for weeks. Cron ran successfully — the shell command executed, cron logged the execution, the cron system considered the job complete. And every single execution was failing at authentication. Cron-ran-successfully monitoring is not the same as job-succeeded monitoring. Any critical automation — cert renewal, backup jobs, replication tasks, anything where "the cron ran" doesn't imply "the intended work happened" — needs both.

"Looks like working" is a distinct failure mode from "broken." cert-manager's zone-walking failure produced no error message, no failed status, no red dashboard indicator. Every log line read as healthy activity. The composition failed silently. That class of failure needs a diagnostic instinct closer to network engineering — read the actual traffic between components, don't trust either component's log about what it thinks it's doing. Error-reading skills don't apply when there are no errors.

Retry loops without circuit breakers can amplify already-loaded conditions. cert-manager's default retry behaviour was correct for transient failures and wrong for persistent ones. Twenty minutes of retrying against an already-loaded Cloudflare account was probably enough additional load to push the account into rate-limit rejection — not the sole cause, but a contribution the tool had no visibility into. Circuit breakers matter not just because they stop wasted work, but because they prevent the retry itself from becoming part of the failure. Know which of your automation has circuit-breaker behaviour and which doesn't.

Discoverability is a distinct axis from correctness. --challenge-alias works. Its documentation exists. What's missing is the connection between the mechanism the docs describe and the class of problem the mechanism solves. That gap turned a fifteen-minute config change into an hour of doc-reading. Documentation authors owe their readers not just correctness but discoverability — the framing that lets a reader find the right feature by describing their problem. Both are needed.

None of these are theoretical. Every one of them was learned by doing the diagnostic in the wrong order the first time.

// closing

Cert-manager saga posts tend to read as heroic sysadmin folklore. Someone stares at cryptic errors for days, has a moment of insight, cracks the case, writes it up as a triumph. The genre is well-established.

Most of them shouldn't be. The plans that collapse to a flag or a config change were never really the plans they looked like during the diagnostic phase. When the fix goes from "three-component distributed bug" to "two DNS records plus a config line," the diagnostic wasn't wrong — the frame was.

Reading this piece back, the whole incident could have ended on day one if the frame had been correct on day one. acme.sh was already in the stack. CNAME delegation was already a documented architecture pattern. --challenge-alias was already a supported flag. The pieces were all there. What wasn't there was the framing that connected them. cert-manager as the migration target was pulling attention toward the wrong tool for the specific problem the incident actually presented.

Article 8's GPU passthrough piece landed a similar-shaped observation about a different domain: the error pointed at the GPU, the culprit was the HBA. Different tools, different failure mode, same operator lesson. When the diagnostic is spending time in the wrong place, the frame is usually the problem — not the diagnostic.

If your renewal cron is running without monitored exit codes, that's the single highest-leverage thing you can fix this week. Not because it prevents this class of failure — but because it turns silent multi-week failures into loud immediate ones. Silent long failures are always worse than loud short ones. Fix the monitoring before you need it.