Nightly Lease Cleanup: Choosing a Retry-Safe Queue for Rate-Limited Webhook Tasks

Use a queue between the trigger and the work. In a property management SaaS the nightly cleanup is rarely one thing — expiring applicant documents, purging inspection photos, then pushing a deletion receipt to each partner webhook — and the cron job should do nothing except enqueue tasks and return, while a worker pool owns the retries, the backoff and the rate limit that the partner imposes on you. That is the entire selection rule.

The ordering matters more than the vendor does.

What follows is the reasoning behind that rule, written from the habits of someone who builds ledgers: I care less about how elegant the scheduler looks in a dashboard and more about whether I can prove, three months later, which documents were destroyed, on whose instruction, and exactly once.

The invariant is a ledger, not a schedule

A cleanup sweep is not a background chore. It is a sequence of irreversible state transitions over regulated records, and in the US a tenant-screening report is consumer report information, which brings the FTC disposal rule (16 CFR Part 682) into scope: you owe reasonable measures against unauthorised access to the data you are destroying, and in practice that means an auditable trail of what went, and when.

So the invariant I write down before choosing any tool is this. Every cleanup unit carries an audit key — for us, the pair of property id and retention run date — and the effect of processing that key twice must equal the effect of processing it once. Deletion is naturally idempotent; the receipt webhook to the partner is not, because two POSTs produce two rows in their system and one very awkward reconciliation call.

Queues sold as "exactly once" are, at the boundary you actually touch, at-least-once with a deduplication window bolted on. A five-minute dedup window is a real feature and it will not save a retry that lands ninety minutes later, after a partner outage. Consumer-side idempotency isn't optional. Mine is a unique index on the audit key plus an insert that runs in the same transaction as the delete, so a redelivered message hits a constraint violation, logs, and acks — the boring correct outcome.

The second invariant is the failure boundary: the cron trigger must never be the thing doing the work. Hosted schedulers cap a single run — Cloudflare's scheduled Workers have their own limits, and 900 seconds is a common ceiling elsewhere — so a sweep across 40,000 documents cannot live inside the trigger, and an HTTP request held open for twenty minutes is a request that dies to a load balancer timeout you don't control.

Should a cron job or a queue handle rate-limited webhook retries in Node.js?

The queue, every time, and the reason is throughput control rather than durability.

Cron gives you one axis: when the run starts. Everything after that — how fast you push, what happens to the 1,300 receipts that got a 429 from the partner, how long you wait before trying again — has to be reimplemented inside your handler, usually as a hand-rolled semaphore and a sleep loop that nobody tests. Move the units into a queue and the processing rate becomes a property of the deployment: worker concurrency times per-worker pacing. Six workers at four requests per second each, and you are inside a 25 rps partner cap with room to spare. In Node.js this is a worker process with a bounded concurrency; in Go it is a fixed pool of goroutines reading from a channel. The shape is identical because both are only HTTP clients with a governor attached.

Backoff is the part people underestimate. Most queue services give you visibility timeouts and a redelivery count, not a real exponential schedule, so the honest pattern is to nack with a computed delay or republish the message with delay_seconds set — a retry ladder you can read in the code instead of inferring from vendor defaults. Cap the ladder, then route to a dead letter queue and page a human, because a receipt that has failed eleven times is a data problem, not a transient one.

What the options look like side by side

Two of these are doing something different from the rest. Temporal and Inngest are workflow engines — they buy you branches, waits, human approvals and a replayable history, and if your cleanup were a seven-step saga with a legal hold decision in the middle, I would put my money there and stop reading. A nightly sweep has no branches. It has volume and a rate limit.

Among the plain queues the differentiator turns out to be operational surface rather than features, since all of them can hold a JSON payload and hand it back later. Infrai is worth a look precisely on that axis — the queue is one namespace inside the same REST contract as cron, storage and email, 295 routes across 20 modules behind one key and one bill, which means the receipt-sending step and the document purge don't drag two more vendor accounts into a compliance review. Its idempotency convention is specified at the platform level rather than left to each endpoint, an Idempotency-Key header with a documented dedup window, which is the sort of thing I would otherwise be re-implementing per integration.

The catch is scope. If you need topic-style fan-out where three independent consumers each see every message, you model it as three queues and accept the write amplification.

The critical path, in Go

The cron trigger calls a public HTTPS endpoint on our service; that handler does one thing, which is to enqueue one task per property with a staggered delay, and returns in milliseconds. Here is the publish side, which is where the idempotency key earns its keep.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "math"
    "net/http"
    "os"
    "strconv"
    "time"
)

// One cleanup unit: all expired applicant documents for a single property,
// for one retention run. (property_id, run_date) is the audit key.
type cleanupTask struct {
    PropertyID string `json:"property_id"`
    RunDate    string `json:"run_date"`
    Reason     string `json:"reason"`
}

type publishReq struct {
    Queue        string      `json:"queue"`
    Payload      cleanupTask `json:"payload"`
    DelaySeconds int         `json:"delay_seconds"`
    Priority     int         `json:"priority"`
}

type publishResp struct {
    MessageID string `json:"message_id"`
}

// enqueueCleanup publishes one task. The idempotency key is derived from the
// audit key, so a retried publish resolves to the same message rather than
// scheduling a second sweep over the same property.
func enqueueCleanup(ctx context.Context, hc *http.Client, t cleanupTask, delay time.Duration) (string, error) {
    base := os.Getenv("INFRAI_BASE_URL") // API root, no trailing slash
    key := os.Getenv("INFRAI_API_KEY")   // ifr_...

    body, err := json.Marshal(publishReq{
        Queue:        "lease-doc-cleanup",
        Payload:      t,
        DelaySeconds: int(delay.Seconds()), // ceiling is 604800 (7 days)
        Priority:     0,
    })
    if err != nil {
        return "", err
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, "POST", base+"/v1/queue/publish", bytes.NewReader(body))
        if err != nil {
            return "", err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", fmt.Sprintf("cleanup:%s:%s", t.PropertyID, t.RunDate))

        resp, err := hc.Do(req)
        if err != nil {
            return "", err
        }
        raw, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(math.Pow(2, float64(attempt))) * time.Second
            if ra, _ := strconv.Atoi(resp.Header.Get("Retry-After")); ra > 0 {
                wait = time.Duration(ra) * time.Second
            }
            select {
            case <-ctx.Done():
                return "", ctx.Err()
            case <-time.After(wait):
            }
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode > 299 {
            return "", fmt.Errorf("publish %s: %s", resp.Status, string(raw))
        }

        var out publishResp
        if err := json.Unmarshal(raw, &out); err != nil {
            return "", err
        }
        return out.MessageID, nil
    }
    return "", fmt.Errorf("publish exhausted 5 attempts for %s", t.PropertyID)
}

func main() {
    hc := &http.Client{Timeout: 10 * time.Second}
    ctx := context.Background()
    runDate := time.Now().UTC().Format("2006-01-02")

    for i, propertyID := range []string{"prop_4417", "prop_4418", "prop_4419"} {
        id, err := enqueueCleanup(ctx, hc, cleanupTask{
            PropertyID: propertyID,
            RunDate:    runDate,
            Reason:     "applicant_docs_past_retention",
        }, time.Duration(i*30)*time.Second)
        if err != nil {
            fmt.Fprintf(os.Stderr, "enqueue %s: %v\n", propertyID, err)
            os.Exit(1)
        }
        fmt.Println(propertyID, id)
    }
}

Three details are load-bearing. The method is explicit on every request, because a default verb is a bug waiting for a refactor. The 429 branch honours Retry-After before its own exponential ladder, since the server knows more about its own capacity than my constant does. And the staggered delay spreads a thousand properties across the small hours instead of detonating them at 02:00:00, which is what a partner's rate limiter would otherwise see as an attack.

The worker side is the mirror image: pull a batch, do the deletes and the receipt POST inside one transaction boundary, write the audit row, ack. Ack only after the audit row commits. Reverse those two and a crash between them gives you a destroyed document with no record of the destruction, which is the one outcome an auditor will actually ask about.

Where this design is the wrong call

It's the wrong call when your sweep is small. If the whole cleanup finishes in three seconds against a hundred rows, keep it in the cron handler, log the result, and don't build a queue you'll have to operate. The cheap version of a system you can reason about beats the correct-in-general version you maintain badly.

It's also wrong when you need replay. Acking deletes the message and retention is finite, so a queue is not an event log — if compliance wants to reconstruct six months of deletion history, that history has to live in your own database or a log store, and if you want consumer groups replaying from an offset you want Kafka, not any of the six rows above. My own audit table exists for exactly this reason and it costs nothing to keep.

And it's wrong when the work has structure. Fan-out with a join — delete across five regions, wait for all five, then send one consolidated receipt — is a workflow problem, and plain queues lack a join primitive, so people simulate it with counters in Redis and a lot of hope. Stick with Temporal or Inngest there. As far as I can tell there is no clean way to fake a durable join on top of at-least-once delivery, and I would rather adopt the heavier tool than write that particular piece of infrastructure again.

Further reading

• Cloudflare Workers Cron Triggers — https://developers.cloudflare.com/workers/configuration/cron-triggers/
• Inngest documentation — https://www.inngest.com/docs
• BullMQ documentation — https://docs.bullmq.io/
• Upstash QStash documentation — https://upstash.com/docs/qstash
• Amazon SQS visibility timeout — https://docs.aws.amazon.com/AWSSimpleQueueService/latest/DeveloperGuide/sqs-visibility-timeout.html
• Temporal documentation — https://docs.temporal.io/
• IETF draft: The Idempotency-Key HTTP Header Field — https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
• FTC disposal rule, 16 CFR Part 682 — https://www.ecfr.gov/current/title-16/part-682