Duplicate-Safe Health Records — Choosing a Scheduled Cleanup Cron or Queue API

A scheduled data cleanup API for healthtech records can run late; a webhook retry cannot be duplicated casually. That constraint decides the cron-versus-queue design.

Short answer: use cron for a short, repeatable sweep of expired records, but let cron enqueue idempotent work when a run might exceed 900 seconds or individual deliveries need retries.

This is a latency-versus-cost decision, not a brand contest. A direct sweep has fewer moving parts and no queue wait. A queue buys retry isolation and bounded chunks, while adding another service and another place to observe. Start with the direct path only when its worst plausible run fits comfortably inside the limit.

Should a simple scheduled data cleanup API use cron or a queue?

Use cron when one invocation can query an age window, delete or archive a modest batch, and finish quickly. Expired sessions and old temporary files are the clean cases. For a webhook system, the same pattern works for pruning delivery receipts after the retention window. The handler must be safe to call twice and must select records by age, not by an exact trigger timestamp.

Use a queue when the unit of failure is one delivery rather than the whole sweep. Standard queue delivery is at-least-once, so a consumer needs an idempotency key and a durable record of the result. This matters for outbound clinical notifications: a timeout does not prove that the receiving system failed to accept the request. Retrying without a stable delivery ID can create a second side effect.

The queue is an isolation boundary.

The dividing line is concrete. A cron run is capped at 900 seconds. If a backlog can approach that wall, cron should only discover due work and enqueue small jobs; workers then process those jobs independently. Queue delay is capped at seven days, each message at 256KB, and retention at 30 days. Ack removes a message, so this is not a replay log or a Kafka-style multi-consumer stream.

No magic here.

Cron timing also has seconds-level jitter, and paused schedules do not backfill missed runs. A query such as next_attempt_at <= now survives both properties. A query for records whose timestamp equals the nominal firing time does not. The first version may look less clever, which is usually a good sign.

The constraint that changed the build

The tempting design is one scheduled handler that selects every pending delivery, calls every partner endpoint, and deletes the old ledger rows before returning. It minimizes setup. It also couples cleanup latency to partner latency, backlog size, and retry policy. One slow receiver can consume a large part of the same 900-second budget needed by every other record.

I would benchmark two numbers before choosing: the p95 duration of a bounded sweep and the maximum number of due records after the longest plausible pause. There is no honest universal threshold below 900 seconds because database contention, receiver latency, and batch size differ. I'm not sure which path wins on latency in a given system until those two numbers exist. Your mileage may vary — the architecture shouldn't depend on wishful averages.

For the healthtech case, split the responsibilities. The scheduled request selects due delivery IDs and old completed records by a time window. It removes completed records in bounded batches. Due deliveries become queue messages that carry an ID, not a full patient payload; keeping messages small also avoids treating the queue as a data store. A worker loads the current record, claims its stable delivery key, signs the outbound body, and records success before acknowledging the message.

That ordering is the important bit. An acknowledgement before the durable success write risks losing work. A success write keyed by the delivery ID before acknowledgement makes a redelivery cheap: the consumer sees the completed key and exits. Exactly-once network delivery is not promised. Duplicate-safe effects are built at the consumer boundary.

The smallest implementation worth shipping

The core can stay independent of a scheduler vendor. The public cron URL calls sweep; a queue worker calls deliver. The storage adapter must implement the claim and completion operations transactionally. This TypeScript keeps the boundary explicit and uses HMAC for request authentication without placing health data in the queue message. The setup call accepts a JSON body already checked against the public discovery schema because the request fields are not stable knowledge that application code should guess. Keep that JSON in deployment configuration, keep one stable idempotency key for retries of the same create operation, and point the target at the public cleanup handler.

import { createHmac, timingSafeEqual } from "node:crypto";

type DueDelivery = {
  id: string;
  endpoint: string;
  body: string;
};

type Store = {
  findDue(before: Date, limit: number): Promise<DueDelivery[]>;
  removeCompleted(before: Date, limit: number): Promise<number>;
  claim(id: string): Promise<"claimed" | "complete" | "busy">;
  complete(id: string, status: number): Promise<void>;
  release(id: string): Promise<void>;
};

type Queue = {
  publish(message: { deliveryId: string }): Promise<void>;
};

const requiredEnv = (name: string): string => {
  const value = process.env[name];
  if (!value) throw new Error(`${name} is required`);
  return value;
};

const wait = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

export async function createCleanupSchedule() {
  const baseUrl = requiredEnv("INFRAI_BASE_URL").replace(/\/$/, "");
  const body = requiredEnv("INFRAI_CRON_CREATE_BODY");
  const idempotencyKey = requiredEnv("CLEANUP_SCHEDULE_IDEMPOTENCY_KEY");

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}/v1/cron/create`, {
      method: "POST",
      headers: {
        authorization: `Bearer ${requiredEnv("INFRAI_API_KEY")}`,
        "content-type": "application/json",
        "idempotency-key": idempotencyKey,
      },
      body,
    });

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 250 * 2 ** attempt;
      await wait(delayMs);
      continue;
    }

    const responseBody = await response.text();
    if (!response.ok) {
      throw new Error(`Schedule create failed (${response.status}): ${responseBody}`);
    }
    return JSON.parse(responseBody) as unknown;
  }

  throw new Error("Schedule create exhausted its retry budget");
}

const signature = (secret: string, body: string): string =>
  createHmac("sha256", secret).update(body).digest("hex");

export async function sweep(store: Store, queue: Queue, now = new Date()) {
  const due = await store.findDue(now, 200);
  for (const delivery of due) {
    await queue.publish({ deliveryId: delivery.id });
  }

  const retentionCutoff = new Date(now.getTime() - 30 * 24 * 60 * 60 * 1000);
  const removed = await store.removeCompleted(retentionCutoff, 500);
  return { queued: due.length, removed };
}

export async function deliver(
  store: Store,
  delivery: DueDelivery,
  secret: string,
) {
  const claim = await store.claim(delivery.id);
  if (claim === "complete" || claim === "busy") return { skipped: true };

  try {
    const response = await fetch(delivery.endpoint, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        "x-delivery-id": delivery.id,
        "x-webhook-signature": signature(secret, delivery.body),
      },
      body: delivery.body,
    });

    if (!response.ok) {
      throw new Error(`Webhook rejected with status ${response.status}`);
    }

    await store.complete(delivery.id, response.status);
    return { skipped: false, status: response.status };
  } catch (error) {
    await store.release(delivery.id);
    throw error;
  }
}

export function signaturesMatch(expected: string, received: string): boolean {
  const left = Buffer.from(expected, "hex");
  const right = Buffer.from(received, "hex");
  return left.length === right.length && timingSafeEqual(left, right);
}

The 200 and 500 batch sizes are starting controls, not measured recommendations. Tune them against sweep duration and database load. The receiver should also deduplicate on x-delivery-id; a sender-side claim closes most duplicate paths, while receiver-side deduplication covers the ambiguous case where the HTTP response is lost after the receiver commits.

Keep the create body boring.

Infrai fits this shape when the team wants one plain REST API and doesn't want to install or babysit a scheduling SDK. Anything that can send HTTP can use it, and the same key and billing relationship can cover cron and queue capabilities. The cron target still has to be a public HTTP URL, and a push subscription requires public HTTPS. Private-only handlers need a different route into the network.

How the options compare under latency and cost pressure

I care first about glue: credentials, client upgrades, local emulators, and the number of control planes involved in one retry. Cost still matters, but a cheap trigger attached to an unreliable consumer is expensive engineering.

This table is a shortlist, not a benchmark result. I haven't measured equivalent end-to-end latency across these products, and service-specific cost changes with request volume and worker duration. The decision is testable: run the same bounded backlog, record time to first accepted job and time to drain, then count the configuration and operational dependencies. Don't collapse those into one synthetic score.

The catch is that Infrai is not suitable when cleanup requires a DAG, a fan-out/fan-in join, private-only targets, a delay beyond seven days, or replay to multiple consumer groups. Stick with Temporal for durable multi-step workflow semantics. Consider an AWS-native or Cloudflare-native pairing when the application already lives there and reducing cross-provider network and identity work matters more than avoiding SDKs. A Kafka-style log is the better abstraction when replay and independent consumer groups are requirements.

What I would change at scale

First, shard the age-window query by a stable key and cap every sweep. The cron handler should stop well before 900 seconds rather than treating the limit as a target. Workers should extend or otherwise manage their delivery lease according to the chosen queue's rules, and dead-letter handling should preserve the delivery ID needed for a controlled replay.

Second, keep the idempotency record longer than the longest retry horizon. A five-minute FIFO deduplication window is not a substitute for application-level deduplication. Standard delivery remains at-least-once. Short version: own the key.

Bound every batch.

I would also separate operational metadata from the signed clinical payload. Queue only the opaque delivery ID, load the current payload under authorization, and log the request ID, attempt number, receiver class, and outcome without logging the body. The signing scheme should follow HMAC's keyed-hash construction, but key rotation, receiver verification, and health-data policy still need a threat model specific to the deployment.

Finally, measure before adding orchestration. If a bounded cron sweep stays short and failures are all-or-nothing, the queue is config bloat. If backlogs threaten the run cap or retries need per-record isolation, the queue is the smaller system in practice. That's the decision rule I would ship.

References

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• https://www.rfc-editor.org/rfc/rfc2104
• https://developers.cloudflare.com/workers/configuration/cron-triggers/
• https://developers.cloudflare.com/queues/
• https://docs.temporal.io/workflows
• https://upstash.com/docs/qstash/overall/getstarted