A rate-limited worker pool makes recurring user reminders an API and queueing problem before it becomes a cron problem. Keep the user's calendar intent, turn each due occurrence into a small job, and let workers drain that queue at a known rate. That adds some latency during a burst, but it protects delivery from a burst that exhausts the pool. The same rule applies to monthly and weekly reminders: cron starts the dispatch, while the worker budget decides how quickly the notifications leave the system.
Short answer: store an IANA timezone with the weekly or monthly rule, send a signed public webhook that only records and enqueues an occurrence, and make the worker idempotent. Treat latency and cost as explicit operating choices rather than promises hidden in a cron expression.
Begin with the worker constraint
A cron tick says “start,” not “how much work can run now.” In a B2B SaaS product, a tenant may schedule a digest for the same local minute as many other tenants. Imagine a Monday morning where hundreds of tenants become due together while the same worker pool is processing imports and exports. If every callback renders a template and waits for a provider, the burst consumes connection slots before the scheduler can report success. The scheduler should call an authenticated endpoint, the endpoint should persist an occurrence key, and a queue should hand work to rate-limited consumers. The callback returns after durable enqueue; template rendering and provider calls stay out of the request path. That split makes the latency-versus-cost decision visible: reserve capacity for fast dispatch, or accept queue age and batch the work.
I use (schedule_id, scheduled_for) as the business idempotency key. Queue delivery is commonly at least once, so a retry must map to the same job instead of sending a second reminder. A message identifier alone is too weak: the identifier can change while the user's intended occurrence has not.
Keep the rule visible in operations. Queue age, callback latency, duplicate suppression, and worker utilization tell you whether the pool is draining. CPU by itself does not.
One sentence matters here: bursts are normal.
What should a weekly or monthly reminder API promise about timezone webhooks?
The API should promise calendar intent, not a frozen UTC timestamp. “Tuesday at 09:00 in Europe/Berlin” should remain a local-time rule when daylight-saving rules change. Store the IANA identifier, local wall-clock time, frequency, and a policy version; compute the next UTC instant as derived state and recompute it after each accepted occurrence. A fixed offset such as +01:00 cannot describe future timezone transitions.
Monthly schedules need a declared policy for a date that does not exist. Reject it, clamp to the last day, or choose the next valid day. Any of those can be defensible; silently selecting one makes support and compliance reviews painful. Your mileage may vary when a tenant's sending window or provider quota is stricter than the worker pool.
The public webhook is a narrow bridge, not the reminder engine. Sign the exact raw body with a timestamped HMAC, reject stale timestamps, compare digests in constant time, persist the occurrence, and then return a success response. RFC 2104 defines the keyed-hash construction; the freshness window is an application policy that should match clock-skew and replay-retention decisions.
import hashlib
import hmac
def valid_signature(raw_body: bytes, header: str, secret: bytes, now: int) -> bool:
try:
sent_at_text, supplied = header.split(".", 1)
sent_at = int(sent_at_text)
except ValueError:
return False
# Five minutes is an example policy; choose a window for your clocks.
if abs(now - sent_at) > 300:
return False
signed = f"{sent_at}.".encode() + raw_body
expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, supplied)
The handler still has to enforce tenant authorization and an idempotency constraint. Authentication answers “who called?”; the occurrence key answers “have I accepted this calendar event already?”
Where do webhook and queue designs fail under load?
The first failure mode is useful work inside the callback. Rendering, calling an email or SMS provider, and waiting for a response couples an internet-facing timeout to every downstream dependency. Return after the ledger write and enqueue; let workers own retries, backoff, suppression checks, and provider-specific rate limits.
The second is an unexamined pause policy. Some schedulers skip occurrences while a schedule is paused, while a business may expect one compensating reminder after resume. Record the pause interval and choose explicitly between “send once after resume” and “skip missed occurrences.” Do not let a retry accidentally decide that policy.
The third is a calendar edge case hidden in a parser. Test a weekly rule across a daylight-saving transition, a monthly rule near the end of a short month, and a leap-day rule. Use a fixed test clock and compare the stored local intent with the derived UTC instant after each transition.
I have spent enough time on OTP and email delivery to distrust a green HTTP response by itself. A 2xx only proves that the bridge accepted a request; it does not prove that a message was delivered. Track scheduled, enqueued, acknowledged, delivered, and suppressed states, and keep reminder content out of logs because destinations and timezone choices are personal data.
How should latency, cost, and endpoint boundaries shape the choice?
| Decision | Latency-leaning option | Cost-leaning option |
|---|---|---|
| Dispatch | Poll frequently and reserve worker capacity | Batch due occurrences on a slower cadence |
| Retry | Short backoff with a dead-letter review path | Fewer attempts with explicit operator review |
| Calculation | Derive each next occurrence immediately | Derive it when a worker claims the schedule |
| Security | Signature, replay cache, and tenant limits | Signature with a smaller replay store |
The trade-off is concrete. A public HTTPS webhook is a good boundary when the ingress can be authenticated and monitored. It is not suitable for a private-only service unless a public gateway forwards into the private worker plane. A queue with at-least-once delivery is a poor fit for a side effect that cannot be made idempotent; use a workflow system or an application-level ledger when the process needs joins, long-running compensation, or an audit trail across steps.
Do not make a unit price the main decision. Measure queue wait, callback time, duplicate rate, and worker utilization with the same tenant mix you expect in production. I'm not sure any single scheduler's “timezone support” label tells you how it handles invalid monthly dates, pause recovery, or replay, so require those behaviors in an acceptance test.
A small rollout that keeps the calendar honest
Start with one tenant and a staging endpoint. Generate weekly and monthly occurrences around a daylight-saving change, a short month, and a leap day. Replay each signed callback and verify that the ledger creates one job for one occurrence.
Then cap concurrency below the provider's documented limit, watch queue age, and exercise the pause and resume policy. Roll out by tenant cohort. Keep the calendar policy version beside each schedule so a later policy change does not silently rewrite existing intent.
The durable design is deliberately plain: calendar data, an occurrence ledger, a narrow webhook, and a worker loop. Cron is only the trigger.
