Node.js Platform Webhooks: Check Event Delivery Across Credential Rotation

TL;DR: During a production API-key rotation, freeze handler edits until you can correlate one expected webhook across the platform's delivery record and your Node.js intake record. Label both sides with an event ID, an attempt ID when one exists, the endpoint identity, and a credential epoch such as old, overlap, or new. That evidence tells you which boundary lost the event. It also keeps one credential's blast radius visible instead of turning a rotation into guesswork.

The table is the decision rule. A platform-side attempt is evidence that dispatch began; an intake record is evidence that your edge received something. Neither proves that the business action completed. Three boundaries, three different questions.

This matters in B2B SaaS because key rotation and webhook triage can collide. The tempting move is to change signature code after an expected account event appears to vanish. Resist it. First establish whether the producer tried the endpoint that your current deployment actually serves.

What should I check if platform webhook events have never arrived?

Picture the path in words: account change -> subscription filter -> delivery attempt -> network edge -> signature check -> durable queue -> worker -> business state. Put an observation point between each arrow that crosses an ownership boundary. You do not need maximal logging. You need enough shared fields to join adjacent records.

Start with a single expected event and a bounded time window. Record its stable event identifier if the platform provides one. Then compare the destination recorded for the attempt with the endpoint identity in deployment configuration. Do not paste a secret, signature, or raw authorization header into a ticket or log.

The absence itself is useful. No producer attempt points upstream of your Node.js process, so changing the handler cannot explain it. A failed attempt plus a matching intake record narrows the search to request validation or response behavior. A producer success plus an intake record moves the investigation past HTTP and toward queueing, deduplication, or worker state.

Stop there.

Consider a hypothetical rotation timeline. At 14:00, the fleet reports that every instance accepts the old and new slots. At 14:05, the producer switches to new. An expected account event carries ID evt-2841, and the delivery record shows an attempt at 14:06 to endpoint billing-prod. Your intake has no received record for that event ID between 14:05 and 14:10. Those facts do not justify changing signature verification: there is no evidence that the verifier ran. Compare the recorded destination with the deployed route and edge records. Now change one fact. If the intake has received, credentialEpoch: "new", and signatureValid: false, the request crossed the edge and the verification boundary becomes the narrow search area. Change it again: signatureValid: true exists but enqueued does not. The queue handoff is now the first missing boundary. The same visible symptom, one absent business update, produces three different owners and three different investigations. This is why correlation beats a speculative handler patch.

Keep the language exact. “Delivered” may mean that an HTTP exchange received an acceptable response; it does not automatically mean that an invoice, tenant, or entitlement changed. Your own telemetry should name those later transitions separately.

Pick this when the producer has no attempt

Use producer-side delivery records when the expected event has no intake counterpart. Check the event type, subscription scope, destination identity, and observation window. During rotation, also check whether the account or environment used to trigger the event is the one attached to the inspected endpoint.

This is a control-plane investigation. The Node.js handler has not earned suspicion yet.

No code change.

Avoid treating a replay button as diagnosis. A replay can create a new attempt, but it does not explain why the original event was absent. Capture the original evidence first: expected event, account, endpoint identity, and time bounds. Then replay only if your operational policy permits it and the consumer is idempotent.

Pick this when an attempt reached the intake

Once both records exist, compare correlation fields before reading application code. The sharpest discriminator during rotation is the credential epoch. It is a label for configuration state, not the credential value.

The trade-off is deliberate: log a small join key set and explicit stage names, but never secret material. OWASP's secrets guidance recommends limiting a secret's scope, rotating it, and ensuring logging does not expose it. An epoch label preserves operational context without widening exposure.

Instrument one Node.js intake deeply

The intake should acknowledge only after its chosen durability boundary. In this example, that boundary is enqueue. The code keeps transport receipt, authentication, and handoff as separate observations. It accepts both credential slots during the planned overlap, while the verifier owns the exact signature scheme required by the webhook contract.

type CredentialEpoch = "old" | "overlap" | "new";

type IntakeContext = {
  eventId: string;
  eventType: string;
  endpointId: string;
  credentialEpoch: CredentialEpoch;
};

interface SignatureVerifier {
  verify(input: {
    rawBody: Buffer;
    signature: string;
  }): Promise<{ valid: boolean; credentialEpoch: CredentialEpoch }>;
}

interface DurableQueue {
  enqueue(input: { id: string; body: Buffer }): Promise<{ jobId: string }>;
}

interface AuditSink {
  write(record: Record<string, string | number | boolean>): void;
}

export function createWebhookHandler(
  verifier: SignatureVerifier,
  queue: DurableQueue,
  audit: AuditSink,
  endpointId: string,
) {
  return async function handle(req: {
    body: Buffer;
    headers: Record<string, string | undefined>;
  }): Promise<{ status: number }> {
    const eventId = req.headers["x-event-id"] ?? "missing";
    const eventType = req.headers["x-event-type"] ?? "unknown";
    const signature = req.headers["x-signature"] ?? "";
    const receivedAt = Date.now();

    const verification = await verifier.verify({
      rawBody: req.body,
      signature,
    });

    const context: IntakeContext = {
      eventId,
      eventType,
      endpointId,
      credentialEpoch: verification.credentialEpoch,
    };

    audit.write({
      stage: "received",
      ...context,
      signatureValid: verification.valid,
      receivedAt,
    });

    if (!verification.valid) {
      return { status: 401 };
    }

    const { jobId } = await queue.enqueue({ id: eventId, body: req.body });

    audit.write({
      stage: "enqueued",
      ...context,
      jobId,
      enqueueLatencyMs: Date.now() - receivedAt,
    });

    return { status: 202 };
  };
}

There is an intentional gap here: the verifier and queue are interfaces. Signature construction, timestamp tolerance, header names, and response requirements belong to the actual webhook contract. Guessing those details would produce code that looks complete and fails at the boundary.

The eventId also deserves a real policy. If the producer guarantees a stable identifier, use it as the queue deduplication key and retain a terminal processing record. If it does not, define an application key from documented stable fields. Never silently substitute the whole payload or a secret-bearing header as an identifier.

During the overlap, graph counts by credentialEpoch and stage. A healthy transition becomes visible as accepted traffic shifts from old toward new. Alert on rejected verification and on a sustained gap between received and enqueued; those signals describe different failures and route to different owners.

Then test the handoff. Send fixtures signed through each configured slot, assert that invalid input never reaches the queue, and assert that a repeated stable event ID does not repeat the business mutation. The tests should inspect audit fields too. Observability is part of the contract here, not decoration.

Rotate without hiding the blast radius

A practical rotation has four states: provision the new credential, enter a bounded overlap, move producers to the new credential, then revoke the old one after evidence shows the transition is complete. Scope each credential to the smallest required system and permissions. That containment matters more than an elaborate dashboard.

Do not log either credential to prove which one matched. Assign the slots fixed labels at configuration load and emit only the label. Access to secret storage and rotation actions should be auditable, while application logs should remain free of secret values.

One subtle trap is deploy ordering. If an instance can receive traffic before it has both allowed verification slots during the overlap, the fleet can alternate between acceptance and rejection. Treat the accepted slot set as deployment configuration, expose its non-secret epoch labels in health metadata, and verify fleet convergence before switching the producer.

This is where a crisp before-and-after view helps. Before the switch, you expect old acceptance. During overlap, both labels may appear. After the switch, new attempts should correlate with new; only then does revocation become an evidence-backed action.

Limits of this field guide

Delivery records can localize a missing event, but they cannot prove downstream business correctness. Intake logs can prove receipt, but incomplete or overly broad logging can create a secrets incident of its own. Keep retention, access control, redaction, and audit requirements aligned with your organization's threat model.

The stopping rule is short: find the last confirmed boundary, name the first missing boundary, and debug only the component between them. During credential rotation, include the epoch in that statement. “Attempt recorded under new; receipt absent at endpoint billing-prod” is actionable. “Webhooks are broken” is not.

References

• OWASP, “Secrets Management Cheat Sheet”: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html