Marketplace Delivery History Explained: Why Webhook Events Need Durable Records

Webhook delivery history exists to preserve marketplace events as durable records, not fleeting notifications. The operational rule is strict: acknowledge intake only after the event identity, receipt time, authenticated source, payload, and processing state cross a durability boundary. If that cannot happen within the latency objective, refuse the request so the sender can retry.

Short answer: webhook delivery history exists because a notification describes an attempt, while an event record supports recovery, deduplication, replay, audit, and a defensible answer to "what happened?" after an outage. The hard platform decision is how much durable intake capacity to fund before backpressure becomes refused traffic.

Do not start capacity planning with requests per second. Start with the longest outage the business expects intake to absorb, multiply it by the admitted event rate and conservative record size, then add retry headroom. Test that estimate under production durability settings.

Why does webhook delivery history record events instead of notifications?

A handler receives a request, runs code, and returns a status. During an outage, several different facts matter: whether the sender attempted delivery, whether the receiver authenticated it, whether the receiver durably accepted it, whether processing started, and whether the intended state transition completed. One success flag destroys distinctions operators need most.

Suppose a seller account update arrives while the downstream account service is unavailable. Intake can accept it only if bounded durable storage remains available. History should preserve the immutable received event and attach attempts separately. Reprocessing creates another attempt; it does not rewrite the original receipt.

Make the capacity argument concrete before choosing an implementation. A planning scenario might admit 2,000 events per second, use a deliberately conservative 4 KiB per stored envelope after accounting for indexes, and require 30 minutes of outage absorption. Those inputs imply 3.6 million envelopes and roughly 14 GiB before replication, retry amplification, attempt rows, or safety margin; they are arithmetic inputs, not measured system performance. Now run the uncomfortable variant: processing returns at only 1,500 events per second while arrivals remain at 2,000. Storage continues to fill after the outage ends. The team must either reserve enough capacity for that drain period, shed some seller tiers under a declared policy, or revise the recovery objective. This is the trade-off that a request-rate dashboard conceals. It also explains why a history feature cannot promise unlimited replay: retention consumes storage, replay competes with live processing, and payload minimization may remove fields that a future consumer wishes it had. A durable ledger is not suitable as an analytics warehouse or as an excuse to accept traffic beyond a tested ceiling. Its job is narrower: establish what the receiver accepted and provide controlled inputs for recovery.

Keep the contract small. Acceptance means "stored for processing," not "all business effects finished." Refusal means the receiver did not take responsibility. A lost response after commit may cause a retry, so stable event identity and atomic insertion matter.

Secrets do not belong in the ledger. Retain a credential identifier and verification outcome when investigation requires them, but keep signing material under controlled access, rotation, and audit. The OWASP Secrets Management Cheat Sheet covers those lifecycle controls.

Use an append-only envelope containing event ID, seller ID, source, receipt time, payload, content digest, and authentication result. Store attempts as children with attempt number, timestamps, worker version, outcome class, and a bounded diagnostic. A latest-state projection is useful, but it should remain rebuildable. Acceptance and execution are separate states.

package intake

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "time"
)

type Event struct {
    ID, SellerID, Source string
    ReceivedAt           time.Time
    Payload              []byte
    ContentDigest        string
    Authenticated        bool
}

type Ledger interface {
    // PutIfAbsent commits durably before returning nil.
    PutIfAbsent(context.Context, Event) (inserted bool, err error)
}

var ErrRefused = errors.New("event was not durably accepted")

func Accept(ctx context.Context, ledger Ledger, now time.Time, e Event) (bool, error) {
    if e.ID == "" || e.SellerID == "" || e.Source == "" || !e.Authenticated {
        return false, ErrRefused
    }
    sum := sha256.Sum256(e.Payload)
    e.ReceivedAt = now.UTC()
    e.Payload = append([]byte(nil), e.Payload...)
    e.ContentDigest = hex.EncodeToString(sum[:])
    inserted, err := ledger.PutIfAbsent(ctx, e)
    if err != nil {
        return false, ErrRefused
    }
    return inserted, nil
}

The interface leaves one serious requirement to its implementation: insertion must be atomic and durable. A memory map is neither after process loss. The boolean keeps duplicate delivery out of the error path; an existing event is known, while a storage error is refusal.

The digest supports comparison, but it cannot replace the sender's stable ID. Identical payloads may represent distinct events. If an ID reappears with a different digest, quarantine the conflict rather than replacing the first record.

Set the spend ceiling before the queue fills

The platform has three levers: reserve more capacity, reduce admitted rate, or accept a shorter survivable outage window. Put the choice in the service review.

There is no universal winner. Define admitted rate, maximum buffered age, maximum ledger bytes, and a refusal threshold below physical exhaustion for each service tier. Alert on distance to the threshold. Backlog can continue growing when recovery throughput is lower than arrival rate.

Separate SLOs are required. Measure durable acceptance over eligible authenticated requests, then measure latency from receipt to completed business effect. Otherwise a healthy intake tier can conceal stalled processing, or slow processing can make correct buffering look unavailable. One number cannot describe both promises.

Verify the outage path

Disable processing while leaving the real durability boundary active. Admit events at the planned rate through the target outage window, restart intake, and verify every acknowledged ID is queryable. Restore processing under a constrained drain rate while observing backlog age, duplicates, refusals, storage growth, and recovery time.

Break the drill immediately before commit, immediately after commit but before response, during credential rotation, and while appending an attempt. The post-commit response loss is revealing: a retry must find the existing record without creating a second business effect.

Use fixed reconciliation sets. Generate 10,003 unique IDs, resend 317 deliberately, then compare generated, acknowledged, stored, and completed ID sets. These are test inputs, not performance claims. Any acknowledged ID absent from storage violates the acceptance contract.

Short tests lie.

Run long enough to reach the configured capacity guardrail. Confirm refusal is explicit, observable, and leaves no partial record. Verify that authentication headers, signing secrets, and unbounded payload fragments do not appear in diagnostics.

Roll back workers without erasing evidence

Rollback consumers, routing rules, and projections, never accepted envelopes. Pause the faulty consumer, record the deployment boundary, restore the last known-good worker, and replay a bounded ID or receipt-time range through normal idempotency controls. Keep the failed attempt and append recovery.

History now earns its storage bill. Operators can separate events received during an outage from events processed by a faulty version, while support can answer a seller without reconstructing truth from application logs. Retention must follow legal and operational requirements; indefinite payload retention is not a default.

A rollback is complete when every accepted event reaches either a terminal processing state or an explicitly owned quarantine state, backlog age falls within the recovery objective, and refusal returns within its error budget. The ledger preserves facts; the runbook restores service.

References

• https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html