Duplicate Alert Control with Retry Idempotency for Pricing Rule Error Metrics in 2026

Short answer: prevent duplicate failure alerts by assigning one dedupe key to each error group or metric condition, recording a successful notification before a worker can retry it, and requiring consecutive healthy polls before resolving the incident. For a pricing-rule rollout, that state machine matters more than the webhook sender.

The page arrives at 02:14: "pricing evaluation failures above threshold." The on-call opens the channel and finds six messages for one condition. Three came from repeated polls; three came from delivery retries. The flag may still be serving the previous rule safely, but the alert stream no longer says how many incidents exist or which rollout cost center owns them. Acknowledging one message does nothing to the other five.

I've been paged by missed jobs and duplicate deliveries. The operational lesson is blunt: delivery is not incident identity.

What the on-call should see before taking action

The useful page represents one incident. It names the pricing rule, flag key, environment, error group or metric condition, first observation time, latest observation time, current count, rollout cohort, and cost-attribution label. It can include recent events as evidence, but those events belong inside the incident; they should not each create a notification.

Work backward from the action. The responder needs to decide whether to pause the pricing rollout, roll back its rule, or leave it running while investigating. A burst of webhook deliveries can't support that decision. A stable incident record can. In a self-built alerting path, fetch group detail or recent events to enrich one alert, then update the existing incident on later polls. For an error-group workflow, GET /v1/errors/group_detail/{error_group_id} is the useful detail boundary. For a metric condition, poll GET /v1/metrics/query, but don't invent server-side filters: its filtering parameters aren't declared in discovery.

This minimal Go client fetches the group detail used to enrich an incident. Set INFRAI_BASE_URL to the documented API base and keep it outside the source so the unlinked example doesn't embed a vendor URL. The retry path honors Retry-After on HTTP 429 and every request declares its method.

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    apiKey := os.Getenv("INFRAI_API_KEY")
    groupID := os.Getenv("ERROR_GROUP_ID")
    if baseURL == "" || apiKey == "" || groupID == "" {
        panic("set INFRAI_BASE_URL, INFRAI_API_KEY, and ERROR_GROUP_ID")
    }

    route := strings.Replace(
        "/v1/errors/group_detail/{error_group_id}",
        "{error_group_id}",
        url.PathEscape(groupID),
        1,
    )
    endpoint := baseURL + route
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, endpoint, nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("request returned %s: %s", resp.Status, body))
        }
        fmt.Println(string(body))
        return
    }
}

The signal that should have fired earlier is not "another event arrived." It is "the named condition moved from healthy to firing." That transition is where the dedupe key is created. A practical key is a canonical tuple such as environment, pricing-rule identifier, condition version, and error group. Hash the tuple if the notification store prefers fixed-width keys, but retain the original dimensions on the incident for debugging and cost attribution.

Keep the condition version in the key. If a threshold changes during rollout, the new rule should not silently inherit the old rule's cooldown. Conversely, don't include volatile fields such as poll time, event count, worker attempt, or webhook request ID. They guarantee a fresh key on every retry, which defeats deduplication.

One page. One owner.

How should a polling monitor dedupe duplicate alerts across webhook sender retries?

Use a small state machine around each condition: healthy, pending, firing, and recovering. The poller evaluates evidence; a durable incident store owns the state; the sender only delivers a transition that has already been assigned an idempotency key. On a worker retry, look up that key before contacting Slack, email, or SMS through your provider. If it was sent, stop. If it is leased by another worker, stop and retry later. If it is new, claim it atomically, send, and persist the provider result.

This ordering closes the obvious race, but there is still a narrow uncertainty window when the provider accepts a webhook and the worker dies before recording success. Use the same client-supplied key with a provider that supports idempotent sends. If the provider doesn't, exactly-once delivery isn't available; prefer a possible duplicate over silently suppressing a page, and make the incident identifier prominent so humans can collapse the duplicates quickly.

The following runnable Go program models the core transition. Polls 1 and 2 fail, a worker retries the firing transition, and polls 3 and 4 recover. Only the first failure and the confirmed recovery produce notifications. Replace the in-memory map with a transactional store before production use.

package main

import (
    "fmt"
    "sync"
)

type incident struct {
    Failures int
    Healthy  int
    Firing   bool
}

type monitor struct {
    mu   sync.Mutex
    sent map[string]bool
    byID map[string]incident
}

func (m *monitor) poll(id string, failed bool) {
    m.mu.Lock()
    defer m.mu.Unlock()

    state := m.byID[id]
    if failed {
        state.Failures++
        state.Healthy = 0
        if !state.Firing {
            state.Firing = true
            m.notifyOnce("firing:" + id)
        }
    } else {
        state.Failures = 0
        if state.Firing {
            state.Healthy++
            if state.Healthy >= 2 {
                state.Firing = false
                state.Healthy = 0
                m.notifyOnce("resolved:" + id)
            }
        }
    }
    m.byID[id] = state
}

func (m *monitor) notifyOnce(key string) {
    if m.sent[key] {
        return
    }
    m.sent[key] = true
    fmt.Println("send", key)
}

func main() {
    m := &monitor{sent: map[string]bool{}, byID: map[string]incident{}}
    condition := "prod:pricing-v3:evaluation-errors"

    m.poll(condition, true)
    m.notifyOnce("firing:" + condition) // Simulated worker retry.
    m.poll(condition, true)
    m.poll(condition, false)
    m.poll(condition, false)
}

Expected output is two lines: one firing notification and one resolution. The second healthy poll matters. Clearing on the first good sample turns ordinary variance into alert flapping, resets the incident too early, and makes the next bad sample look new. Your mileage may vary on whether two, three, or more healthy polls fit the service; decide from the poll interval and tolerated detection delay, then put that choice in the runbook.

Instrument the pricing-rule rollout for attribution

A flag answers who receives the new rule. Observability must answer what that exposure did and who owns its cost. Emit a stable flag key, rule version, environment, evaluation outcome, and cost-center label with the error or metric. Avoid raw customer identifiers in the dedupe key. They explode incident cardinality and can turn one broken rule into thousands of pages. Aggregate customers into the rollout cohort unless an individual account genuinely requires separate response.

For the initial page, include the first and latest timestamps plus a bounded set of recent evidence. Later polls should update the incident count and latest timestamp. They should not bypass the cooldown merely because the count changed. An escalation can be a separate transition, with a separate key, when the condition crosses a materially higher threshold or remains firing past a defined duration. Document both transitions so the person editing the pricing rule knows which page they can trigger.

There is a cost-accounting trap here. Counting notification attempts as incidents makes a retrying sender appear to create more failures, while counting only successful sends hides provider errors. Track three separate measures: condition transitions, delivery attempts, and accepted deliveries. Attribute the underlying condition to the pricing rule and cohort; attribute notification traffic to the alerting system. This preserves the question the rollout owner actually asks: did the new rule cause evaluation failures, or did the messenger merely retry?

Also instrument silence. A poller that stops running produces no error metric, so a downstream alert on evaluation failures cannot detect its own absence. Use an external heartbeat monitor for that path. Healthchecks is the cleaner fit for "the task should have run but didn't"; a self-built failure poller is not a substitute.

Which alerting approach fits the failure mode?

The decision is less about feature count than ownership. A managed monitor is appropriate when the team wants threshold evaluation, routing, escalation, and incident lifecycle in one product. A self-built poller is reasonable when the condition is narrow, its attribution model is application-specific, and the team is prepared to own durable state and notification delivery.

The catch is ownership. Infrai is not suitable when the team expects a turnkey paging product, distributed trace or span-tree queries, source-map decoding, crash symbolication, Session Replay, or synthetic and heartbeat monitoring. Stick with Datadog or Grafana Alerting when managed threshold and routing workflows are the requirement; use Sentry when error investigation is centered on its issue model; add Healthchecks when silence is itself the failure. The self-built route earns its place only when custom cost attribution and a small integration surface outweigh the on-call burden.

No vendor fixes a weak key. If the tuple doesn't identify the incident boundary, retries will expose that mistake under load.

Set thresholds by budgeting false positives

Cooldown and recovery thresholds trade notification volume for detection speed. A five-minute cooldown is not automatically safer than a one-minute cooldown, and a requirement for three healthy polls is meaningless without the poll interval. Express both in elapsed time in the runbook, even if the implementation counts polls. Then test these sequences before rollout: repeated failures, alternating healthy and failed samples, a worker retry after delivery, two workers racing on the same transition, a changed condition version, and a poller that stops entirely.

Start with the response decision. If one failed evaluation can charge or block a customer incorrectly, the firing threshold may be one sample, while duplicate suppression and rollback automation carry the noise control. If failures are retried safely inside the application, requiring consecutive bad polls may avoid paging on transient errors. I'm not sure which threshold is right without the pricing rule's failure semantics, retry window, poll interval, and rollback cost. Those four inputs should be explicit in the change review.

Getting the threshold wrong has a real false-positive cost: responders learn to skim the channel, duplicate pages obscure the first timestamp, and a noisy rollback can stop a healthy pricing rollout. Getting deduplication wrong is worse because it corrupts the evidence used to tune that threshold. Preserve one incident timeline, distinguish condition transitions from sender retries, and make recovery deliberate. Then the page can lead to an action instead of an inbox cleanup.

References

Further reading:

• Feature Toggles
• Datadog Monitors
• Grafana Alerting
• Sentry Alerts
• Healthchecks Documentation
• Console Do Not Track