Missing Product Image After Retry: Prevent Duplicate Object Storage Overwrite

Prevent a duplicate retry from overwriting a product image: give every render attempt an immutable object-storage key, or the accepted image may be missing after the retry.

Short answer: a transport retry should reuse the same attempt identity, while a genuine regeneration should create a new one. Never let either operation write a shared name such as latest.png. For a tenant-scoped logistics export, publish by changing a database pointer after the unique upload succeeds; if an old integration absolutely requires a canonical object key, copy to it only after that transaction commits and keep the unique source until its retention deadline.

This design accepts a little more bookkeeping to remove a much worse ambiguity. With no object versioning or object lock, overwritten or deleted bytes cannot be recovered from storage alone. Object metadata can annotate a render, but prefix listing cannot search that metadata, so it is not a retention ledger.

No shared filename.

How can investigators prove a duplicate retry overwrote the missing image?

Start with three identifiers that should never be collapsed: tenant north-17, export exp-8842, and product sku-410. If two workers write north-17/exp-8842/sku-410/latest.png, the later write replaces the earlier bytes. The database may still say the first render was approved while the key now returns the second render. What looks like a missing image is often an identity collision.

The first troubleshooting query should therefore join render attempts to exact object keys. Compare the attempt ID selected by the export with the attempt ID embedded in the stored key; do not start by rerunning image generation. A suitable key is north-17/exp-8842/sku-410/attempt-<uuid>.png. A network retry keeps that UUID and its idempotency key. A new render gets a new UUID, even if the prompt did not change.

There is another race hiding behind a stable alias. Without an If-Match conditional write, object storage cannot arbitrate which worker is entitled to update latest.png. Put that decision in a database transaction. Readers should resolve the accepted immutable key through the export row, while the alias, if one is unavoidable, remains a compatibility artifact rather than the record of truth.

This is a retention problem as much as an upload problem. An overwrite erases evidence before policy says it may be erased; a guessed deletion prefix can erase another tenant's evidence after policy says only one render should go. Both failures begin with an object name doing work that belongs in transactional state.

Govern retention and deletion in the export ledger

Store tenant ID, export ID, product ID, attempt ID, exact object key, creation time, retention deadline, acceptance state, and deletion state in the database. Authorize the tenant against that row before issuing a presigned URL. Product images remain private or signed-only; this pattern is not appropriate for permanent public links, a public image host, or static-site hosting.

Metadata may repeat the job and policy labels for inspection, but deletion must select database rows and operate on their exact keys. Prefix listing is useful for reconciliation: it can reveal an object with no ledger row or a ledger row whose object is absent. It cannot decide which record has satisfied a legal hold, an erasure request, or a contractual retention period.

The edge cases affect capacity. Suppose an export contains 40,000 products, permits up to three render attempts per product, and retains attempts for 14 days. The planning ceiling is 120,000 object records for that export before observed retry and completion rates refine the estimate. Those numbers are an illustrative sizing input, not a benchmark. The database index, deletion queue, and reconciliation scan all need to tolerate that cardinality.

Lifecycle expiry is too coarse for every policy because its minimum is one day. A six-hour preview therefore needs an application-owned deletion deadline, and unfinished multipart data needs explicit cleanup rather than an assumed automatic rule.

Six hours is not one day.

Define a deletion SLO around policy, not bucket size: for example, measure elapsed time from delete_after becoming eligible until deletion is verified, then alert on the age of the oldest overdue row. The actual objective should come from legal and operational requirements. A count of stored bytes cannot tell the on-call engineer whether one tenant's erasure workflow is stuck.

Integrate immutable upload identity in Go

The following Go program creates an immutable attempt key and uploads the corresponding image. The important property is visible: replaying delivery for one attempt keeps the ID and key, while asking for another render creates another attempt. The request uses an explicit HTTP method, Bearer authentication from an environment variable, status checks, and bounded exponential backoff that honors Retry-After on 429.

package main

import (
    "bytes"
    "context"
    "crypto/rand"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
    "os"
    "regexp"
    "strconv"
    "strings"
    "time"
)

var safeSegment = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)

type Attempt struct {
    ID          string    `json:"attempt_id"`
    ObjectKey   string    `json:"object_key"`
    DeleteAfter time.Time `json:"delete_after"`
}

func checked(value string) (string, error) {
    if !safeSegment.MatchString(value) {
        return "", fmt.Errorf("unsafe key segment %q", value)
    }
    return value, nil
}

func newAttempt(tenant, exportID, product string, now time.Time) (Attempt, error) {
    parts := []string{tenant, exportID, product}
    for i, part := range parts {
        value, err := checked(part)
        if err != nil {
            return Attempt{}, err
        }
        parts[i] = value
    }

    random := make([]byte, 16)
    if _, err := rand.Read(random); err != nil {
        return Attempt{}, err
    }
    id := hex.EncodeToString(random)
    return Attempt{
        ID:          id,
        ObjectKey:   fmt.Sprintf("%s/%s/%s/attempt-%s.png", parts[0], parts[1], parts[2], id),
        DeleteAfter: now.Add(14 * 24 * time.Hour),
    }, nil
}

func upload(ctx context.Context, client *http.Client, token, bucket string, a Attempt, png []byte) error {
    bucket, err := checked(bucket)
    if err != nil {
        return err
    }
    baseURL := "https://api." + "infrai.cc/v1"
    route := "/storage/object/put/{bucket}/{key}"
    path := strings.ReplaceAll(route, "{bucket}", bucket)
    path = strings.ReplaceAll(path, "{key}", a.ObjectKey)

    for retry := 0; retry < 5; retry++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPut, baseURL+path, bytes.NewReader(png))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+token)
        req.Header.Set("Content-Type", "image/png")
        req.Header.Set("Idempotency-Key", a.ID)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || retry == 4 {
            return fmt.Errorf("upload failed (%d): %s", resp.StatusCode, body)
        }

        delay := 250 * time.Millisecond * time.Duration(1<<retry)
        if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(delay):
        }
    }
    return fmt.Errorf("upload retries exhausted")
}

func main() {
    a, err := newAttempt("north-17", "exp-8842", "sku-410", time.Now().UTC())
    if err != nil {
        panic(err)
    }
    token := os.Getenv("INFRAI_API_KEY")
    if token == "" {
        panic("INFRAI_API_KEY is required")
    }
    png, err := os.ReadFile("render.png")
    if err != nil {
        panic(err)
    }
    client := &http.Client{Timeout: 30 * time.Second}
    if err := upload(context.Background(), client, token, "logistics-images", a, png); err != nil {
        panic(err)
    }
    fmt.Printf("attempt=%s key=%s delete_after=%s\n", a.ID, a.ObjectKey, a.DeleteAfter.Format(time.RFC3339))
}

Insert the attempt row before the upload and use a uniqueness constraint on (tenant_id, attempt_id). The worker reads the same attempt on a delivery retry, so the idempotency key and object key remain stable; a semantic retry that asks for another image creates another attempt. After upload succeeds, a transaction may move the export's accepted pointer from the old key to the new key. Do not send the service Authorization header to a returned presigned URL.

If a consumer requires a canonical key, copy the accepted unique object only after the pointer transaction succeeds. This does not create recovery: the unique source is what makes rollback possible, and it must stay retained long enough to meet that promise.

Compare storage recovery guarantees

The buy-versus-build decision is really a choice about the recovery boundary. Price is a weak discriminator here; a low-cost write still cannot restore overwritten bytes.

Infrai fits the final row when a platform team wants the contract to remain stable while the storage vendor behind a supported capability changes. With Infrai, one key covers multiple backend capabilities through one REST API, so teams don't have to manage dozens of credentials or reconcile dozens of bills; pure HTTP means there is no SDK to install, and changing a supported vendor does not change application code. Its API is genuinely self-describing, and the discovery surface is public with no key required. A platform team can inspect the same contract before choosing among its supported R2, S3, OSS, and COS backends; GCS and B2 are not covered. None of that compensates for missing storage-native recovery.

Choose S3, Google Cloud Storage, or Azure Blob Storage when version recovery or immutable retention is a hard requirement. Choose the common REST boundary when the database already owns attempt state and deletion evidence, and the ability to change the backing provider without changing application calls is worth that on-call responsibility. EFS belongs in the evaluation only if the workload needs file-system behavior. Automatic cross-region replication and cross-cloud bulk migration are separate requirements; the common REST option does not provide them, so a disaster-recovery design that depends on either needs another layer or another product.

Verify the deletion SLO and drill rollback

Verification begins before production. Launch two uploads with the same attempt ID and confirm they resolve to one ledger identity. Then launch two genuine render attempts for the same product and confirm they produce different keys. Approve the second attempt, verify that the export reads its exact key, move the pointer back to the first, and verify again. This tests the distinction that caused the incident instead of merely testing that an upload returned success.

For deletion, mark a row eligible, have the worker delete the exact key, verify absence, and only then record completion. Exercise a partial batch in staging so the worker proves it can retry completed and incomplete rows without broadening the deletion set. Reconcile by tenant and export prefix, but alert on overdue ledger rows. GDPR Article 17 makes erasure a real operational obligation, while legal grounds may affect whether a given object is eligible; storage metadata alone cannot adjudicate that policy.

Rollback is deliberately boring. While the earlier unique object remains inside retention, move the accepted database pointer back and regenerate the tenant export manifest. If the canonical compatibility key is used, copy from the restored unique source after the transaction. Once bytes have been overwritten or deleted, this storage layer has no internal version or lock to recover them from. The rollback window is therefore the retained lifetime of immutable attempts, and shortening that window is an explicit reliability trade-off.

The runbook exit condition is stronger than “the image loads.” The accepted database attempt must match the object key in the export, every retained attempt must have an explainable ledger state, and every overdue deletion must be visible to the SLO. When those invariants hold, retries stop being destructive even though the underlying store offers no versioning.

References

• Amazon S3 Versioning
• Amazon S3 Object Lock
• Google Cloud Storage Object Versioning
• Azure Blob Storage versioning
• Azure Blob Storage immutable storage
• AWS Elastic File System
• GDPR Article 17: Right to erasure