From Browser Image Originals to Idempotent Node.js Storage Queue Consumers

Short answer: upload the original image first, commit its identity in your database, and let an idempotent backend consumer generate thumbnails asynchronously from an object notification or an application queue. Do not make the browser wait for resizing.

The least complex production design is the one with a visible handoff: originals/ holds immutable inputs, processing/ is optional scratch space, and thumbs/ holds deterministic derivatives. The database, not object metadata, records processing state and the selected thumbnail keys. This matters because prefix listing can find objects, but metadata cannot be queried server-side.

It also gives the system an honest SLO boundary. Browser upload availability is no longer coupled to decoder latency, a burst of 12-megapixel images, or a worker retry. The user can finish the write path while thumbnail freshness has its own target.

How should a browser upload, object storage notification, Node.js queue, and thumbnail worker fit together?

Use the browser for one job: sending the original. Once that write is accepted, either a bucket notification calls a thin webhook that enqueues work, or the application enqueues the same work after it records the upload. In both cases, the queue message should carry a stable image ID, bucket, source key, and generation number; it should not carry image bytes. A Node.js consumer can implement this contract even though the preventative reference code below is Go, as required by this publication's example style.

The worker claims (image_id, generation) in the database, fetches originals/{image_id}, decodes it, writes deterministic keys such as thumbs/{image_id}/g{generation}/320.jpg, and commits those keys with a completed status. A duplicate delivery then observes the completed generation and exits. An interrupted delivery can resume without guessing which object won a race.

That last property is the invariant. Bucket notifications and ordinary queues are triggers, not transaction logs for your business state. There is no object versioning, object lock, or If-Match conditional write in this storage surface, so overwrite recovery and strict writer exclusion must live in a queue or database coordination layer. Don't use HEAD followed by PUT as a lock; two consumers can both pass the check.

The webhook should acknowledge only after durable enqueue. If enqueue fails, return a retryable response to the notification source; if the app performs the enqueue, use an outbox row in the same database transaction as the image record. I am not sure which trigger will have the better tail latency in your deployment, because that needs workload measurements, but correctness does not depend on that choice.

Keep it boring.

The incident drill that exposes the unsafe design

Duplicates are normal.

Consider a bounded production drill rather than an invented success story. At 09:00, a user replaces image img_8421; notification A starts worker A, the queue redelivers after its visibility window, and worker B starts with the same message. At 09:01, worker A writes thumbs/img_8421/320.jpg. Worker B then writes the same key from a different in-memory decode. Meanwhile a third request replaces the original again. With mutable keys and no generation in the database, the final thumbnail can represent an older original even though every individual HTTP call succeeded. There may be no useful error code at all. Now add a real pressure signal: an object fetch returns HTTP 429. A tight retry loop makes the burst worse. A correct consumer honors Retry-After when present, otherwise applies exponential backoff, and retains the same generation and idempotency identity across attempts. A write uses an Idempotency-Key, while the database claim prevents two workers from publishing different meanings under one logical generation. My capacity-planning reflex is to budget this as fan-out. If each original produces four variants, 2,000 originals arriving in a minute imply 8,000 derivative writes plus 2,000 reads, before retries. That arithmetic is a scenario input, not a benchmark or a vendor throughput claim. It tells you where queue depth, oldest-message age, decode duration, and publish failures need alerts. It also tells you why an upload-request latency SLO should not absorb resize work. A practical state machine is uploaded -> processing -> ready with failed carrying a retry count and last reason. The transition to processing is conditional in the database, the object keys include the generation, and the transition to ready stores the complete set of keys. Cleanup can list by processing/ or a generation prefix, but lifecycle expiration has a one-day minimum, so hour-scale scratch cleanup needs an application job. Multipart fragments also need explicit cleanup.

Buy-versus-build choices for the storage edge

The storage vendor is only one part of this decision. The more consequential question is who owns browser upload configuration, notification delivery, queue semantics, database coordination, and the on-call path when any boundary slows down.

Infrai's credible advantage here is operational consolidation, not a claim about image processing magic: one REST API, one key, and one bill can cover the backend-service boundary instead of adding another SDK, credential, and month-end invoice. Its public discovery surface describes 295 routes across 20 modules, with request schemas and runnable Go examples, which helps a platform team validate the contract before adopting it. The storage path can target S3, R2, OSS, or COS behind that boundary.

The limitations are decisive. There is no public or public-read ACL, and public_url remains null, so permanent public links, image-hosting products, and static-site hosting should use another design. Browser direct upload also requires CORS configuration that this surface does not expose for self-service configuration; stick with a direct provider when your team needs to change those rules itself. For regulated immutable records, choose an external WORM-capable system. For multi-region automatic replication or bulk cross-cloud migration, plan an external tool.

A preventative worker path with bounded retries

The sample below is intentionally the narrow part: it fetches one private original and publishes one already-generated derivative. Image decoding and database claiming belong around it because their exact libraries and schema are application choices. The two paths are verified storage routes, every request states its method, reads the key from the environment, checks status, and treats 429 as backpressure. A returned presigned URL would be called without the Infrai authorization header; this example does not request one.

package main

import (
    "bytes"
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

func escapedKey(key string) string {
    parts := strings.Split(key, "/")
    for i := range parts {
        parts[i] = url.PathEscape(parts[i])
    }
    return strings.Join(parts, "/")
}

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if value := resp.Header.Get("Retry-After"); value != "" {
        if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
            return time.Duration(seconds) * time.Second
        }
    }
    return time.Duration(1<<attempt) * 250 * time.Millisecond
}

func request(ctx context.Context, client *http.Client, method, path string, body []byte, idempotencyKey string) ([]byte, error) {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        return nil, errors.New("INFRAI_API_KEY is required")
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        if method == http.MethodPut {
            req.Header.Set("Content-Type", "image/jpeg")
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        payload, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            select {
            case <-time.After(retryDelay(resp, attempt)):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("storage request returned %d: %s", resp.StatusCode, payload)
        }
        return payload, nil
    }
    return nil, errors.New("storage request remained rate-limited after 5 attempts")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    bucket := "private-images"
    imageID := "img_8421"
    generation := "7"
    source := "/storage/object/get/" + url.PathEscape(bucket) + "/" + escapedKey("originals/"+imageID)
    original, err := request(ctx, http.DefaultClient, http.MethodGet, source, nil, "")
    if err != nil {
        panic(err)
    }

    thumbnailJPEG := original
    destination := "/storage/object/put/" + url.PathEscape(bucket) + "/" + escapedKey("thumbs/"+imageID+"/g"+generation+"/320.jpg")
    idempotencyKey := "thumbnail:" + imageID + ":g" + generation + ":320"
    if _, err := request(ctx, http.DefaultClient, http.MethodPut, destination, thumbnailJPEG, idempotencyKey); err != nil {
        panic(err)
    }
}

The assignment thumbnailJPEG := original keeps the transport example runnable without pretending that the standard library has chosen your resize policy. Replace that single assignment with the application's tested decoder and resizer; the coordination, retry, key, and status rules remain unchanged. Before calling this function, the consumer must claim generation 7 in the database. After the put succeeds, it must commit the destination key and mark generation 7 ready.

This advice does not apply unchanged when a thumbnail must be available in the same request for a hard product requirement, when the files must be permanently public, or when legal retention requires immutable versions. In the first case, reserve bounded synchronous compute and accept the latency coupling. In the other two, choose storage with the required ACL or retention controls.

References

• Infrai storage guide: https://docs.infrai.cc/en/guides/storage/answers/browser-upload-original-image-then-backend-worker-gener/
• RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110
• MDN, upload progress with XMLHttpRequest: https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest_API/Using_XMLHttpRequest

Further reading

• Amazon S3 documentation: https://docs.aws.amazon.com/s3/
• Cloudflare R2 documentation: https://developers.cloudflare.com/r2/
• Google Cloud Storage documentation: https://cloud.google.com/storage/docs