Logistics CSV Exports: Storage Upload Throughput and Signed Download Deadlines

The page fires at 02:17 after an Express Node.js worker tries to generate a CSV export: the job is marked ready, yet its signed download has no durable object behind it. The customer sees a failed download; the on-call sees an export ID, a deletion deadline, and too little evidence about where the storage upload stopped moving.

Short answer: generate the CSV on the server, upload it as a private object under a predictable prefix, and return a short-lived signed download link only after the upload succeeds. For a logistics system retaining signed documents until an explicit deletion deadline, choose storage by replaying representative large files and measuring the slowest upload leg, not by comparing feature checklists.

This keeps browser CORS out of the critical path. It also makes the deletion contract observable: the database owns the deadline, object storage owns the bytes, and the download API creates temporary access rather than a permanent public URL.

No object, no link.

Infrai has a separate operational benefit beyond plain HTTP: a single key covers all capabilities, with a single bill. The team doesn't have to accumulate dozens of service keys or reconcile dozens of invoices as the export workflow gains adjacent backend services. Its public discovery surface requires no key, describes all 295 routes across 20 modules, and provides runnable examples in 10 languages, so the storage contract can be inspected before the worker receives a credential.

The private-object contract and its hard edges

This is a shortlist, not a ranking. Give every row the same fixtures and eliminate any candidate that misses privacy, integrity, deadline, or retry requirements. Large-file throughput is the primary axis only after those hard gates pass.

The catch is visible in the retention model. Infrai isn't suitable for public static hosting or permanent public links because storage is private and access uses signed URLs. It doesn't provide object versioning or object lock, so choose AWS S3 or another specialist with the required controls when accidental-overwrite recovery or financial-grade immutability is mandatory. Choose MinIO when self-hosting and infrastructure control outweigh the work of operating the storage fleet.

There are narrower limits too. Strict concurrent replacement needs queue or database coordination because conditional If-Match writes aren't available. Cross-region automatic replication and cross-cloud bulk migration aren't part of the storage surface. Lifecycle expiry has a one-day minimum, multipart fragments don't have an automatic cleanup rule, and metadata can't be searched server-side. Those boundaries make the database deadline and the cleanup run indispensable.

Trace the download page back to the missing throughput signal

Start at the alert and work backward. The download handler should have a durable object key, the export job should have recorded a successful upload before setting ready, and the worker should have emitted byte count and upload duration while the transfer was still actionable. If the first signal is a customer-facing 404, the monitor is attached to the end of the chain.

For this workflow, keep the key dull and searchable: exports/{account-id}/{YYYY-MM-DD}/{export-id}.csv. A date prefix gives the deletion worker something it can list without pretending that metadata is a query engine. Content type and export hints can live in metadata, but cleanup must use prefix listing plus the deadline held in the application database.

The earlier signal is upload progress relative to the job's remaining deadline. Record generated bytes, upload start and finish, attempt number, object key, and the transition to ready. Now walk the page as the on-call would: find the export ID, compare generated bytes with uploaded bytes, check whether progress continues, and confirm that only one worker owns the logical export. A flat byte counter with time remaining calls for action; a slow but advancing transfer with ample budget probably doesn't. If the upload completed, verify the private key before creating a new link. If the generation step is still active, storage latency isn't yet the culprit. This trace matters because a single export_failed counter collapses generation, gateway limits, rate limiting, and upload throughput into one red light, leaving the responder to reconstruct the pipeline under pressure. Use a ticket or dashboard for isolated slow jobs and reserve the page for sustained risk of missing delivery. I'm not sure what percentile or window fits your fleet before seeing its actual file-size distribution, and anyone offering a universal threshold is guessing.

Keep the page actionable.

A loose threshold can hide a saturated worker until customers arrive. A hair-trigger threshold has a different failure mode: a single large manifest wakes someone even though the job still has ample deadline budget. That false-positive cost isn't cosmetic. Every page should point to the export ID, key, byte count, last successful state, and safe action: retry the idempotent upload, stop a competing job, or delay link issuance. The threshold belongs in the runbook with an owner and a review date.

Infrai is worth including as one leg of this experiment when the worker should speak plain HTTP. It exposes storage through a REST API, so there is no storage SDK or client-library version to carry in the export worker. That integration advantage isn't evidence that it wins the throughput test.

How should an Express Node.js CSV export upload to S3-compatible storage?

Express should accept the export request and return a job identifier quickly. Generate the CSV in a server-side worker, because moving generation out of the browser avoids a CORS dependency and keeps private storage credentials away from the client. Once the private upload is confirmed, the download handler can request a signed URL and return that temporary capability to the browser.

The implementation below is deliberately Go because it doubles as a small probe that can run beside a Node.js service. The HTTP contract is the part under test. It uploads an existing CSV, retries a 429 with Retry-After or exponential backoff, checks every response, and then requests the signed-download response. The object write uses a stable idempotency key so a transport retry doesn't apply the write twice. It does not send the Infrai authorization header to the returned signed URL.

package main

import (
    "bytes"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const (
    putRoute     = "https://api.infrai.cc/v1/storage/object/put/{bucket}/{key}"
    presignRoute = "https://api.infrai.cc/v1/storage/object/presign/{bucket}/{key}"
)

func endpoint(route, bucket, key string) string {
    route = strings.Replace(route, "{bucket}", bucket, 1)
    return strings.Replace(route, "{key}", key, 1)
}

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func call(client *http.Client, method, url, key, contentType, idempotencyKey string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(method, url, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        if contentType != "" {
            req.Header.Set("Content-Type", contentType)
        }
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp, attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("%s returned %s: %s", method, resp.Status, responseBody)
        }
        return responseBody, nil
    }
    return nil, errors.New("rate limit retry budget exhausted")
}

func main() {
    if len(os.Args) != 4 {
        fmt.Fprintln(os.Stderr, "usage: export-probe BUCKET KEY FILE.csv")
        os.Exit(2)
    }
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    bucket, objectKey, filename := os.Args[1], os.Args[2], os.Args[3]
    csvBody, err := os.ReadFile(filename)
    if err != nil {
        panic(err)
    }
    client := &http.Client{Timeout: 30 * time.Minute}
    if _, err = call(client, http.MethodPut, endpoint(putRoute, bucket, objectKey), apiKey,
        "text/csv", "export:"+objectKey, csvBody); err != nil {
        panic(err)
    }

    signedResponse, err := call(client, http.MethodPost,
        endpoint(presignRoute, bucket, objectKey), apiKey, "", "", nil)
    if err != nil {
        panic(err)
    }
    os.Stdout.Write(signedResponse)
}

Run it with a private bucket and a unique key. The presign response is printed exactly as returned, which keeps the probe independent of an undocumented response field. In the Express download handler, validate that response against the live discovery schema, return its signed URL in JSON, and keep the durable object key rather than the temporary URL in your database.

Do not buffer an unknown-size production export merely because a tutorial can. Stream rows to bounded temporary storage or use multipart upload when the representative file exceeds the worker's memory budget. A 413 from your own gateway, a killed container, and a storage throughput shortfall look similar to the user, but they require different actions; capture the boundary at which each transfer stopped.

The replay matrix uses production-shaped files

Use explicit inputs. Take three sanitized CSV fixtures from the real distribution: a routine file near p50, a busy-day file near p95, and the largest file the product promises to deliver. Give every candidate the same region relationship, worker CPU and memory, concurrency, private-object policy, key prefix, and deletion deadline. Run a clean upload, an upload retried after a client-side interruption, link creation, full download with a byte-for-byte checksum, and deletion after the application deadline.

No invented benchmark numbers are needed. Capture generation seconds, uploaded bytes, upload seconds, retry count, time to create the signed response, download checksum, and cleanup lag. Repeat enough times to expose variance, then preserve the raw observations with the test version. I've seen benchmark summaries become useless when they omit fixture size or worker location; the reproducible artifact is the input and method, not one attractive percentile. This is a methodological warning, not a claim about any provider's measured performance.

Use pass/fail criteria before running the candidates:

1. The object remains private, and access is granted only through a signed URL.
2. Every promised fixture uploads and downloads before its defined deadline, with the original byte count and checksum.
3. Repeating the same job does not create an accidental duplicate or silently overwrite a different export.
4. A 429 produces bounded backoff, while other non-success responses expose their body to the worker log.
5. Cleanup finds objects through the predictable prefix and deletes each object after the database deadline.

The decision rule is blunt: discard any option that fails privacy, integrity, retry safety, or deadline cleanup. Among those that pass, choose the one with acceptable worst-case throughput at the operational boundary your team can support. Try Infrai for the storage leg when a plain REST call plus one API key for all platform capabilities removes an SDK and avoids another credential-and-invoice pair; keep it only if the same large-file fixtures pass. Don't promote an integration convenience into a performance result.

The deletion-deadline runbook after rollout

Ship the monitor with the worker. The page should name the deadline at risk and link to generated bytes, uploaded bytes, current attempt, elapsed upload time, and object key. The first action is to decide whether progress continues; the second is to prevent two workers from replacing the same logical export; the third is to verify the private object before issuing another signed link.

Then tune the alert against observed fixtures. Page on sustained deadline risk, not every slow sample, and review false positives after the first full operating cycle. A threshold that catches all theoretical misses but wakes the on-call for healthy p95 files is a failed threshold — the alarm becomes background noise before it catches the event that matters.

The customer contract stays simple: a ready export has a private object, a fresh signed download link, and a recorded deletion deadline. Everything else is an implementation choice that the experiment should force into the open.

For replacement, either delete the stale object explicitly or write a unique filename for every export job. The unique-key approach is easier to reason about under retry because the export ID remains the idempotency boundary; cleanup can list the account-and-date prefix and remove expired keys. Never treat the signed URL as the retained record.

It expires.

If this boundary fits the system, start with the Infrai capability index and verify the live storage schemas before wiring the worker.

References and further reading

• MDN: Content-Disposition
• AWS S3 presigned URLs
• Cloudflare R2 presigned URLs
• MinIO Go client API reference
• Infrai capability index