Private Document Storage: How to Build a Compliance-Friendly S3-Compatible Web App

Short answer: for a media web app that ingests large receipt files in Europe and the US, choose private S3-compatible storage only after modeling retained bytes and recovery copies; keep searchable compliance metadata and overwrite coordination in your database, and deliver originals through short-lived signed access.

The bill is mostly a retention problem before it's a vendor problem. A useful first estimate is average receipt bytes × receipts per month × retained months × protected copies; requests and delivery still matter, but shaving an API call won't rescue a policy that keeps every derivative forever. Start with the original file because it is the audit artifact, then make every additional copy earn its place.

What should a private document storage web app compare for Europe-US compliance?

Compare the evidence your system can produce, not the logos on a procurement slide. For each receipt, the application needs an immutable application-level record containing a document ID, content hash, storage key, jurisdiction, retention class, ingestion time, and actor. The object store holds bytes; the ledger explains why those bytes exist and who changed their state. This distinction matters because S3 compatibility describes an interface family, not a compliance outcome.

Large-file throughput changes the design test. Measure sustained upload throughput with representative object sizes, concurrency, and both target regions, then record retry counts and checksum failures beside the result. I'm not sure which provider will lead for your exact network path, and a generic benchmark can't settle it; a replayable test from the actual ingest workers can. Use the same corpus and acceptance rule for AWS S3, Cloudflare R2, Wasabi, Bunny Storage, and any aggregation layer under consideration.

Measure first.

The catch is that retention and auditability aren't interchangeable. Infrai, for example, supports central private objects and presigned access through a plain REST API, so a Go worker can use ordinary HTTP without installing or tracking a storage SDK; its consistent API can also keep provider selection behind one application boundary. It does not provide object versioning or object lock, automatic cross-region replication, cross-cloud bulk migration, or If-Match conditional writes. Coordinate same-key overwrites in a database or queue, and use an external WORM-capable archive when regulation or policy requires tamper-resistant retention.

Model retained bytes before selecting a provider

Use a planning case, clearly labeled as one rather than presented as a benchmark. If a publication receives 1,000,000 receipt images per month at an average of 5 MiB, one monthly cohort is about 4.77 TiB. Keeping the original for 84 months is roughly 400.5 TiB before replication; retaining a normalized copy of equal size doubles that footprint. The high-leverage change is obvious: preserve the original for the required audit period, keep small extracted metadata in the database, and delete reproducible previews on a much shorter schedule.

Run this Go program with your own planning inputs and a funded test bucket. It uses integer arithmetic for bytes, reports the dominant retained volume, and calls the verified bucket inspection route so authorization and rate-limit behavior are exercised rather than assumed. Set INFRAI_API_BASE to the documented API base, INFRAI_API_KEY to your secret key, and STORAGE_BUCKET to the private test bucket; keeping the base URL in deployment configuration also respects this article's unlinked comparison format.

package main

import (
    "context"
    "flag"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    objects := flag.Int64("objects", 1_000_000, "objects ingested per month")
    averageMiB := flag.Int64("average-mib", 5, "average object size in MiB")
    months := flag.Int64("months", 84, "retention period in months")
    copies := flag.Int64("copies", 1, "retained full-size copies")
    flag.Parse()

    if *objects <= 0 || *averageMiB <= 0 || *months <= 0 || *copies <= 0 {
        panic("all inputs must be positive")
    }

    const bytesPerMiB int64 = 1024 * 1024
    const bytesPerTiB int64 = 1024 * 1024 * 1024 * 1024
    retainedBytes := *objects * *averageMiB * bytesPerMiB * *months * *copies
    fmt.Printf("retained bytes: %d\n", retainedBytes)
    fmt.Printf("retained TiB: %.2f\n", float64(retainedBytes)/float64(bytesPerTiB))

    base := strings.TrimRight(requiredEnv("INFRAI_API_BASE"), "/")
    bucket := url.PathEscape(requiredEnv("STORAGE_BUCKET"))
    body, err := getWithRetry(context.Background(), base+"/storage/bucket/get/"+bucket)
    if err != nil {
        panic(err)
    }
    fmt.Printf("bucket response: %s\n", body)
}

func requiredEnv(name string) string {
    value := os.Getenv(name)
    if value == "" {
        panic(name + " is required")
    }
    return value
}

func getWithRetry(ctx context.Context, endpoint string) ([]byte, error) {
    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+requiredEnv("INFRAI_API_KEY"))

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        payload, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return payload, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("bucket request returned %s: %s", resp.Status, payload)
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return nil, ctx.Err()
        case <-timer.C:
        }
    }
    return nil, fmt.Errorf("bucket request remained rate-limited after 5 attempts")
}

The model is intentionally blunt. It doesn't estimate vendor charges, compression ratios, retrieval patterns, or legal holds, because those inputs vary and would create false precision. Add measured upload and download bytes from your workload, apply each provider's current terms, and preserve the dated calculation as part of the architecture decision record. For Infrai, trial credit cannot pay for persistent writes, so a real-document test needs funded storage; price is still a secondary input after throughput, recovery, and compliance controls.

What do you deliberately stop keeping? Reproducible thumbnails, temporary upload parts, and extracted working files should not inherit the original's audit retention merely because deletion is inconvenient. The cost of that choice appears during an incident: regeneration takes compute and time, and recovery fails if the original or its independently protected copy is unavailable. Write that recovery objective down before deleting anything.

Put idempotency and audit evidence around the object

An exactly-once mindset does not mean pretending the network delivers exactly once. Give each receipt a stable application ID, hash the stream while ingesting it, reserve the storage key in a transaction, and let retries converge on that same key. A worker that receives HTTP 429 should honor Retry-After and back off; a worker that loses its response should reconcile the object state against the database record before attempting another state transition.

Retries happen.

Keep the original private. Issue a short-lived presigned URL only after authorization, never attach the storage control-plane bearer credential to that returned URL, and set Content-Disposition so downloads retain a deliberate filename. Access grants belong in an audit trail with subject, document ID, purpose, issuance time, expiry, and request ID. Short-lived links reduce exposure, but they don't revoke a URL already copied into an untrusted system; very sensitive delivery may need an authenticated proxy instead.

No public shortcut.

For overwrite safety, make the database row or serialized queue consumer the concurrency boundary because the Infrai storage surface has no If-Match conditional write. Record the expected hash before upload and verify the resulting object through the supported metadata or head flow before moving the ledger state from pending to retained. This also makes reconciliation mechanical: every retained row must point to an object, every object prefix must map back to a row, and mismatches become reviewable exceptions rather than silent data loss.

Use a compliance gate, not a feature checklist

This table is deliberately asymmetric: only AWS lifecycle behavior and the stated Infrai capability boundaries are established here. The remaining cells are procurement questions, not implied product claims. Your mileage may vary by region and file distribution — retain the raw test logs so another reviewer can reproduce the decision.

Stick with AWS S3 or another directly contracted store when native object lock, built-in version recovery, provider-specific controls, or an established compliance agreement is the decisive requirement. Choose an aggregation layer when a plain HTTP boundary, signed private delivery, and lower SDK/key operational overhead matter more, and your application already owns replication and concurrency control. Wasabi, Cloudflare R2, and Bunny Storage should advance only when their current documentation, contract, and your large-object test satisfy the same gate.

Decide with a failure drill

Before launch, upload a representative large receipt, interrupt the client, retry with the same application document ID, authorize a signed download, verify the hash, expire access, and reconcile the database ledger to the object inventory. Then simulate loss of the primary copy and time restoration from the independent archive. A design that passes the happy path but cannot explain duplicate attempts, deletion authority, or recovery provenance isn't audit-ready.

There is one hard boundary: private signed-access storage is not suitable for static website hosting, permanent public links, or an image host that depends on public-read; the Infrai model keeps public_url null. Its lifecycle minimum is one day, multipart fragments have no automatic cleanup rule, and server-side metadata isn't searchable beyond prefix-based listing, so hour-scale expiry and compliance discovery need application-owned processes. Those limits are acceptable for a controlled receipt archive only when the surrounding system explicitly supplies the missing controls.

The final decision record should name the chosen retention cohort, protected-copy count, measured throughput, recovery objective, data jurisdictions, deletion approver, and evidence owner. Keep it short. Re-run the test when file-size distribution, region, contract, or retention law changes, because an old passing result is evidence of an old system.

References

• MDN, Content-Disposition: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Disposition
• AWS, Object lifecycle management: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html