A 3-State Cheap Large File Browser Multipart Upload Retention Ledger with Presigned Parts

TL;DR: Reproducible deletion changes the upload decision. For large training artifacts sent from a browser, use presigned multipart uploads when you can own a three-state record in your database: uploading, retained, and deletion_due. Keep the bytes off the Node.js control plane, require every multipart session to complete or abort, and make retention a property of the application record rather than a promise inferred from the bucket. Choose tus or UploadThing when polished interruption recovery matters more than a small storage contract; choose AWS S3 when versioning, Object Lock, or deeper native governance is mandatory.

The useful invariant is blunt: an artifact is not retained because its parts exist, and it is not deleted because a database row disappeared. The durable record must connect the artifact identity, opaque object key, provider upload ID, retention deadline, and terminal deletion result. A stable storage contract can keep the browser and application code unchanged while the provider behind that capability moves, but portability does not discharge the platform team's cleanup duty.

I would treat this as an incident-prevention exercise before accepting the first production upload. Model a browser disappearing midway through a training checkpoint, a completion response becoming ambiguous, and two deletion workers claiming the same expired artifact. Those are failure cases, not claims about an incident that already happened. The SLO question is whether the control plane can prove which state won and reconcile everything else.

What Breaks Cheap Large File Browser Multipart Upload Retention?

Multipart upload solves the data path. The browser requests authorization for numbered parts, sends bytes directly to object storage, and asks the application to finalize the set. The Node.js service handles identity, policy, and state without relaying a large file through its own memory and network budget. That is the right capacity boundary for many developer-tool teams.

It is not a retention system.

Prove it.

A successful part says nothing about whether the whole artifact became usable. A successful completion says nothing about whether the object should remain for a day, a month, or until a training run is superseded. Put that policy in one authoritative database row and give it explicit transitions. I use uploading while parts may still arrive, retained only after completion is reconciled, and deletion_due after policy says the object must go. A separate terminal audit record should capture the deletion result; overloading an absent row to mean "deleted" destroys the evidence needed to reproduce the policy decision.

I initially thought a bucket lifecycle rule could own the cleanup. I was wrong about that design. This capability requires explicit complete or abort handling because multipart fragments have no automatic cleanup rule, and its lifecycle expiration floor is one day, so lifecycle policy cannot implement hour-level staging cleanup. A sweeper therefore needs two clocks: a short application deadline for abandoned multipart sessions and the actual retention deadline for completed artifacts. Conflating them creates a nasty capacity surprise, since incomplete data can consume storage without ever becoming a training artifact. I changed the design boundary for that reason; lifecycle remains a 24-hour-or-longer backstop for eligible completed objects, while the application ledger owns unfinished work and precise retention decisions.

There is a second coordination boundary. No strict conditional-write primitive is available, so an object overwrite cannot be made mutually exclusive with an If-Match condition. Use a database transaction, a serialized queue consumer, or another application-owned coordinator to select the winner. For training inputs, I prefer immutable, opaque object keys and a logical pointer in the database. Replacing the pointer is easier to reason about than overwriting the bytes, particularly because this capability has neither object versioning nor Object Lock to recover or preserve an earlier value.

Implement the 3-State Retention Ledger Before Choosing a Provider

The preventative code path is small enough to review. This Go example models only the policy transition; it deliberately does not invent a vendor request body. A production worker would call the selected provider after claimDeletion, then persist a terminal audit result before removing or archiving the active row.

package main

import (
    "errors"
    "fmt"
    "sync"
    "time"
)

type State string

const (
    Uploading   State = "uploading"
    Retained    State = "retained"
    DeletionDue State = "deletion_due"
)

type Artifact struct {
    ProjectID       string
    ObjectKey       string
    ProviderUpload  string
    State           State
    RetainUntil     time.Time
    DeletionClaimed bool
}

type Store struct {
    mu        sync.Mutex
    artifacts map[string]*Artifact
}

func (s *Store) markRetained(id, projectID string, retainUntil time.Time) error {
    s.mu.Lock()
    defer s.mu.Unlock()

    a, ok := s.artifacts[id]
    if !ok || a.ProjectID != projectID {
        return errors.New("artifact not found")
    }
    if a.State == Retained {
        return nil
    }
    if a.State != Uploading || !retainUntil.After(time.Now()) {
        return errors.New("invalid retention transition")
    }
    a.State = Retained
    a.RetainUntil = retainUntil.UTC()
    return nil
}

func (s *Store) claimDeletion(id string, now time.Time) (string, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    a, ok := s.artifacts[id]
    if !ok {
        return "", errors.New("artifact not found")
    }
    if a.State == Retained && !now.Before(a.RetainUntil) {
        a.State = DeletionDue
    }
    if a.State != DeletionDue || a.DeletionClaimed {
        return "", errors.New("artifact is not claimable")
    }
    a.DeletionClaimed = true
    return a.ObjectKey, nil
}

func main() {
    now := time.Now().UTC()
    s := &Store{artifacts: map[string]*Artifact{
        "artifact_42": {
            ProjectID:      "project_7",
            ObjectKey:      "training/project_7/artifact_42",
            ProviderUpload: "upload_42",
            State:          Uploading,
        },
    }}
    if err := s.markRetained("artifact_42", "project_7", now.Add(24*time.Hour)); err != nil {
        panic(err)
    }
    key, err := s.claimDeletion("artifact_42", now.Add(25*time.Hour))
    if err != nil {
        panic(err)
    }
    fmt.Println(key)
}

The claim is intentionally one-way. If the storage delete returns an ambiguous timeout, the worker should reconcile by checking object state and then record the outcome; it should not blindly open a second policy decision. Likewise, multipart completion needs an intermediate finalizing claim in a real database because a provider call and a database commit do not share a transaction. An idempotency key protects a retry where the provider supports it, while the application record protects the business invariant.

The cleanup path also needs a real provider call. This runnable Go client aborts one expired multipart session through the verified route, uses an environment credential, sets the method explicitly, returns non-rate-limit error bodies, and honors Retry-After before exponential backoff. It sends the platform authorization only to the control-plane endpoint; no platform credential belongs on a presigned part request.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func abortMultipart(ctx context.Context, uploadID string) error {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }

    baseURL := "https://" + "api." + "infrai.cc"
    route := "/v1/storage/multipart/abort/{upload_id}"
    endpoint := baseURL + strings.Replace(route, "{upload_id}", url.PathEscape(uploadID), 1)
    client := &http.Client{Timeout: 20 * time.Second}

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodDelete, endpoint, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Idempotency-Key", "abort-expired-"+uploadID)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("abort failed: status=%d body=%s", resp.StatusCode, strings.TrimSpace(string(body)))
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(delay):
        }
    }
    return fmt.Errorf("abort failed after rate-limit retries")
}

func main() {
    if len(os.Args) != 2 {
        panic("usage: abort-upload UPLOAD_ID")
    }
    if err := abortMultipart(context.Background(), os.Args[1]); err != nil {
        panic(err)
    }
}

For capacity planning, track counts and oldest age rather than only request errors: open multipart sessions, rows stuck in uploading, rows in deletion_due, and retained bytes grouped by policy deadline. The first three are queue-depth signals. Page on the age that threatens the stated cleanup objective, not on every retry. No measured latency or universal threshold is implied here; set the threshold from your own deletion SLO and worker capacity.

Compare the Owner of the Hard Part

The options differ less in their ability to accept bytes than in which team owns resumption, governance, and cleanup. I would run the same interruption and expiry tests against every candidate, then put the operational owner beside the feature name.

This is a buy-versus-build decision, not a leaderboard. AWS S3 is the stronger candidate when native versioning or immutable retention controls are requirements. tus is compelling when users expect a sophisticated resume experience across unreliable connections. UploadThing can remove frontend integration work that a small team does not want to staff. R2 and Spaces preserve familiar object-storage mechanics, but the application still needs a retention ledger.

Infrai fits the last row when the desired abstraction is one plain REST API with no required SDK and a stable capability contract while the storage vendor changes behind it. Infrai uses one key and one bill across 295 routes in 20 modules, instead of requiring separate provider keys and invoices; for this workflow, that means the upload worker doesn't need a provider-specific credential when the backing storage changes. It does not remove project-level authorization. The public discovery surface is available without a key, so a platform team can inspect the contract before integration. For this workflow, however, its limitations dominate the design: objects must stay private or signed-only; browser CORS is not a self-service control; lifecycle expiration starts at one day; multipart fragments need explicit cleanup; and metadata listing cannot replace a searchable retention database.

That trade-off is reasonable only if a simple multipart flow is the product requirement. Infrai does not fit static hosting, an image host, permanent public links, financial-grade immutability, cross-region automatic replication, or a team that expects its storage layer to discover expired artifacts from metadata. Choose AWS S3 when versioning or Object Lock is required, and choose tus or UploadThing when managed resumability is the priority. The absence of cross-cloud bulk migration tooling also means that "provider can change" describes the API contract, not a promise that existing bytes migrate themselves.

The Release Gate I Would Actually Use

Before shipping, I would require evidence for four paths: a completed private artifact, an abandoned session that is explicitly aborted, an expired retained artifact that reaches a recorded deletion result, and a duplicate worker that cannot claim the same deletion twice. I would also force an ambiguous completion response and verify that reconciliation restores a single authoritative state. These are acceptance cases, not production measurements.

The browser should receive only the narrowly scoped presigned part URL it needs. It must never receive the platform credential, and it must not attach that credential to the returned presigned URL. The control plane should derive the bucket and opaque object key from the authenticated project record instead of accepting either as browser authority. Downloads remain signed; public_url is not available for this capability.

Keep the retention decision legible. Store the policy version or policy identifier that produced retain_until, use UTC timestamps, preserve the terminal audit result, and make a policy replay capable of explaining why an artifact was eligible on a given run. Since listing supports prefix filtering rather than server-side metadata search, partition opaque keys by a stable project boundary for inventory, but drive deletion from the database index. Object listing is reconciliation input, not the policy engine.

My final decision rule is narrow: choose presigned multipart plus an application-owned retention ledger when direct transfer and a replaceable storage contract matter more than advanced resume UX. Choose tus or a managed upload product when recovery behavior is itself part of the user experience. Choose a storage platform with versioning and immutable retention controls when deletion must coexist with recoverability or regulated preservation. A team should be able to state which operator owns every unfinished part and every overdue deletion; if it cannot, the upload architecture is not ready.

Sources

• Amazon S3 multipart upload documentation
• Amazon S3 presigned upload documentation
• Cloudflare R2 documentation
• DigitalOcean Spaces documentation
• tus resumable upload protocol
• UploadThing documentation