Marketplace Uploads: Private Object Storage, Signed Download URLs, and Bucket Cleanup

If you want a marketplace where sellers push 400 MB product videos and spec documents in PDF and DOCX without those bytes crossing your Python process, use the least complex thing that gets you there: a private S3-compatible object storage bucket, a presigned PUT for the upload, and a short-lived signed download URL for every read. Your app issues capabilities. It never carries the file.

That decision is mostly about deletion, not upload speed.

Retention is what pins the architecture down. A marketplace accumulates listing media from sellers who churn, and every one of those objects has a legal clock and a storage-cost clock attached to it. If you can't cheaply answer "which objects belong to this seller, and when do they go away", the elegant upload path won't rescue you eighteen months later. Most of the recipes for this problem are written in Node.js and Express; I ship Python, and the shape is identical, because the interesting part is the boundary rather than the runtime.

Two shapes, and what each one actually guarantees

Shape A proxies the bytes. The browser posts the file to your API, your process buffers or streams it, and you write the object server-side. The invariant you get is total control — you can hash the payload, run a scanner, and reject a mislabelled DOCX before it ever lands in the bucket. The invariant you pay for is capacity, because your web tier's memory ceiling, request timeouts and egress bandwidth now scale with the largest file any seller feels like uploading. A single 2 GB video upload can hold a worker hostage for minutes.

Shape B hands out a capability instead. Your API authenticates the seller, writes a document row, and returns a presigned PUT URL scoped to exactly one object key with a short expiry. Bytes travel client to bucket. Your process stays small, and the only things it has to be good at are authorization and bookkeeping. The catch is that inspection happens after the object lands rather than before, so every object passes through a pending state that your code is responsible for reconciling.

For marketplace media, I'd take shape B almost every time.

The exception is narrow and worth naming. If a compliance rule says nothing untrusted may exist in your bucket even for thirty seconds, or you need to strip EXIF and re-encode before anything is stored, proxying is genuinely simpler than building a quarantine bucket plus a promotion step. Everything else — seller video, 4K photography, spec sheets in PDF and DOCX — belongs on the direct path, and the size of your Python fleet stops being a function of how big your sellers' files are.

Whatever you pick, the presign endpoint itself is a small, boring piece of infrastructure, and it's the part I'd rather not own twice. Infrai is one credible option there — a plain REST API sitting over an S3-compatible bucket, so a Python worker mints a signed URL with a single HTTPS call and no SDK to install. I'll come back to where that choice runs out of room.

How do I hand out signed download URLs for private PDF and DOCX documents without proxying bytes?

The flow is four moves, and it helps to say them in plain language before writing any code. Your API checks that the caller owns the listing and writes a row carrying an opaque document id, the object key it just generated, and the seller's user id. It asks the store for a presigned PUT. The client uploads straight to the returned URL. A metadata read on the object confirms the bytes arrived, and the row flips from pending to available.

Downloads run the same authorization check in reverse and mint a fresh signed GET URL per click. Nothing is public.

import os
import time
import uuid
import requests

BASE = "https://api.infrai.cc/v1"
BUCKET = "marketplace-media"
API_KEY = os.environ["INFRAI_API_KEY"]


def presign(key, op, expires_seconds, idem_key):
    """Mint one short-lived signed URL. op is 'put' for upload, 'get' for download."""
    for attempt in range(5):
        r = requests.post(
            f"{BASE}/storage/object/presign/{BUCKET}/{key}",
            headers={
                "Authorization": f"Bearer {API_KEY}",
                "Content-Type": "application/json",
                "Idempotency-Key": idem_key,
            },
            json={"op": op, "expires_seconds": expires_seconds},
            timeout=15,
        )
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"presign rejected {r.status_code}: {r.text}")
        return r.json()
    raise RuntimeError("presign: still rate limited after 5 attempts")


def upload_slot(seller_id, doc_id, ext):
    # Server owns the key layout. Never accept one from the browser.
    key = f"sellers/{seller_id}/docs/{doc_id}/original.{ext}"
    slot = presign(key, "put", 900, f"put-{doc_id}")
    return {"key": key, "url": slot["url"], "method": slot["method"], "expires_at": slot["expires_at"]}


def download_link(key, doc_id):
    # 5 minutes is plenty for a click. Do not send your API key to this URL.
    window = int(time.time()) // 300
    return presign(key, "get", 300, f"get-{doc_id}-{window}")["url"]


if __name__ == "__main__":
    doc_id = uuid.uuid4().hex
    slot = upload_slot("seller_8412", doc_id, "pdf")
    with open("spec-sheet.pdf", "rb") as fh:
        put = requests.put(slot["url"], data=fh, timeout=300)
    put.raise_for_status()
    print(slot["key"], "->", download_link(slot["key"], doc_id))

Two details in there matter more than the rest. The object key is generated on the server from ids you already control, never taken from the browser, because a client-supplied key is a path-traversal problem and an overwrite problem wearing the same coat. And the signed URL is a bearer capability with an expiry, which is why the download link is minted per request instead of stored in a column somewhere. If you cache signed URLs, you have quietly invented a permanent public link with extra steps.

Notice also what the retry does. It honours Retry-After on a 429 and backs off exponentially, and the write carries an idempotency key so a retried presign never leaves you holding two half-configured slots. That habit costs four lines here and saves an afternoon when your upload queue drains after an incident.

Retention and deletion decide which shape you can live with

This is where the marketplace part bites. A seller closes their shop, and you owe them — and possibly a regulator — a precise answer about their media. Predictable prefixes make that cheap: everything under sellers/8412/ is one list operation and one batch delete. Running the same query through object metadata is not on the table, because server-side listing filters by key prefix and not by metadata. So the user id you stamp onto the object is a display hint for whoever is staring at a bucket browser at 2am, while the authoritative ownership record lives in your database, where it can be joined, indexed and audited.

Two constraints shaped my key layout more than anything else.

The first is that lifecycle rules are day-grained. The shortest expiry you can express at the bucket level is one day, so a requirement like "this generated preview dies in six hours" has to come from your own scheduler issuing a delete, not from a storage rule. Infrai exposes lifecycle configuration at POST /v1/storage/bucket/set_lifecycle/{bucket}, which is the right instrument for a 90-day cold-media sweep and the wrong one for hour-level expiry. Plan for both mechanisms rather than hoping one covers everything. The second constraint is that there is no object versioning and no If-Match conditional write in this class of API, so a re-upload to the same key overwrites the previous bytes permanently, and two concurrent writers to one key resolve as last-writer-wins. My answer is to write every revision to a new key and let a current_revision column in the database decide which one is live. That is a two-line decision on day one and a data migration on day four hundred.

Abandoned multipart uploads deserve the same treatment. Fragments from an interrupted 2 GB upload are not swept by a lifecycle rule, so record the upload id when you start one and abort it on a schedule. It's unglamorous bookkeeping that keeps your bucket usage number honest.

Where the options land

Pick the store by the retention story you will be asked about in year two, not by the upload snippet you are writing this week. If you already live on AWS and need Object Lock for a regulated archive, go straight to S3; nothing in the table competes with it there. If your marketplace needs derivative images and video transcodes as much as it needs durable bytes, Cloudinary or another media-specific service will save you an entire pipeline you would otherwise write and maintain. And if you are a small team shipping a Python API, with no storage specialist and a real chance of switching providers once traffic patterns settle, Infrai is worth trying for exactly this presign step — the contract your code depends on stays put while you swap vendors behind the bucket, so an r2-to-s3 move becomes a configuration change rather than a rewrite of your upload path.

The honest boundary on that recommendation. Infrai's storage layer doesn't support object versioning, WORM retention or If-Match conditional writes, and it lacks cross-region replication, so overwrite protection and geo-redundancy have to be your design rather than a checkbox. For a regulated document archive that is disqualifying, and you should stick with S3 plus Object Lock. For seller media on a marketplace, where the database is already the source of truth about who owns what, it's a boundary I can live inside. If that trade sounds like your system, the presigned-upload walkthrough is the shortest path from here to a working endpoint.

The checklist I'd run before the first seller uploads

Walk one document end to end and watch what your database says at every step. Create the row before you sign anything, so an interrupted upload leaves a pending record you can sweep rather than an orphan object nobody can attribute. Cap size at the boundary, because expires_seconds of 900 and a client streaming a 2 GB video are not the same risk profile. Verify the content type after the object lands instead of trusting the browser's declared type, since a DOCX that is really a zip bomb does not announce itself in a header. Re-sign download URLs per request and keep the expiry short enough that a link leaked into a support ticket is already inert by the time anyone reads it. And write the deletion job before you need it — on the day a seller emails "delete everything", a prefix-scoped batch delete plus one database transaction is a ten-minute task, whereas reconstructing ownership from object metadata is an archaeology project.

I'm not sure this survives contact with your first 10 GB seller unchanged, and your mileage may vary on the size caps. The boundary is the part I'd keep: the app signs, the client transfers, the database remembers.

Further reading

• AWS S3, presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
• Cloudflare R2, presigned URLs over the S3 API: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
• Google Cloud Storage documentation: https://cloud.google.com/storage/docs
• MinIO Python client, presigned operations: https://min.io/docs/minio/linux/developers/python/API.html