When I started thinking seriously about building a large-file transfer product, Amazon S3 felt like the obvious architecture.

It is reliable.

It scales.

It has excellent tooling.

Every backend framework knows how to work with it.

And the standard implementation is incredibly straightforward:

User selects file
      ↓
Upload to S3
      ↓
Generate download URL
      ↓
Recipient downloads file

For most products, that is a perfectly reasonable design.

In fact, if I were building document storage, backups, team collaboration, or a Dropbox-style product, I would probably reach for object storage immediately.

But ButterShare was supposed to solve a slightly different problem.

I was not trying to build:

a place where users store files

I was trying to build:

a way to move a file from one device to another

That distinction eventually changed the entire architecture.

Instead of making S3 the center of the product, I started experimenting with direct peer-to-peer transfer using WebRTC.

The architecture became closer to:

Sender
   ↓
Receiver

instead of:

Sender
   ↓
S3
   ↓
Receiver

That sounds like a small change.

It is not.

It changes bandwidth economics, infrastructure complexity, user experience, privacy assumptions, failure modes, and even what the product fundamentally is.

This is why I chose P2P instead of building ButterShare around S3.

First: S3 is not the problem

I want to make this clear before going any further.

S3 is excellent.

Object storage is one of the most useful infrastructure primitives we have.

If your application needs files to remain available after the uploader disappears, object storage makes enormous sense.

For example:

• profile images
• backups
• user-generated content
• product assets
• documents
• media libraries
• asynchronous downloads
• team collaboration
• long-term storage

S3 is extremely good at those things.

My decision was not:

S3 is bad.

It was:

S3 solves a different problem than the one I want ButterShare to solve.

That difference is important.

Transfer and storage are not the same thing

Suppose Alice has a 100GB video project.

Bob needs that project.

There are two ways to think about the problem.

Storage-first approach

Alice uploads the file somewhere.

Alice
  ↓
Cloud storage

Bob later downloads it.

Cloud storage
  ↓
Bob

The cloud copy acts as the middleman.

This is asynchronous.

Alice can upload the file, close her computer, go offline, and Bob can download it hours later.

That is incredibly useful.

But there is another model.

Transfer-first approach

Alice and Bob are both online.

Alice's goal is simply:

Get these bytes from my machine to Bob's machine.

So instead:

Alice
  ↓
Bob

No permanent intermediate copy is required.

That is the model that interested me.

The 100GB thought experiment

Large files make the architectural difference easier to see.

Imagine someone transfers a 100GB file.

With an S3-style workflow:

Sender → S3
100GB

Then:

S3 → Recipient
100GB

The user thinks they are performing one transfer.

But from the infrastructure perspective, the data has moved through two major legs.

You also temporarily have to store 100GB.

Now imagine 1,000 transfers like that.

You are no longer building a tiny file-sharing app.

You are operating a serious storage and bandwidth system.

That may be exactly what you want.

But I started asking:

Why should my infrastructure handle the entire file twice if the sender and receiver are online at the same time?

That question pushed me toward P2P.

What an S3 architecture would look like

The conventional architecture is actually very attractive.

Your backend generates a presigned upload URL.

The browser uploads directly to object storage.

Something like:

Browser
   |
   | request upload URL
   v
Backend
   |
   | presigned URL
   v
Browser
   |
   | upload
   v
S3

Then the recipient gets a download URL:

Recipient
   |
   | download
   v
S3

This avoids sending the large file through your application server.

That is already much better than:

Browser
   ↓
Node.js server
   ↓
S3

for huge files.

Your Node server does not need to proxy 100GB.

S3 handles the actual upload and download.

This architecture is mature, simple, and reliable.

So why not stop there?

Because ButterShare's goal was not storage-backed sharing.

The cost model changes as files get bigger

For small files, infrastructure cost is often barely noticeable.

A user sending:

2MB

is not going to destroy your startup.

But the economics look different when your product specifically encourages:

10GB
50GB
100GB
200GB+

Now you care about:

• object storage
• upload traffic
• download traffic
• egress
• API requests
• multipart upload
• abandoned uploads
• lifecycle cleanup
• replication
• logging

And most importantly:

the bigger your product succeeds, the more file data your infrastructure has to handle

That creates a very direct relationship:

more users
=
more storage
+
more bandwidth
+
higher infrastructure cost

P2P changes that relationship.

P2P moves the data plane away from my servers

With a peer-to-peer architecture, my backend still exists.

But it mostly handles coordination.

For example:

Sender
   |
   | create session
   v
Signaling server
   |
   | connection information
   v
Receiver

Once WebRTC establishes the peer connection:

Sender <----------------------> Receiver

The actual file can travel directly between them when network conditions allow.

My backend is responsible for things like:

• creating transfer sessions
• exchanging signaling data
• handling room state
• coordinating peers
• expiring sessions

It does not need to become the permanent home of every file.

That is a very different infrastructure problem.

Control plane vs data plane

This was a useful way for me to think about the architecture.

Control plane

Small messages.

Things like:

Create room
Join room
Peer connected
SDP offer
SDP answer
ICE candidate
Transfer started
Transfer finished

These messages are tiny.

They are easy and cheap to handle.

Data plane

The actual file:

100GB video

That is the expensive part.

With an S3 architecture, the cloud owns the data plane.

With P2P, I try to keep the data plane between the users whenever possible.

That separation is one of the biggest reasons I liked the architecture.

P2P does not mean "no servers"

This is one of the biggest misconceptions around WebRTC.

People hear:

peer-to-peer

and imagine:

No backend
No infrastructure
No server bills

Unfortunately, physics is not that generous.

You still need infrastructure.

For ButterShare, that can include:

• application servers
• signaling
• session coordination
• STUN
• TURN
• monitoring
• analytics
• abuse prevention
• databases or temporary state

And TURN deserves special attention.

TURN is the part that ruins the perfect diagram

In the ideal case:

Sender <----------------------> Receiver

Great.

But users live behind:

• NAT
• corporate firewalls
• carrier-grade NAT
• VPNs
• restrictive routers
• mobile networks

Sometimes a direct connection cannot be established.

Then WebRTC may use a TURN relay.

The architecture becomes:

Sender
   ↓
TURN server
   ↓
Receiver

Now your infrastructure is handling the bandwidth again.

So P2P does not magically eliminate bandwidth costs.

It changes how often you need to pay them.

That means one of the most important metrics in a P2P file-transfer product is:

What percentage of transfers are direct?

If most transfers are direct, the infrastructure economics can be very attractive.

If most transfers require TURN, your architecture starts looking much more bandwidth-heavy.

But TURN is still not S3

Even when a file goes through TURN, there is an important conceptual difference.

With object storage:

Sender
   ↓
Stored file
   ↓
Recipient

With TURN:

Sender
   ↓
Relay
   ↓
Recipient

TURN relays traffic.

It does not need to create a persistent copy of the full file for later downloading.

That means you still avoid many storage-related concerns.

No persistent file storage changes the operational burden

If I stored arbitrary user files, I would immediately inherit an entire category of problems.

For example:

Storage lifecycle

How long do files stay?

1 hour?
24 hours?
7 days?
30 days?

At some point you need cleanup jobs.

Abandoned uploads

What happens when someone uploads 80GB of a 100GB file and closes the browser?

Do you keep the partial upload?

For how long?

Storage quotas

If the product is free, users will eventually test how free it really is.

You need policies.

Abuse

If your servers permanently store arbitrary files, users can upload things you absolutely do not want to host.

Now you need systems around:

• abuse reports
• takedowns
• malware
• illegal content
• spam
• automated storage abuse

Direct transfer does not eliminate every abuse issue.

But not becoming a long-term file host changes the problem dramatically.

Privacy was another reason

There is also a product philosophy difference.

With traditional cloud transfer, the file usually exists in three places during the workflow:

Sender device
Cloud provider
Recipient device

With direct transfer, the goal is closer to:

Sender device
Recipient device

That is appealing for a product whose main job is:

move this file

rather than:

store this file for me

Again, this does not magically make every P2P implementation private or secure.

The details matter.

You still need secure signaling, strong session identifiers, encrypted transport, good session expiration, and careful metadata handling.

But architecturally, not retaining the actual file reduces how much user content my service needs to hold.

I like that.

Why didn't I just use S3 multipart upload?

That was one of the obvious alternatives.

S3 supports multipart uploads, which are excellent for large files.

Instead of uploading one huge object in one request, you upload parts.

Conceptually:

100GB file

Part 1
Part 2
Part 3
...
Part N

If one part fails, you retry that part instead of restarting the entire upload.

It is a proven solution.

But multipart upload solves:

How do I reliably upload a very large object to cloud storage?

My question was different:

Do I need to upload the object to cloud storage at all?

That is the architectural fork.

The cloud-storage architecture has a major UX advantage

There is one area where S3 wins very clearly:

asynchronous transfer.

Suppose Alice uploads a file at 2 PM.

She closes her laptop.

Bob opens the link at midnight.

With S3:

Works perfectly.

With P2P:

Alice is offline.
No transfer.

That is a major limitation.

With ButterShare's direct-transfer model, both peers need to participate in the live transfer.

That is not a small tradeoff.

It fundamentally changes the use case.

Why I accepted that tradeoff

Because I wanted to optimize for:

"I have a huge file.
You are online.
I need to give it to you now."

Examples:

• video editor sending footage
• photographer sending RAW files
• developer sending a VM image
• designer sending a large project
• friend sending a huge archive
• creator moving footage between devices

For this workflow, I do not necessarily need persistent cloud storage.

I need a reliable pipe.

That is the niche.

This is why I don't think P2P replaces S3

They are complementary.

If I need:

storage
persistence
sharing later
multiple downloads
CDN delivery

I would choose object storage.

If I need:

live transfer
one sender
one receiver
large file
both online

P2P starts looking very attractive.

The architecture should follow the use case, not ideology.

The biggest advantage: no duplicate cloud journey

This is the simplest way I explain the product to myself.

With cloud storage:

Your file is here.

Destination is there.

But first send it somewhere else.

That "somewhere else" is valuable because it gives you persistence.

But if persistence is unnecessary, it can feel redundant.

With P2P:

Source → Destination

That directness becomes more meaningful the larger the file is.

Large files expose the difference

Imagine transferring a 20MB PDF.

Does the architecture matter?

Probably not to the user.

Whether it goes:

Sender → S3 → Receiver

or:

Sender → Receiver

the whole thing may happen quickly.

Now imagine:

150GB

The intermediate step becomes much more visible.

The user may have to upload 150GB before the recipient can even start downloading, depending on the implementation.

Direct transfer lets the destination receive data as it is being sent.

Time-to-first-byte is different too

In a standard upload-then-share flow:

Upload 100GB
      ↓
Finish upload
      ↓
Recipient starts

In a direct-transfer flow:

Sender starts
      ↓
Recipient starts receiving

almost immediately after the connection is established.

This changes the experience.

The receiver does not necessarily have to wait for an entire cloud upload to finish first.

The bandwidth bottleneck still exists

P2P does not make the user's internet connection faster.

This is important.

Suppose:

Sender upload:
20 Mbps

Receiver download:
500 Mbps

The transfer is still constrained by the sender.

A 100GB file is still a 100GB file.

WebRTC does not violate physics.

The architectural advantage is not:

the file becomes tiny

It is:

the sender's upload is going toward the actual destination rather than an intermediate storage system first

That is a much more honest way to think about it.

P2P also creates harder engineering problems

Choosing P2P was not choosing the easier architecture.

In many ways, S3 would have been much easier.

With S3, you get:

• durable storage
• resumable multipart uploads
• globally mature infrastructure
• reliable HTTP downloads
• straightforward authorization
• mature SDKs
• monitoring
• excellent tooling

With P2P, I have to think about:

• signaling
• NAT traversal
• TURN
• WebRTC Data Channels
• browser differences
• connection failures
• memory
• backpressure
• reconnection
• resumability
• long-running browser sessions
• mobile background behavior

So why do it?

Because those engineering problems are directly aligned with the product I want.

S3 turns the server side into the easy part

Imagine implementing the S3 version.

Your backend might create a presigned URL:

const uploadUrl = await createPresignedUploadUrl({
  key: fileId,
  contentType: file.type
})

The client uploads:

await fetch(uploadUrl, {
  method: "PUT",
  body: file
})

Then you save metadata:

file ID
owner
size
storage key
expiration

Finally, create a download URL.

That is relatively straightforward.

Now compare that to P2P.

The P2P version requires connection negotiation

Before a single byte moves, I need:

Sender joins room
Receiver joins room
Exchange SDP
Exchange ICE candidates
Attempt direct connection
Possibly fallback to TURN
Open Data Channel
Negotiate transfer metadata
Start streaming chunks

That is much more complicated.

The cost benefit comes with engineering complexity.

There is no free lunch.

The first naive P2P implementation was terrible

The first instinct is:

const buffer = await file.arrayBuffer()
channel.send(buffer)

Works great for a small screenshot.

Then someone selects:

50GB

Goodbye browser.

Large P2P transfer requires chunking.

Something like:

const CHUNK_SIZE = 64 * 1024

let offset = 0

while (offset < file.size) {
  const chunk = file.slice(offset, offset + CHUNK_SIZE)
  const buffer = await chunk.arrayBuffer()

  channel.send(buffer)

  offset += chunk.byteLength
}

But even that is not enough.

You also need flow control.

Backpressure became one of the most important concepts

If JavaScript produces chunks faster than the connection sends them, WebRTC queues them.

You can see that queue using:

channel.bufferedAmount

If you ignore it:

send
send
send
send
send
send

eventually memory starts growing.

So a real sender needs to respect the buffer.

For example:

async function waitForDrain(channel) {
  const MAX_BUFFER = 4 * 1024 * 1024

  while (channel.bufferedAmount > MAX_BUFFER) {
    await new Promise(resolve => setTimeout(resolve, 10))
  }
}

Then:

while (offset < file.size) {
  await waitForDrain(channel)

  const chunk = file.slice(offset, offset + CHUNK_SIZE)
  const buffer = await chunk.arrayBuffer()

  channel.send(buffer)

  offset += chunk.byteLength
}

This is the type of problem you never need to think about when S3 is handling everything.

The receiver has an even bigger challenge

Suppose I successfully stream a 100GB file.

What happens on the receiver?

The naive version:

const chunks = []

channel.onmessage = event => {
  chunks.push(event.data)
}

Then:

const blob = new Blob(chunks)

That is fine for a 20MB file.

For 100GB, it is ridiculous.

You need progressive writing.

The receiver needs to write chunks somewhere as they arrive rather than keeping the entire file in memory.

That means browser filesystem APIs, writable streams, or other platform-specific strategies.

This was one of the moments where I realized:

Direct large-file transfer is basically a systems problem disguised as a web app.

Browser compatibility becomes infrastructure

If you build around S3 and HTTP, browsers already know how to upload and download files very well.

When you move more of the transfer engine into the browser, browser behavior becomes part of your infrastructure.

You now care deeply about:

Chrome
Edge
Firefox
Safari
iOS Safari
Android Chrome

And they do not all behave identically.

You care about:

• filesystem APIs
• memory behavior
• background tab suspension
• WebRTC quirks
• download handling
• mobile lifecycle

Your "server infrastructure" may be smaller.

But your client-side complexity becomes much larger.

Mobile is where P2P gets painful

A desktop user might leave a browser tab open for an hour.

A mobile user will:

• switch apps
• lock their phone
• receive a call
• enable battery saver
• move between Wi-Fi and cellular
• let the OS suspend the browser

For a storage-backed upload, the architecture can sometimes recover more naturally.

For a live P2P connection, both devices matter.

That makes mobile reliability one of the hardest parts.

Reconnection is mandatory for huge files

If I am asking someone to transfer:

100GB

I cannot reasonably say:

If your Wi-Fi drops after 99GB, please start again.

That is unacceptable.

P2P means I need application-level resume logic.

For example, the receiver can say:

{
  "fileId": "abc123",
  "receivedBytes": 84639214822
}

The sender can then continue from:

file.slice(receivedBytes)

Of course, a real implementation needs more validation than that.

But resumability becomes part of your transfer protocol.

S3 gives you mature multipart-upload patterns.

With P2P, you own this problem.

So why not let S3 handle failures for me?

Because every architecture has tradeoffs.

S3 gives me easier reliability.

P2P gives me a different economic and product model.

I decided the complexity was worth exploring because the defining promise of ButterShare is not:

We built another upload box.

It is:

Move large files directly between devices.

If I hide an S3 upload behind the UI, I am building a different product.

The economics become especially interesting for free products

This mattered to me because ButterShare is designed to be accessible without charging someone just because their file is large.

Cloud storage products usually have a very understandable reason for limits.

If every free user can upload:

200GB

and leave it sitting on your infrastructure, costs can explode.

So products impose:

• storage caps
• transfer caps
• expiration
• paid tiers
• monthly limits

That is completely rational.

But if the service does not permanently store the file, the economics change.

It does not make bandwidth free.

It does not eliminate TURN.

But the business is no longer paying to keep every file around.

That opens up different product possibilities.

Storage creates another problem: unused files

Suppose a user uploads 70GB.

They generate a link.

The recipient never downloads it.

With object storage:

you still stored 70GB

Maybe only for a day.

Maybe a week.

But the resource was consumed.

With live P2P transfer:

no receiver
=
no 70GB transfer

This aligns infrastructure usage more closely with actual transfers.

I find that elegant.

Multiple recipients change the answer

Now let's say Alice needs to send the same 100GB file to:

100 people

Suddenly P2P is less obviously attractive.

Alice would potentially need to upload enormous amounts of data repeatedly.

This is where cloud storage/CDN architecture becomes much better.

Upload once:

Alice → Cloud

Then distribute:

Cloud → 100 recipients

That is exactly what cloud infrastructure is good at.

Again:

P2P is not universally better.

It is optimized for a different topology.

My ideal P2P use case

The architecture makes the most sense when the shape looks like this:

1 sender
1 receiver
both online
large file
one-time transfer
no long-term storage required

That is the core scenario I optimize for.

Once the requirements move away from that, object storage becomes increasingly attractive.

S3 would win for collaboration

Imagine building:

shared project folders
version history
team permissions
comments
persistent links
multiple devices

I would absolutely want centralized storage.

Trying to turn pure P2P into a Dropbox replacement would create unnecessary pain.

ButterShare is intentionally focused on transfer.

Not collaboration storage.

That product boundary matters.

Architecture affects the user interface too

With S3, I could show:

Upload complete.

You can close this page.

With P2P, I need to say:

Keep this tab open while the transfer is in progress.

That is objectively less convenient in some situations.

But P2P gives me advantages elsewhere:

• no cloud upload stage
• no account needed for storage
• no storage quota in the traditional sense
• no waiting for a full upload before live transfer can proceed
• direct cross-device workflow

The UX tradeoff reflects the architecture.

Architecture also changes what "file size limit" means

In a storage-backed system, limits may come from:

• plan restrictions
• maximum object size
• upload gateway limits
• account storage quotas
• infrastructure policy

With a streaming P2P design, I do not need to allocate storage equal to the entire file on my server.

So the application can avoid imposing a traditional cloud-storage file-size cap.

But practical constraints still exist.

For example:

• sender storage
• receiver storage
• browser capabilities
• network stability
• available memory
• transfer duration
• device behavior

So "no file size limit" should never mean:

There are no physical limitations anywhere.

It means the product is not intentionally saying:

Sorry, 5GB maximum.
Pay $20.

just because the service would otherwise have to store the file.

One thing I really like: simpler data retention

If the file is not stored on my infrastructure, there is less file data to manage after the transfer.

I do not need a lifecycle like:

uploaded
available
downloaded
expired
scheduled for deletion
deleted

for every file object.

The transfer session can be much more ephemeral.

Something like:

created
peer joined
connected
transferring
completed
expired

That feels much closer to the product I wanted to build.

But metadata still needs discipline

Not storing file bytes does not mean:

collect everything else forever

You still need to think about what metadata is necessary.

For example:

Do I really need to permanently store:

filename
file size
IP address
recipient metadata
transfer history

for every transfer?

Maybe not.

A privacy-conscious architecture is not just about avoiding object storage.

It is also about minimizing unnecessary metadata.

Security still matters enormously

WebRTC transport is encrypted, but security does not end there.

You still need to think about:

• session entropy
• room guessing
• unauthorized peers
• session expiration
• signaling abuse
• rate limits
• replay
• malformed messages

For example, if your room ID is too predictable:

ABC001
ABC002
ABC003

then your architecture has a problem even if the underlying transport is encrypted.

The entire system matters.

P2P makes abuse different, not nonexistent

This is another important distinction.

If I am not storing files, I reduce my role as a file host.

But users can still abuse:

• signaling endpoints
• TURN bandwidth
• session creation
• automated connections
• resource-intensive transfers

So abuse prevention becomes more network-oriented.

You may need:

rate limiting
session limits
TURN credentials
temporary tokens
connection quotas

You have fewer storage-abuse problems but more real-time infrastructure concerns.

TURN abuse can become very expensive

Imagine someone intentionally forces relay connections and pushes huge amounts of traffic through your TURN servers.

Congratulations.

Your supposedly cheap P2P application just became a bandwidth pipe.

TURN credentials should not be treated casually.

Short-lived credentials, authentication, rate limiting, and monitoring matter.

This is one area where P2P architecture can surprise people.

Metrics I care about more than storage usage

In an S3 architecture, I might watch:

storage consumed
object count
download bandwidth
upload bandwidth
lifecycle cleanup

In P2P, I care more about:

peer connection success rate
direct connection percentage
TURN percentage
average transfer size
average transfer duration
completion rate
disconnect rate
resume rate
browser failure rate

These numbers tell me whether the architecture is actually working.

Completion rate matters more than theoretical savings

Let's say P2P reduces infrastructure cost dramatically.

Great.

But if:

30% of 100GB transfers fail

then the architecture is a failure.

Users do not care that my AWS bill is low.

The product has to work.

That is why my priority order is roughly:

1. Reliability
2. Compatibility
3. UX
4. Performance
5. Infrastructure efficiency

Cheap broken software is still broken software.

Why I wouldn't build this with my Node.js server in the middle

There is another architecture I could have chosen:

Sender
   ↓
Node.js server
   ↓
Receiver

No S3.

Just stream everything through my backend.

For small transfers, this can work.

For huge transfers, I would be turning my app server into a giant bandwidth proxy.

Every transfer consumes:

• inbound bandwidth
• outbound bandwidth
• open connections
• server resources

At scale, that does not sound attractive.

If my goal is direct transfer, WebRTC is much more aligned with the architecture.

What about WebSockets?

WebSockets are excellent for real-time communication.

I could theoretically do:

Sender
   ↓
WebSocket server
   ↓
Receiver

But again, the server is carrying every byte.

For a chat message:

perfect

For:

100GB

I would rather not make my server the middleman if I can avoid it.

That is another reason WebRTC Data Channels are interesting.

Why WebRTC instead of reinventing transport

Browsers already expose WebRTC.

It gives me:

• encrypted transport
• peer connection negotiation
• ICE
• STUN/TURN integration
• Data Channels
• cross-browser networking primitives

That is a huge amount of functionality.

Building a native P2P protocol across every platform would be much more difficult.

The browser becomes the common runtime.

That means:

Windows
Mac
Linux
Android
iPhone

can potentially participate without dedicated native clients.

That distribution advantage matters a lot to me.

The browser is also the biggest limitation

The beauty is:

No app installation.

The pain is:

You are inside a browser.

You do not control:

• lifecycle
• memory limits
• background execution
• filesystem support
• OS behavior

So the architecture is a constant tradeoff.

The browser gives incredible reach.

Native apps give more control.

For ButterShare, I decided reach was worth the constraints.

P2P changes scaling in an interesting way

In a storage-centric product, user growth often means your central infrastructure moves proportionally more file data.

In a P2P product, some of that load is distributed to the edge.

That is conceptually interesting.

Instead of:

10 users → server traffic
1,000 users → much more server traffic
100,000 users → enormous server traffic

the peers carry much of the data themselves when direct connections succeed.

Your signaling infrastructure can scale very differently from the files themselves.

This is one of the most attractive technical properties of the design.

But connection count still matters

Even if your server is not handling the bytes, you may still have thousands or millions of sessions.

You need to scale:

• signaling connections
• WebSockets
• room state
• ephemeral session data
• TURN credential generation
• monitoring

P2P reduces one category of scaling problem.

It does not eliminate scaling.

Why I didn't choose a hybrid storage-first architecture

A hybrid model is also possible.

For example:

Try P2P first.

If direct transfer fails:
upload to S3.

This sounds great.

But it dramatically increases product complexity.

Now you have two transfer engines:

P2P path
cloud path

And you need to answer:

• When do we switch?
• Does the user know?
• Who pays for storage?
• How long is the cloud copy retained?
• Do we start uploading immediately as fallback?
• What if P2P reconnects?
• How does resume work across both modes?

A hybrid architecture may make sense later.

But for a focused product, simplicity of product behavior has value.

Why not automatically upload while trying P2P?

You could do both:

Sender → Receiver
   \
    → S3 backup

Then if the peer connection fails, the recipient continues from the cloud.

Very robust.

Also potentially very expensive.

You just recreated the cloud bandwidth problem even when direct transfer works.

Again, architecture follows priorities.

The simplicity I wanted was conceptual, not technical

This is a funny thing about ButterShare.

The technical implementation is complicated.

But the product concept is extremely simple:

I have a file.

You need the file.

Let's connect.

I wanted the infrastructure to reflect that as much as possible.

That was more important to me than choosing the easiest backend implementation.

What I would choose for different products

If I were starting other products today, my choices would be different.

Photo backup app

S3/object storage.

Team document workspace

S3/object storage.

Video hosting platform

Object storage + CDN.

Software distribution

Object storage + CDN.

One-time 100GB transfer between two online people

P2P becomes very interesting.

The phrase:

use the right tool for the job

is boring.

It is also correct.

A simplified architecture for ButterShare

At a high level, the system looks like this:

               +-------------------+
               | Signaling Service |
               +-------------------+
                    ↑          ↑
                    |          |
                    |          |
                    ↓          ↓
               Sender      Receiver
               Browser      Browser
                   \          /
                    \        /
                     \      /
                      \    /
                WebRTC Connection
                        |
                        |
                 Direct if possible

If direct connection cannot be established:

Sender
   |
   v
TURN relay
   |
   v
Receiver

The important part is that the file does not need to become a traditional stored object before the recipient can receive it.

The decision came down to product identity

The more I thought about it, the less this felt like a pure infrastructure decision.

It became a product decision.

If I built ButterShare around S3, the product would effectively be:

Upload files to ButterShare and share them.

With P2P, the product becomes:

Use ButterShare to connect two devices and transfer files.

Those are subtly different.

I wanted the second one.

What I gave up by not choosing S3

There are real downsides.

I gave up easy:

• asynchronous downloads
• persistent links
• sender-offline downloads
• one-to-many distribution
• mature multipart infrastructure
• global cloud delivery
• simple browser downloading
• durable server-side resume state

Those are significant benefits.

I do not pretend otherwise.

What I gained

In exchange, I gained an architecture designed around:

• direct transfer
• very large files
• minimal persistent file storage
• potentially lower central bandwidth requirements
• no traditional cloud-storage quota requirement
• cross-platform browser access
• simple one-time transfer sessions

For ButterShare, those benefits align closely with the product.

That is why the tradeoff made sense.

The lesson I would give another developer

Do not start with:

Should I use S3 or WebRTC?

Start with:

What exactly is my product doing with the file?

Ask:

Does the file need to exist tomorrow?

Does the sender need to stay online?

Will multiple people download it?

Does it need versioning?

Is this storage or transfer?

How large are the files?

Who pays for bandwidth?

How important is app-free browser access?

The answers will usually tell you which architecture makes sense.

My decision tree

This is roughly how I think about it.

If the answer to:

Must the file remain available when the sender goes offline?

is:

Yes

use cloud storage.

If:

Does the file need to be downloaded by many people?

is:

Yes

cloud storage/CDN probably wins.

If:

Is this basically long-term storage?

is:

Yes

use object storage.

But if:

one sender
one receiver
both online
large file
one-time transfer

then P2P deserves serious consideration.

Final thoughts

Choosing P2P instead of S3 made ButterShare harder to build.

There is no question about that.

S3 would have solved many problems for me immediately.

With WebRTC, I had to care about:

• peer discovery
• signaling
• STUN
• TURN
• NAT traversal
• browser memory
• chunking
• backpressure
• progressive writes
• reconnection
• resume
• mobile browsers

That is a lot of engineering.

But it also created a product architecture that matches the problem I actually wanted to solve.

I am not trying to create another cloud drive.

I am trying to make this:

Device A
   ↓
Device B

feel as simple as possible.

That is why I chose P2P.

Not because S3 is bad.

Not because P2P is magically free.

Not because WebRTC solves everything.

But because for one-time, extremely large transfers between two online devices, sending the file directly to its destination makes a lot of sense.

And once I started thinking about file transfer that way, it became very difficult to go back to:

Upload it somewhere else first.

I'm building that idea into ButterShare, a browser-based peer-to-peer file transfer tool for sending large files directly between devices without relying on traditional cloud file storage.

ButterShare:

https://buttershare.com/

TL;DR

I chose P2P instead of building ButterShare around S3 because the product is primarily about transferring files, not storing them.

With S3:

Sender → Cloud → Receiver

With P2P:

Sender → Receiver

P2P gives me:

• no requirement to permanently store every file
• potentially less centralized bandwidth usage
• direct device-to-device transfer
• better economics for very large one-time transfers
• a browser-based cross-platform workflow

But I also accept:

• harder engineering
• TURN costs
• both users needing to be online
• more browser complexity
• harder resume and reliability problems

For Dropbox-style storage, I would choose object storage.

For sending a 100GB file between two people who are online right now, I think P2P is a much more interesting architecture.