For almost two months, my media delivery path looked like this:
Browser
↓
Application / working server
↓
Nginx
↓
local proxy_cache
↓ cache MISS
Bunny Storage
Images, videos, and audio files lived in Bunny Storage.
Users did not access Bunny directly.
My Nginx server requested objects from Bunny Storage, authenticated to Storage on the server side, cached the response on local SSD, and then served the public URL itself.
On paper, this was a perfectly reasonable architecture.
And when the requested file was already in the Nginx cache, it was extremely fast.
The problem was everything that happened when it wasn't.
After two months of production logs, connection measurements, Nginx configuration changes, MP4 Range experiments, and some very specific ETag errors, I eventually replaced Bunny Storage as the runtime origin with a private media server connected over WireGuard.
The biggest lesson was not that self-hosting is universally better.
It was this:
A fast cache can hide a slow or unpredictable cold path for a surprisingly long time.
And for a media-heavy site, the cold path matters much more than I originally thought.
First, an important distinction: this was not Bunny CDN
I want to make this explicit because otherwise the comparison would be misleading.
I was not using Bunny CDN for this experiment.
I was using Bunny Storage directly as the storage origin behind my own Nginx proxy cache.
So my architecture was essentially:
My Nginx
↓
Bunny Storage API / storage endpoint
not:
User
↓
Bunny CDN
↓
Bunny Storage
Those are different architectures.
Bunny's normal Storage + Pull Zone/CDN setup adds a globally distributed delivery layer designed specifically for serving content to users.
That was not what I was testing.
I still think Bunny CDN is a very good product, and this article should not be read as "Bunny is bad."
This is about a much narrower question:
What happens when Bunny Storage itself becomes the runtime origin behind your own Nginx cache?
For my workload, the answer became increasingly complicated.
Why I built it this way in the first place
My workload is mostly immutable media:
- AVIF
- JPEG
- PNG
- MP4
- WebM
- audio
- other static files
There are a lot of files, with very different sizes.
Some images are tiny.
Some videos are much larger.
I wanted storage separated from the working server, but I also wanted my own Nginx to control:
- public URLs
- cache headers
- cache eviction
- missing-file behavior
- redirects
- media fallbacks
- byte ranges
- video seeking
- HTTP status codes
So I built a classic reverse-proxy cache:
┌── HIT ── local SSD
│
Browser → Nginx cache
│
└── MISS ── Bunny Storage
The design had an obvious advantage.
A popular file only had to come from Bunny once.
After that:
Browser
↓
Nginx
↓
local SSD
No external storage connection.
No remote object transfer.
Very fast.
And that worked exactly as expected.
The mistake was judging the architecture mostly by what happened on a HIT.
A cache HIT tells you almost nothing about your origin
This sounds obvious now.
It did not feel obvious while the system was running.
A popular image gets requested frequently:
request
↓
cache HIT
↓
local SSD
↓
fast
A rarely visited image behaves differently:
old page
↓
file was evicted
↓
cache MISS
↓
connect to Bunny Storage
↓
retrieve object
↓
populate cache
↓
respond
This made older and less popular pages an accidental stress test of the real origin path.
And that is exactly where I started noticing something strange.
The popular pages felt fine.
Older pages could feel dramatically worse.
Sometimes one image would take much longer than everything around it.
Sometimes an image would appear to hang.
That pattern eventually made sense once I looked closely at the Nginx error logs.
The production logs showed that some requests were stalling before the file even started downloading
The most important errors looked like this:
upstream timed out
while connecting to upstream
and, even more interestingly:
upstream timed out ... while SSL handshaking to upstream
That distinction matters.
This wasn't:
large file
↓
slow download
It was sometimes:
cache MISS
↓
try to establish upstream connection
↓
TCP / TLS
↓
timeout
The actual media body had barely entered the picture yet.
Nginx was having trouble completing the connection to the remote storage endpoint.
That matched what I was seeing in the browser.
An image wasn't necessarily loading slowly.
Sometimes the useful transfer had not really started.
This was not a one-request anomaly
One diagnostic snapshot from July 30 contained:
317
upstream-timeout matches in the last:
5,000
lines of the media error log.
A snapshot from the previous day contained:
578
matches.
Those numbers are log matches, not unique users or unique failed objects.
I don't want to turn them into a metric they aren't.
But they do establish something useful:
This wasn't one unlucky request that happened to fail during debugging.
The failure pattern was occurring repeatedly in production.
It also wasn't one permanently bad Bunny Storage IP
During that period, the storage hostname resolved to addresses including:
109.61.89.53
109.61.89.54
109.61.89.55
109.61.89.57
79.127.226.193
I saw connection or TLS-related failures across different addresses in that set.
That does not tell me what the underlying root cause inside the network was.
From my logs alone, I cannot honestly say whether it was:
- a storage backend
- routing
- peering
- my hosting provider's path to Bunny
- balancing behavior
- another network component
I didn't isolate those variables.
What I can say is much narrower:
From my server, the external path to Bunny Storage occasionally failed to establish an upstream connection within the expected time.
That distinction matters.
I don't need to know which router or backend caused the delay to know that it existed in my application's critical path.
One measurement showed exactly why averages were misleading
I tested individual resolved Bunny Storage addresses.
Several paths looked completely reasonable:
109.61.89.53 total ≈ 29.8 ms
109.61.89.54 total ≈ 28.5 ms
109.61.89.57 total ≈ 42.7 ms
79.127.226.193 total ≈ 43.6 ms
Nothing alarming there.
But one measured path looked like this:
109.61.89.55
TCP 1.017782 s
TLS 1.048306 s
TOTAL 1.054474 s
That doesn't mean:
Bunny Storage has 1-second latency.
It clearly didn't.
Most of my measured paths were tens of milliseconds.
The interesting part was the variance.
I could get something like:
30 ms
29 ms
43 ms
1,054 ms
from paths behind the same storage hostname.
That one-second measurement happened before considering normal object-transfer time.
And production logs also showed cases that went beyond "slow" and reached actual connection or TLS timeouts.
This is where average latency becomes a dangerous metric.
For a gallery, p99 matters more than the average image
Imagine a page loading 80 images.
Suppose 79 of them load in:
30–50 ms
and one takes:
1,000 ms
The average can still look respectable.
The page does not.
The user experiences:
████ image
████ image
████ image
████ image
[empty]
████ image
████ image
████ image
If another object hits a multi-second timeout, the experience becomes even worse.
The browser doesn't render "average latency."
It renders individual objects.
That's why media-heavy pages are especially sensitive to long-tail latency.
What matters is often not:
mean request time
but:
p95
p99
slowest cold object on the page
This became one of the most useful performance lessons from the entire migration.
Why older content exposed the problem more often
My old local Nginx media cache became large.
One July snapshot showed roughly:
cache size: ~36 GB
cache files: >355,000
configured limit: ~35 GB
root filesystem: ~93% used
Nearby snapshots approached:
~400,000 cache entries
The exact numbers changed as the cache evolved, but the important part was structural:
The cache could not hold everything forever.
So:
popular media
↓
frequent access
↓
likely still cached
↓
fast HIT
while:
old / rare media
↓
less frequently accessed
↓
more likely evicted
↓
MISS
↓
remote origin exposed
That explains something I found confusing at first.
Why could a site with a huge local cache still feel bad on random old pages?
Because the cache was making the good cases very good while doing nothing to eliminate the cost of the remaining cold misses.
Then video made the architecture much more complicated
Images were only half the story.
Video introduced a completely different problem.
Browsers don't necessarily download an MP4 from byte zero to the end.
They can request byte ranges.
For example:
Range: bytes=0-1048575
Then the user seeks somewhere else:
Range: bytes=50000000-51048575
The server can reply:
206 Partial Content
and return only the requested bytes.
This is essential for normal video seeking.
It also makes caching much more interesting.
My first approach: include the browser Range in the cache key
One early configuration effectively worked like this:
proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;
proxy_cache_key "$scheme|$host|$request_uri|range=$http_range";
proxy_cache_valid 200 206 301 302 30d;
This prevented two different byte ranges from incorrectly sharing one cache entry.
But there was an obvious downside.
Browser ranges are arbitrary.
One client may request:
bytes=0-1048575
another:
bytes=0-999999
another may seek and request:
bytes=58321473-59370048
So one physical MP4 could become:
video.mp4 + Range A → cache object A
video.mp4 + Range B → cache object B
video.mp4 + Range C → cache object C
video.mp4 + Range D → cache object D
...
I didn't measure how many gigabytes this fragmentation consumed, so I'm not going to invent a number.
But the fragmentation mechanism was directly visible in the cache key.
This made me look for something more deterministic.
Nginx Slice looked like the right solution
Nginx has a Slice module specifically for caching large resources in fixed-size segments.
Instead of arbitrary browser ranges:
0–734129
917283–1500000
2000000–2768129
I could normalize the cache into predictable pieces:
0–1 MB
1–2 MB
2–3 MB
3–4 MB
...
The relevant configuration looked roughly like this:
slice 1m;
proxy_set_header Range $slice_range;
proxy_cache_key "$scheme|$host|$uri|$slice_range";
proxy_cache_valid 200 206 30d;
This architecture is much cleaner for reuse.
A user asks for bytes somewhere inside:
1–2 MB
Nginx fetches and caches that normalized slice.
Another user later requests a different sub-range inside the same chunk.
The same cached slice can potentially satisfy it.
Great.
Except production started showing this:
etag mismatch in slice response while reading response header from upstream
That error became one of the most interesting parts of the whole investigation.
The video problem was ETag consistency, not "different signatures"
When I first tried to describe the issue informally, I thought of the pieces as having different "signatures."
That's not the correct technical description.
The relevant HTTP value was:
ETag
Not:
AccessKey
Not a signed URL.
Not a cryptographic signature.
An ETag is a validator for a particular representation of a resource.
Conceptually, Nginx wants something like:
slice 1
bytes 0–1 MB
ETag: A
slice 2
bytes 1–2 MB
ETag: A
Both pieces appear to belong to the same representation.
Now imagine this:
slice 1
ETag: A
slice 2
ETag: B
Nginx has a very good reason not to blindly combine those pieces.
They may represent different versions of the underlying file.
If it continued anyway, the final response could theoretically become:
first part of version A
+
second part of version B
That's corrupt data.
So Nginx aborting the sliced transaction is protective behavior.
And this wasn't theoretical — I had the exact error in production
On July 30, repeated slice subrequests for one MP4 generated:
etag mismatch in slice response
while requests were reaching storage addresses including:
109.61.89.53
109.61.89.57
79.127.226.193
109.61.89.55
On July 31, the same class of error appeared again for another MP4 while slice requests moved among addresses from the same storage pool.
What does that prove?
It proves:
Nginx received slice responses for the same MP4 whose validators were not consistent enough for Nginx to safely assemble them.
What does it not prove?
It does not prove exactly why the ETags differed.
At the time I did not log the literal ETag returned by every individual subrequest.
So I cannot honestly reconstruct the exact internal mechanism.
The movement between different upstream addresses makes backend-response inconsistency a plausible explanation.
But that's an inference.
The production fact is the Nginx error itself.
This does NOT mean Bunny Storage cannot serve Range requests
This distinction is important.
Range requests themselves worked.
I was receiving and caching:
206 Partial Content
So the lesson is not:
Bunny Storage does not support byte ranges.
The narrower issue was the combination of:
Nginx Slice
+
multiple upstream responses
+
ETag consistency required by Nginx
And video could simultaneously suffer from the other problem:
upstream timed out while connecting to upstream
So video had two independent failure modes:
- Origin connection / long-tail latency
- Slice / ETag consistency
That made the overall system much harder to reason about.
I eventually stopped using Slice for MP4
I later moved MP4 back toward ordinary browser Range behavior.
Conceptually:
proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;
proxy_cache_key "$scheme|$host|$uri";
proxy_no_cache $http_range;
proxy_cache_valid 200 30d;
Partial responses would no longer be allowed to masquerade as complete cached files.
That removed the problematic sliced MP4 assembly.
But it exposed a trade-off.
A cold video Range might now require:
Browser
↓
Nginx
↓
MISS
↓
external storage origin
again.
I had fixed one layer of complexity by becoming more dependent on another.
At some point I had to ask a more fundamental question.
I realized I was building part of a CDN in front of a storage service
Look at the mechanisms I had accumulated:
proxy_cache- cache locking
- stale responses
- background updates
- custom cache keys
- Range handling
206 Partial Content- Nginx Slice
- normalized 1 MB chunks
- upstream keepalive
- TLS session reuse
- connection retries
- timeout tuning
None of these are bad features.
They are useful tools.
But together they made me ask:
What do I actually need from my origin?
My answer was surprisingly boring.
I needed something that could:
store immutable files
+
return their bytes
That's it.
I wasn't trying to build a globally distributed public edge network between my two servers.
I had one working server that needed media from one storage server.
So I tried the simplest possible version of that architecture.
The replacement: a boring private media origin
The new design is:
Browser
↓
Working server
↓
Nginx + local media cache
│
├── HIT
│
└── MISS
↓
WireGuard
↓
Media server
↓
Nginx
↓
SSD
The media server does almost nothing.
It stores files.
Nginx serves them.
The origin listener is private.
The working server connects to a fixed private address over WireGuard.
Public HTTPS still terminates on the working server.
Inside the encrypted WireGuard tunnel, the origin connection can be plain HTTP:
http://private-ip:port
I don't need a second TLS negotiation inside a tunnel that is already encrypted.
Why a tiny server is enough for this kind of job
For a pure static-file origin, the application logic is basically nonexistent.
There is no:
- SSR
- database
- application runtime
- authentication system
- template rendering
- expensive API logic
The critical resources are much more boring:
- enough SSD capacity
- enough sequential/random disk performance
- enough network throughput
- enough file descriptors
- sensible Nginx connection limits
For this kind of role, I'd consider a machine in the class of:
~1 vCPU
~1 GB RAM
hundreds of GB of SSD
as a reasonable starting point for a small static origin, assuming the expected bandwidth, concurrency, and disk behavior fit.
That's not a universal sizing rule.
My current validation host has more headroom, so the latency numbers below should not be interpreted as a scientific benchmark of exactly a 1-vCPU / 1-GB machine.
The point is architectural:
Static file origins are cheap workloads compared with many application workloads.
The new cold path has far fewer moving parts
The old MISS path looked approximately like this:
Working server
↓
public DNS
↓
one of several storage addresses
↓
public network path
↓
TCP connection
↓
TLS handshake
↓
Storage request
↓
object
The new path looks like:
Working server
↓
WireGuard
↓
fixed private IP
↓
Nginx
↓
SSD
The network still exists.
Encryption still exists.
Failure is still possible.
But I removed variables that did not add much value to this specific one-to-one origin relationship.
Most importantly, the cold path became predictable.
The difference was immediately visible in the measurements
After the migration, small real-file Range requests from the working server to the media origin were generally around:
CONNECT ≈ 9.5–12.5 ms
TTFB ≈ 19–23 ms
TOTAL ≈ 19–23 ms
I ran sequential and parallel checks.
Results included:
10/10 successful
20/20 sequential successful
20/20 parallel successful
On the media origin itself, the same small local request typically took roughly:
0.5–0.9 ms
That tells me something useful about where the ~20 ms comes from.
The origin isn't spending 20 ms finding the file.
Most of that time is the network path between the two machines.
Which is exactly what I want from a boring static origin.
Important: this does not mean a full video downloads in 20 ms
This distinction is easy to lose when discussing latency numbers.
My:
~20 ms
measurements were small Range/TTFB-style tests.
They measure the cost of reaching the origin and starting to receive useful data.
A:
500 MB
video obviously does not transfer completely in 20 ms.
Full transfer time depends on:
file size
÷
available throughput
The win here was not infinite bandwidth.
The win was making the cold connection path cheap and stable.
Comparing one observed old outlier with the new path
Remember the old measurement:
1.054474 seconds
And the new cold request:
~20 ms
Mathematically:
1054 ms / 20 ms ≈ 52×
That comparison is striking.
But it needs to be stated correctly.
I am not claiming:
My media server is 52× faster than Bunny Storage.
That would be a terrible benchmark.
I am saying:
One real old origin path I measured took about 1.054 seconds, while my current cold origin requests are around 20 ms.
Those are two observed network paths.
The normal old Bunny measurements were much better:
~29–44 ms
The important difference was not merely the best case.
It was the tail.
I cared about eliminating this:
30 ms
31 ms
43 ms
1,054 ms
timeout
and getting something much closer to:
20 ms
20 ms
21 ms
19 ms
22 ms
For perceived web performance, reducing variance can matter more than improving an already-good average.
Something surprising happened: I no longer needed a huge cache to make old pages feel fast
My old media cache had grown to roughly:
35–36 GB
and hundreds of thousands of cache entries.
The new cache did not need to immediately grow anywhere near that size.
At my latest check it was only around:
3.9 GB
with:
max_size = 25 GB
min_free = 8 GB
Yet old pages felt dramatically faster.
That initially sounds backwards.
The old cache was much larger.
Shouldn't it have been better?
The answer is that the two systems optimize different things.
Old system:
HIT = very fast
MISS = unpredictable
New system:
HIT = very fast
MISS = also fast
A larger cache reduces how frequently you miss.
A good origin reduces the cost of every miss.
I had spent a lot of time optimizing the first variable.
The second one turned out to matter more.
MP4 became simpler after the origin moved too
With a private, predictable origin, I can use a much simpler MP4 strategy.
For some MP4 files, I can let a cold request trigger a full-object cache fill:
cold MISS
↓
request full MP4 from private origin
↓
store one complete local cache object
↓
serve browser byte ranges locally
Once the complete MP4 exists locally, Nginx can satisfy normal browser Range requests from that cached file.
Conceptually:
Browser
↓ Range
Working Nginx
↓
complete MP4 in local cache
↓
206 Partial Content
No remote slice assembly.
No arbitrary remote Range fragments becoming separate cache objects.
No need to combine pieces fetched from different storage responses.
One file.
One cache entry.
Normal local byte-range serving.
But full-object caching is not free either
There is a trade-off here.
Suppose the MP4 is:
500 MB
The first viewer watches:
10 seconds
If my cache strategy fills the entire object, the origin may still send all:
500 MB
to the working server.
A pure Range request could have transferred far less.
For my file sizes and access patterns, I accept that trade-off because subsequent seeking and subsequent viewers can reuse one simple local object.
But I would not blindly use this design for every video workload.
For a library of mostly cold, multi-gigabyte videos, I would evaluate things such as:
- stable cache slicing
- HLS
- DASH
- a real video CDN
- Bunny Stream
- another dedicated video platform
Architecture follows workload.
The important change is that I now get to make that decision based on video behavior, instead of also fighting an unpredictable origin path underneath it.
Nginx Slice is not the villain either
I want to be equally careful here.
Nginx Slice is a useful feature.
It exists specifically because caching huge files as fixed byte ranges can be much more efficient than waiting for complete cache fills.
For an immutable file and stable origin:
0–1 MB
1–2 MB
2–3 MB
...
can be an excellent model.
My problem wasn't:
Slice is broken.
My problem was:
My production slice requests encountered ETag inconsistencies across upstream responses.
Now, if I use slicing between my two servers, every slice comes from:
one origin
one Nginx
one filesystem
one file representation
That is a much simpler system to debug.
One URL benchmark is not enough
Another lesson from this migration is how easily origin testing can become unrealistic.
Consider:
curl same-file.avif
curl same-file.avif
curl same-file.avif
curl same-file.avif
After the first request, you're often benchmarking:
cache
not:
origin
Even proxy_cache_lock only helps requests competing for the same cache key.
If a page loads:
image-1.avif
image-2.avif
image-3.avif
...
image-50.avif
and all 50 are cold, those are still 50 different objects.
A realistic media-origin test should therefore include:
- different files
- cold objects
- parallel requests
- small and large objects
- Range requests
- seeking patterns
One cold URL and twenty distinct cold URLs are very different tests.
What I would monitor now
If I were building a similar system again, my Nginx access log would include at least:
$upstream_addr
$upstream_connect_time
$upstream_header_time
$upstream_response_time
$upstream_cache_status
$request_time
$status
Then I would analyze:
HIT latency
MISS latency
MISS p95
MISS p99
upstream connect p95
upstream connect p99
separately.
Do not average HIT and MISS together.
They are different execution paths.
A site can have:
95% extremely fast HITs
+
5% terrible MISSes
and still feel broken whenever a user happens to open the wrong page.
For video, I would test actual seeking behavior
A successful request for:
Range: bytes=0-1048575
does not prove that video delivery is healthy.
I would test at least:
Beginning of the file
Range: bytes=0-1048575
Somewhere in the middle
Range: bytes=50000000-51048575
End of the file
A suffix or final-byte range.
Random seeking
Several non-sequential ranges.
And for each, inspect:
HTTP status
206 Partial Content
Content-Range
Content-Length
Accept-Ranges
ETag
If using Slice, I would explicitly compare validators between multiple slices.
That test would have exposed my MP4 problem much earlier.
Production traffic also showed what a warm cache should look like
One useful sanity check was comparing public traffic from the working server with traffic arriving from the media origin.
In one 60-second production snapshot:
public traffic to users: ~96.84 Mbit/s
traffic from media origin: ~7.07 Mbit/s
During that measurement window there were no new:
502
503
504
upstream timeout
errors.
In another 30-second snapshot:
public TX: 61.36 Mbit/s
origin RX: 1.40 Mbit/s
The byte ratio in that window was approximately:
2.28%
origin traffic relative to public TX.
I would not call the inverse of that number a cache-hit ratio.
That would be incorrect.
Public traffic contains more than media, and a byte ratio is not a request ratio.
But it does demonstrate the architectural behavior I wanted:
users consume a lot of media
↓
working server serves mostly cached bytes
↓
origin only supplies a much smaller cold-fill stream
The media server is acting like an origin, not like a second copy of all public traffic.
Self-hosting did not magically remove infrastructure problems
Moving the media origin to my own server gives me control.
It also gives me responsibility.
Now I have to care about:
- disk health
- backups
- restore procedures
- free space
- filesystem health
- Nginx updates
- firewall rules
- connection limits
- monitoring
- redundancy
A single private media origin is also a single point of failure for uncached objects.
If it disappears:
cached media → may continue working
new MISS → cannot fill
If I need high availability, I need another origin, replication, failover, or another layer entirely.
So this is not:
VPS good, managed storage bad.
It's a trade-off.
For my workload, I preferred operational responsibility over origin unpredictability.
Another team may reasonably make the opposite choice.
When I would still choose Bunny
There are several situations where I would absolutely consider Bunny again.
For example:
- globally distributed users
- managed edge delivery
- managed redundancy
- rapidly increasing storage
- minimal infrastructure administration
- large geographic distance between users and my servers
And in that case, I would evaluate Bunny the way its platform is intended to be used for delivery:
Storage
+
Pull Zone / CDN
rather than assuming my:
Nginx
↓
Storage API
architecture represents Bunny's best delivery model.
It doesn't.
That's why I don't think "Bunny vs VPS" is the right framing.
The more accurate comparison is:
Bunny Storage as my private Nginx origin vs a private server as my Nginx origin.
For that very specific role, my private origin fits my workload better.
The real lesson wasn't about Bunny
Over those two months I tuned:
cache size
cache keys
Range behavior
206 handling
Nginx Slice
1 MB chunks
keepalive
TLS reuse
retry behavior
stale responses
connection timeouts
All of those were legitimate engineering changes.
But the change that improved the system the most was much simpler:
I changed the origin.
That changed the question I now ask whenever I design a cache.
I used to start with:
How do I maximize my cache-hit ratio?
Now I start with:
What happens when the object is not in cache?
If the answer is:
stable origin
predictable TTFB
correct Range behavior
consistent validators
understandable failure modes
then cache is an optimization.
Great.
But if the architecture only feels fast because users usually avoid the cold path, the cache may be hiding the real problem.
A cache should make a good origin cheaper, not make a bad cold path invisible
That is the mental model I kept from this migration.
A good architecture can look like:
HIT
↓
very fast
MISS
↓
still fast enough
Instead of:
HIT
↓
excellent
MISS
↓
please don't happen
Today, I can open old pages containing media that almost certainly wasn't requested recently, and the experience is still fast.
Sometimes the object is cached.
Sometimes it isn't.
From the browser, the difference is often difficult to notice.
That is exactly what I wanted.
The private origin isn't a CDN.
It isn't magic.
It doesn't eliminate operational responsibility.
It's just:
WireGuard
↓
Nginx
↓
SSD
And after two months of tracing connection timeouts, inspecting cold cache misses, experimenting with MP4 byte ranges, and staring at:
etag mismatch in slice response
making the cold path boring turned out to be the biggest performance improvement of all.
