We publish a handful of technical articles every day on our company blog, and every article gets two real photographs: a cover and one inline image. The pipeline is boring on purpose. Pick photos, download them with curl, copy them to the server, insert the article, check the page.
One morning every file in the batch downloaded "successfully". curl exited with 0 and the files had the right names and the .jpg extension. Then a quick look at the folder showed something was off.
$ ls -l foto/
-rw-r--r-- 1 user user 29 cover-1.jpg
-rw-r--r-- 1 user user 29 cover-2.jpg
...
$ file foto/*
foto/cover-1.jpg: HTML document, ASCII text
foto/cover-2.jpg: HTML document, ASCII text
Twenty-nine bytes. Not a photo, an error page with a .jpg name. This post covers why it happened and the checks we added so that a file like that can't reach a live page again.
What went wrong: an ID that looks like a file name
We take photos from Unsplash, whose license allows free commercial use. Each photo has a short public ID that appears in its page URL:
https://unsplash.com/photos/4Ya_nyKge5k
The image files themselves live on a CDN, with URLs like this:
https://images.unsplash.com/photo-1542831371-29b0f74f9713?w=1600&h=900&fit=crop
Our script built the CDN URL from the short ID:
curl -s -o cover.jpg "https://images.unsplash.com/photo-${ID}?w=1600&h=900&fit=crop&fm=jpg"
It looked right, and for a while we never noticed, because we had been copying CDN URLs by hand. The first day the script used the short IDs directly, every request returned a 404 with a tiny HTML body. The short public ID (4Ya_nyKge5k) and the storage key (1542831371-29b0f74f9713) are two different identifiers. One is for people and URLs, the other is for the file store. Nothing in the format tells you that.
The fix for the URL itself is to get the real file URL from a source that knows it. With Unsplash's official API, every photo object has a urls block (raw, full, regular...) pointing at the CDN, so you never construct it yourself. (If you use the API, its guidelines also ask you to call the photo's download_location endpoint when someone downloads it.) For one-off manual work, the photo's download link redirects to the real file, and the Location header shows the storage key:
curl -sI "https://unsplash.com/photos/${ID}/download" | grep -i '^location:'
But the URL was only half the bug. The other half was that a 404 turned into a "successful" download. That part has nothing to do with Unsplash.
Why curl said everything was fine
By default curl doesn't treat an HTTP error status as a failure. A 404 or 500 is a complete, valid HTTP response. curl saves the body and exits with 0. With -s there's no progress output either, so there's nothing on screen to look at.
There are three things worth adding to almost every download in a script.
1. Make HTTP errors fail.
curl -fsS -o cover.jpg "$URL" # -f: exit 22 on HTTP >= 400, keep no body
curl --fail-with-body -sS -o cover.jpg "$URL" # curl 7.76+: fail, but keep the body for debugging
-S brings back error messages that -s would hide. With set -e in the script, a 404 now stops the batch.
2. Look at what the server says it sent.
read -r code type < <(curl -sS -o cover.jpg -w '%{http_code} %{content_type}\n' "$URL")
if [ "$code" != 200 ] || [[ "$type" != image/* ]]; then
echo "bad download: $code $type ($URL)" >&2
exit 1
fi
-w prints transfer details after the download. Checking the content type catches the case where the status is 200 but the body is a login page, a consent page or a bot check. That last one is becoming common: a CDN or WAF answers an automated client with a "checking your browser" page and a 200 or 403.
3. Look at the bytes, not the name.
The extension is a label you chose. The first bytes of the file are what it actually is. A JPEG starts with FF D8 FF, a PNG with 89 50 4E 47, WebP with RIFF....WEBP.
file --mime-type -b cover.jpg # image/jpeg ... or text/html
A small validator we now run on every image
The shell checks catch failed downloads. Before publishing we also want to know the image is complete and has the size the page expects. This is the Python version we run over the folder before anything is uploaded:
import sys
from pathlib import Path
from PIL import Image # pip install pillow
EXPECTED = {"-kapak.jpg": (1600, 900), "-1.jpg": (1200, 675)}
MIN_BYTES = 10_000
def check(path: Path) -> list[str]:
problems = []
data = path.read_bytes()
if len(data) < MIN_BYTES:
problems.append(f"too small ({len(data)} bytes)")
if not data.startswith(b"\xff\xd8\xff"):
problems.append(f"not a JPEG (starts with {data[:12]!r})")
return problems
try:
with Image.open(path) as im:
im.verify() # structure check, catches truncated files
with Image.open(path) as im:
im.load() # full decode
size = im.size
except Exception as e:
return problems + [f"cannot decode: {e}"]
for suffix, want in EXPECTED.items():
if path.name.endswith(suffix) and size != want:
problems.append(f"size {size}, expected {want}")
return problems
bad = 0
for p in sorted(Path(sys.argv[1]).glob("*.jpg")):
issues = check(p)
if issues:
bad += 1
print(f"FAIL {p.name}: {'; '.join(issues)}")
print(f"{bad} bad file(s)")
sys.exit(1 if bad else 0)
A few notes from using it:
-
verify()has to be followed by a freshopen(). Afterverify()the image object can't be used for anything else. - The size threshold is crude, but useful. A 1600x900 photo at quality 80 is usually well above 100 KB. Anything under 10 KB is almost always an error page, a placeholder, or a blank image.
- Checking dimensions catches CDN parameters that were ignored. If
fit=cropdoesn't apply, you get the original aspect ratio, and the layout jumps.
Then check the live URL, not the local file
A correct file on disk can still be broken on the page. Since that morning we have hit two more ways for a good image to disappear after upload.
Line endings in file names. A file list produced by a Windows script and consumed by a bash loop carried a trailing \r into every file name. The files were on the server, ls showed them, and every URL returned 404, because the real name was cover.jpg\r. This finds them:
ls | grep -c $'\r' # should print 0
The image goes through more than one URL. Our site serves images through an optimizing loader endpoint that resizes on the fly, not the raw file. The raw file URL can be 200 while the resized version fails. So the post-publish check requests both, and a headless browser loads the article and counts images that didn't decode:
// Playwright, after page.goto(...)
const broken = await page.evaluate(() =>
[...document.querySelectorAll("main img")]
.filter(img => img.complete && img.naturalWidth === 0).length
);
naturalWidth === 0 on a completed image means the browser got something it couldn't decode. That's exactly what a 29-byte HTML file served as image/jpeg looks like.
Takeaways
- An HTTP error is not a curl error. Use
-for--fail-with-body, and-Swith-s. - Check
%{http_code}and%{content_type}for every download in a script. Bot-check pages often come back with a 200. - Trust magic bytes, not extensions:
file --mime-type, or the first three bytes of a JPEG. - Decode the image before publishing it. Pillow's
verify()plusload()catches truncated and fake files. - A public ID in a URL isn't necessarily the storage key. Take file URLs from the API response instead of building them.
- After upload, test the URL the visitor actually loads, including any resizing endpoint, and count undecodable images in a real browser.
None of this is clever. It is the set of checks we wish we had from the first day. We run this pipeline for the blog at Alesta WEB (alestaweb.com). The morning it failed, nothing reached a reader, mostly because a check looked at the bytes and not the exit code.
