A contact sheet is the quickest way to look through a night of camera clips: one image per clip, a grid of small frames, and you scan the grid instead of scrubbing each video. ffmpeg can build one in a single line, and the two lines people usually reach for are fps=1/10,tile=... and thumbnail,tile=.... I built both against a test clip where something appears for three seconds, and neither sheet showed it. The thumbnail filter did not miss it by bad luck. Given a batch of frames, it is built to pick the one that looks most like the rest of the batch, and in a camera clip that is the empty yard.
Disclosure: I make an Android camera app. Nothing here uses its code or describes how it records or names files. Everything below runs on a computer with ffmpeg 4.4.2 (the Ubuntu 22.04 package), bash and Python 3 with Pillow.
What the tests showed:
- One frame every 10 seconds missed a 3-second event. That part is expected: the samples fell either side of it.
-
The
thumbnailfilter missed it too, even though the event filled 31 of the 100 frames in its batch. It only picked the event when the event filled 51 of 100. -
The frame
fps=1/10puts in the first cell is not the frame at 0 seconds. It is the frame at 4.9 seconds, so each cell shows a moment about five seconds later than the 0, 10, 20 you would write under it. - One frame per second caught the event in three cells, and a 0.6-second event in six of ten trials, which is the arithmetic you would expect.
The test clip
To know what a sheet should show, the clip has to be made with a known event in it. This is 60 seconds of grey, slightly noisy frames at 10 frames per second, with a red square standing in the middle from 31.0 to 34.0 seconds:
# 60 s at 10 fps of a grey, slightly noisy "empty yard",
# with a red square standing in the frame from 31.0 s to 34.0 s.
make_clip() { # make_clip OUT START END
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "color=c=0x606060:s=640x360:r=10:d=60,noise=alls=12:allf=t" \
-vf "drawbox=x=280:y=140:w=80:h=80:color=red:t=fill:enable='between(t,$2,$3)'" \
-c:v libx264 -pix_fmt yuv420p -g 20 "$1"
}
make_clip clip.mp4 31 34
A real clip has a moving tree and changing light, not a flat grey field, so this is the easy case. If a method misses a solid red square on a still background, it will not do better on a real yard.
To check a sheet without squinting at it, this script cuts the sheet back into cells and marks each cell that contains the red square:
# red.py SHEET COLS ROWS -> one mark per cell: RED if the square is in it
import sys
from PIL import Image
im = Image.open(sys.argv[1]).convert("RGB")
cols, rows = int(sys.argv[2]), int(sys.argv[3])
cw, ch = im.width // cols, im.height // rows
marks = []
for r in range(rows):
for c in range(cols):
cell = im.crop((c * cw, r * ch, (c + 1) * cw, (r + 1) * ch))
n = sum(1 for (R, G, B) in cell.getdata() if R > 180 and G < 90 and B < 90)
marks.append("RED" if n > 50 else ".")
print(" ".join(marks))
Three sheets of the same clip
# Sheet 1: one frame every 10 s, six cells.
ffmpeg -hide_banner -loglevel error -y -i clip.mp4 \
-vf "fps=1/10,scale=320:-1,tile=3x2" -frames:v 1 every10.png
echo "fps=1/10 3x2 : $(python3 -W ignore red.py every10.png 3 2)"
# Sheet 2: the thumbnail filter, one pick per 100 frames (10 s), six cells.
ffmpeg -hide_banner -loglevel error -y -i clip.mp4 \
-vf "thumbnail=100,scale=320:-1,tile=3x2" -frames:v 1 thumb.png
echo "thumbnail 3x2 : $(python3 -W ignore red.py thumb.png 3 2)"
# Sheet 3: one frame every second, sixty cells.
ffmpeg -hide_banner -loglevel error -y -i clip.mp4 \
-vf "fps=1,scale=160:-1,tile=10x6" -frames:v 1 every1.png
echo "fps=1 10x6 : $(python3 -W ignore red.py every1.png 10 6)"
fps=1/10 3x2 : . . . . . .
thumbnail 3x2 : . . . . . .
fps=1 10x6 : . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RED RED RED . . . . . . . . . . . . . . . . . . . . . . . . . .
The every-10-seconds sheet has six cells and none of them contains the square. The thumbnail sheet has six cells and none of them contains it either. The one-frame-per-second sheet shows it in three cells, which are the 32nd, 33rd and 34th; why it is those three and not 31 to 33 comes down to which frame fills each cell, covered further down.
What the thumbnail filter picks
The ffmpeg filter documentation describes thumbnail in one sentence: "Select the most representative frame in a given sequence of consecutive frames." Its n option sets the batch, and the documented default is 100. With a 10 fps clip, that is one pick per 10 seconds, so six picks for this clip.
The filter logs each pick. Here are the six:
ffmpeg -hide_banner -i clip.mp4 -vf "thumbnail=100" -f null - 2>&1 \
| grep -o "frame id #[0-9]* (pts_time=[0-9.]*) selected from a set of [0-9]* images"
frame id #91 (pts_time=9.100000) selected from a set of 100 images
frame id #47 (pts_time=14.700000) selected from a set of 100 images
frame id #89 (pts_time=28.900000) selected from a set of 100 images
frame id #6 (pts_time=30.600000) selected from a set of 100 images
frame id #54 (pts_time=45.400000) selected from a set of 100 images
frame id #7 (pts_time=50.700000) selected from a set of 100 images
The fourth batch covers 30.0 to 39.9 seconds, the batch the square is in, and the pick was the frame at 30.6 seconds, before the square appears.
Representative has a specific meaning in the source. In the 2.7 source listing, each frame gets an "RGB color distribution histogram", the filter builds the "average histogram of the N frames", and then the code comment says it will "find the frame closer to the average using the sum of squared errors". The comment at the top of the file states the purpose: "to reduce the risk of an inappropriate selection (such as a black frame)". That listing is old, and I ran 4.4.2, so I am not claiming the code is identical; what I checked is that the 4.4.2 behaviour matches that description.
So the filter is doing what it was designed for. For a film's cover image, the frame closest to the average avoids a black frame or a flash. In a camera clip, the frame closest to the average is the scene with nothing in it, and an event is, by definition, the frames that are not average.
To see where it tips, I made the square stay longer:
# make_clip is the function from the test clip above
for end in 34 36 39; do
make_clip m.mp4 31 $end
ffmpeg -hide_banner -loglevel error -y -i m.mp4 \
-vf "thumbnail=100,scale=320:-1,tile=3x2" -frames:v 1 m.png
echo "square on screen 31-$end s : $(python3 -W ignore red.py m.png 3 2)"
done
square on screen 31-34 s : . . . . . .
square on screen 31-36 s : . . . RED . .
square on screen 31-39 s : . . . RED . .
From 31 to 34 seconds the square fills 31 of the 100 frames in that batch, and the pick is empty. From 31 to 36 it fills 51 of 100, and the pick shows it; at 31 to 39 it fills 81. In this test the event had to be most of the batch before it became the representative frame. Real footage will move the tipping point, but not the direction: the filter favours whatever the batch mostly looks like.
Which frame fills a cell
The second surprise was in the plain fps=1/10 sheet. To see exactly which source frame lands in each cell, I made a clip where every frame writes its own frame number into its pixels, then read the numbers back out of what each filter kept:
# A clip whose every frame carries its own frame number in its pixels:
# left half = n mod 256, right half = 80 * (n div 256). 600 frames, 10 fps.
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "color=c=black:s=64x64:r=10:d=60,format=gray,geq=lum='if(lt(X\,32)\,mod(N\,256)\,floor(N/256)*80)'" \
-c:v ffv1 counter.mkv
for vf in "fps=1/10" "fps=1/10:round=up" "select='not(mod(n\,100))'"; do
ffmpeg -hide_banner -loglevel error -i counter.mkv -vf "$vf" -vsync 0 \
-f rawvideo -pix_fmt gray - | python3 -c '
import sys
d = sys.stdin.buffer.read(); px = 64 * 64
frames = [d[i*px + 645] + 256 * round(d[i*px + 690] / 80) for i in range(len(d) // px)]
print(sys.argv[1].ljust(26), "source frames:", frames)' "$vf"
done
fps=1/10 source frames: [49, 149, 249, 349, 449, 549]
fps=1/10:round=up source frames: [0, 100, 200, 300, 400, 500]
select='not(mod(n\,100))' source frames: [0, 100, 200, 300, 400, 500]
At 10 frames per second, frame 49 is 4.9 seconds in and frame 349 is 34.9 seconds in. So with the default settings the cell you would label 30 seconds holds a frame from 34.9 seconds. That is consistent with the documented round option of the fps filter, whose default in ffmpeg -h filter=fps is near: frames are rounded to the nearest output slot, and the one that ends up in each slot comes from late in its window. round=up, or selecting frames by number with select, gives frames 0, 100, 200 and so on, so cell N shows the moment you expect.
It also explains the one-frame-per-second sheet above: with the default rounding, the cell for second 31 holds the frame from 31.4 seconds, and the cell for second 33 holds the frame from 33.4. The square shows in the cells for 31, 32 and 33, which are the 32nd to 34th cells because the grid starts at second 0.
This does not change how many events a sheet catches, since the spacing is still 10 seconds. It changes where you go looking: open the clip at 30 seconds because that cell looked interesting, and what you saw is about five seconds later.
How short an event a sheet can show
A sheet with one frame every T seconds is a sample. If the event lasts at least T, a sample lands inside it; if it is shorter, it depends on where it falls. To check, I moved a 0.6-second square through ten positions inside one second, using the same clip recipe, and built a one-frame-per-second sheet each time:
event 31.0-31.6 s : cells showing it = 1
event 31.1-31.7 s : cells showing it = 1
event 31.2-31.8 s : cells showing it = 1
event 31.3-31.9 s : cells showing it = 1
event 31.4-32.0 s : cells showing it = 1
event 31.5-32.1 s : cells showing it = 0
event 31.6-32.2 s : cells showing it = 0
event 31.7-32.3 s : cells showing it = 0
event 31.8-32.4 s : cells showing it = 0
event 31.9-32.5 s : cells showing it = 1
Six of ten, which is 0.6 seconds of event out of every 1 second of spacing. A person walking quickly across a narrow frame can be on screen for well under a second, so the spacing has to be chosen from the shortest thing you care about, not from how many cells you want.
A sheet I would use for a night of clips
With that in mind, this is the version I ended up with. One frame every 2 seconds, rounded up so cell N is second 2N, 30 cells per sheet, so each sheet is exactly one minute, and longer clips simply produce more sheets:
# One frame every 2 s, 30 cells per sheet (one minute), one or more sheets per clip.
mkdir -p sheets
for f in clips/*.mp4; do
name=$(basename "$f" .mp4)
ffmpeg -hide_banner -loglevel error -y -i "$f" \
-vf "fps=1/2:round=up,scale=240:-1,tile=6x5" "sheets/${name}_%02d.jpg"
done
ls sheets
2026-10-16_2301_01.jpg
2026-10-16_2314_01.jpg
2026-10-16_2314_02.jpg
2026-10-16_2314_03.jpg
I ran it on two test clips: the 60-second one with the square, which gave one sheet with the square in cells 17 and 18 (seconds 32 and 34), and a 150-second blank clip, which gave three sheets, the last one half filled and padded with black cells.
A few notes on the choices:
- 2 seconds is a guess for what I care about, not a rule. Anything on screen for 2 seconds or more is certain to get a cell; anything shorter may not get one. Pick the number from the event, then accept the sheet count.
-
One minute per sheet makes the labels trivial. Cell 17 of sheet 01 is second 32, every time, because of
round=up. - Looking for change is a different tool. If you want the frames where something happened rather than an even sample, start from the scene-change score, which I compared with two other filters in an earlier test on a still camera's view.
Things I did not test
-
Real footage. The event here is a solid colour on a flat background. Real noise and lighting will move the
thumbnailtipping point; I would not trust the 51-of-100 figure outside this test. -
Other ffmpeg versions. Results are from 4.4.2. The current documentation lists a
logoption forthumbnailthat 4.4.2 does not have, so newer builds may differ elsewhere too. - Variable frame rate. Phone recordings often are. My clips are a fixed 10 fps; on a variable-rate clip a batch of 100 frames is not a fixed 10 seconds.
If you want a phone that records in the background so there is a folder of clips to make sheets from, the app I make is Background Camera RemoteStream on Google Play. The commands above do not depend on it.
The frame that looks like all the others is the one a camera clip has plenty of.
