I Fixed a Bug That Lived in FFmpeg for 8 Years.

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

FFmpeg is the decoder underneath most of the internet's video. If you have ever uploaded a clip, watched a stream, or generated a thumbnail, some version of FFmpeg probably touched that file. Buried inside libavcodec is a decoder for RASC, the format used by RemotelyAnywhere's old screen capture tool. It is not a format most people have heard of. It is also not a format anyone checks twice before feeding to a transcoder, which is exactly why bugs in it can sit quietly for years.

This is the story of one that sat there for eight.

Bug Fix or Performance Improvement

I first ran into this bug the way a lot of researchers find things: reading someone else's writeup. A researcher going by bikini had published a proof of concept on GitHub (bikini/exploitarium) for a heap out-of-bounds write in decode_dlta(), the function that applies delta frames in the RASC decoder. Their PoC was solid. It built a 64x1 PAL8 frame, drove the decoder into a 32-bit write that landed one byte before the end of a 64-byte plane, and used the leftover 3 bytes to flip a callback pointer sitting right after it. The payoff was a Calculator popping open on Windows. Fun, clean, and it proved the write was real.

But a 64-byte pop is still a single data point. I wanted to know if the bug was really about that specific frame width, or if the width was incidental to something more structural.

So I sat down with the source. decode_dlta() walks a frame row by row using a cursor called cx, bounded by w * s->bpp (the row width in bytes). The boundary check lives in a macro:

#define NEXT_LINE                        \
    if (cx >= w * s->bpp) {              \
        cx = 0;                          \
        cy--;                            \
        b1 -= s->frame1->linesize[0];    \
        b2 -= s->frame2->linesize[0];    \
    }                                    \
    len--;

Read that closely and the problem jumps out: NEXT_LINE checks cx against the row width, but it only runs after the decoder has already read or written through cx. Four of the DLTA run types (4, 7, 12, and 13) do a 32-bit access before that check ever fires. Run type 7 is the cleanest example:

case 7:
    fill = bytestream2_get_le32(&dc);
    while (len > 0 && cy > 0) {
        AV_WL32(b1 + cx, AV_RL32(b2 + cx));
        AV_WL32(b2 + cx, fill);
        cx += 4;
        NEXT_LINE
    }
    break;

For a PAL8 frame, s->bpp is 1, so the row width in bytes equals the frame width in pixels. If an attacker sets the DLTA region so that cx starts at width - 1, that 4-byte write lands 1 byte inside the row and 3 bytes past it. The row allocation ends exactly where the frame says it should. The write does not.

I named it RowSpill, because that is literally the mechanic: a row-boundary access spilling past the row's own plane allocation.

To answer my original question, I rebuilt the trigger at two geometries under ASAN, an allocation-boundary checker that flags reads and writes past the edge of a heap buffer:

• width=64, height=1: ASAN reports a 64-byte region, first invalid byte at plane+64.
• width=128, height=1: ASAN reports a 128-byte region, first invalid byte at plane+128.

Same 3-byte spill, same offset from the row edge, twice the plane size. The primitive has nothing to do with 64 specifically. It scales with the row, and it will trigger on any PAL8 width of 4 or more as long as the DLTA region's x field lines up with the last byte of the row. That is the piece the original report did not establish, and it is the piece that matters for anyone deciding how seriously to treat the bug.

Here is the actual ASAN output from run type 13 at the 128-byte geometry:

==15203==ERROR: AddressSanitizer: unknown-crash on address 0x5100000007bf
READ of size 4 at 0x5100000007bf thread T0
    #0 0x... in decode_dlta rasc.c:457
    #1 0x... in decode_frame rasc.c:712
    ...
0x5100000007c0 is located 0 bytes after 128-byte region [0x510000000740,0x5100000007c0)
SUMMARY: AddressSanitizer: unknown-crash rasc.c:457 in decode_dlta

"0 bytes after" is ASAN's way of saying the access starts inside the buffer and ends outside it, which is exactly the partial-overlap spill the bug produces.

The path from an untrusted file to this code is direct. RASC is registered against the RASC FourCC in libavformat/riff.c, so any AVI container claiming that tag gets routed straight to decode_dlta(). No special container tricks, no chained bugs, just an AVI file. I built a 1414-byte one and confirmed it triggers the same write through the plain ffmpeg -i rowspill_128x1.avi -f null - command line, no library code required.

For impact, I moved past the Calculator pop and went for something that argues its own severity: I placed a function pointer immediately after the plane, used the OOB write to redirect it with a fully bitstream-controlled 32-bit value (run type 13 reads its fill straight off the wire, so an attacker chooses all 32 bits), and had the hijacked handler read the first lines of /etc/passwd and write a marker file to /tmp/pwned. That is a controlled read and a controlled write from a media file, which is a different conversation than a novelty popup.

This became CVE-2026-58049, CWE-787, rated 8.6 under CVSS 3.1 and 8.8 under CVSS 4.0. My own working estimate going in was 8.1, and the gap is worth being honest about: I scored it assuming user interaction was required to open the file and weighted confidentiality and integrity high based on what the PoC actually demonstrated. The assigned score treats it as requiring no user interaction and leans harder on availability. Both readings are defensible. It is a good reminder that a CVSS vector is an argument, not a measurement, and two careful people can land in different places on the same bug.

Code

The fix is small on purpose. It adds one helper and calls it before every 32-bit access in the four affected run types:

diff --git a/libavcodec/rasc.c b/libavcodec/rasc.c
--- a/libavcodec/rasc.c
+++ b/libavcodec/rasc.c
@@ -320,6 +320,11 @@ static int decode_move(AVCodecContext *avctx,
     return 0;
 }

+static inline int dlta_room(unsigned cx, unsigned w, unsigned bpp, unsigned need)
+{
+    return cx + need <= w * bpp;
+}
+
 #define NEXT_LINE                        \
     if (cx >= w * s->bpp) {              \
         cx = 0;                          \
@@ -418,6 +423,8 @@ static int decode_dlta(AVCodecContext *avctx,
         case 4:
             fill = bytestream2_get_byte(&dc);
             while (len > 0 && cy > 0) {
+                if (!dlta_room(cx, w, s->bpp, 4))
+                    return AVERROR_INVALIDDATA;
                 AV_WL32(b1 + cx, AV_RL32(b2 + cx));
                 AV_WL32(b2 + cx, fill);
                 cx++;
@@ -427,6 +434,8 @@ static int decode_dlta(AVCodecContext *avctx,
         case 7:
             fill = bytestream2_get_le32(&dc);
             while (len > 0 && cy > 0) {
+                if (!dlta_room(cx, w, s->bpp, 4))
+                    return AVERROR_INVALIDDATA;
                 AV_WL32(b1 + cx, AV_RL32(b2 + cx));
                 AV_WL32(b2 + cx, fill);
                 cx += 4;
@@ -443,6 +452,8 @@ static int decode_dlta(AVCodecContext *avctx,
             while (len > 0 && cy > 0) {
                 unsigned v0, v1;

+                if (!dlta_room(cx, w, s->bpp, 4))
+                    return AVERROR_INVALIDDATA;
                 v0 = AV_RL32(b2 + cx);
                 v1 = AV_RL32(b1 + cx);
                 AV_WL32(b2 + cx, v1);
@@ -454,6 +465,8 @@ static int decode_dlta(AVCodecContext *avctx,
         case 13:
             while (len > 0 && cy > 0) {
                 fill = bytestream2_get_le32(&dc);
+                if (!dlta_room(cx, w, s->bpp, 4))
+                    return AVERROR_INVALIDDATA;
                 AV_WL32(b1 + cx, AV_RL32(b2 + cx));
                 AV_WL32(b2 + cx, fill);
                 cx += 4;

Merged upstream, unmodified, as commit 7ae64dba3c, via PR #23628, "avcodec/rasc: Check that 32-bit DLTA accesses stay within the row."

Review did not just wave it through. One of the maintainers, James Almer, asked a fair question during review: the cursor cx starts at zero each row while b1/b2 are already offset by x, so shouldn't cx start at x instead? It is a reasonable thing to double check on a patch touching pointer arithmetic. The answer holds up: b1 and b2 are computed from x up front, and the earlier x + w <= avctx->width check already guarantees the whole row fits, so checking that cx stays under the remaining width is sufficient without re-adding x into the cursor. Small exchange, but it is exactly the kind of scrutiny you want on a security patch, and I would rather have that conversation happen in review than skip it.

Patched behavior: same crafted AVI, ffmpeg -i rowspill_128x1.avi -f null - now exits 69 with "Invalid data found when processing input" instead of silently decoding a corrupted frame.

My Improvements

Three things changed between bikini's report and what shipped:

Scope. I did not stop at confirming the 64-byte case. I built a parametrized variant harness that takes a run type (4, 7, 12, or 13) and a frame width (64 or 128) as arguments, and ran all eight combinations under ASAN:

All eight trigger. That table is the difference between "here is a bug at one width" and "here is a bug class that will keep resurfacing at any width until someone fixes the actual check." It is also what convinced me the fix belonged at the NEXT_LINE boundary itself, not as a special case for one run type.

Impact. Swapping the Calculator pop for a controlled read of /etc/passwd and a controlled write to /tmp/pwned was not about being flashier. It was about giving the CVSS vector something concrete to point at instead of "a window opened."

A stronger impact proof

Bikini's proof redirected a callback to launch Calculator. That is a fine demonstrative primitive. I wanted something that would make a security team flinch.

My proof uses a 128x1 PAL8 frame. Run type 13. And a Bucket struct that places a function pointer just past the plane:

typedef struct {
    uint8_t  plane[FRAME_W];
    void   (*handler)(void);
    uint8_t  palette[1024];
    uint64_t cookie;
} Bucket;

The DLTA write lands at plane[FRAME_W-1..FRAME_W+2]. Byte 0 of the fill value lands in the last in-row plane byte. Bytes 1, 2, and 3 land on the low three bytes of the handler pointer.

The fill value is computed at runtime. Those three bytes become the low three bytes of a target handler address:

fill = (uint32_t)(((target_addr & 0x00ffffffu) << 8) | 0x41u);

After decode, bucket_b->handler points where I want it. The hijacked handler then writes a marker file and reads /etc/passwd:

=== RowSpill PoC ===
Vulnerability: heap OOB write in FFmpeg RASC DLTA decoder
Geometry: 128x1 PAL8 (128-byte plane allocation)
Run type: 13 (per-iteration 32-bit bitstream-controlled fill)
[addr] handler_safe  =0x56248c054c80
[addr] handler_pwned =0x56248c054d8f
[ptr] bucket_a handler =0x56248c054c80
[ptr] bucket_b handler =0x56248c054d8f (expected 0x56248c054d8f)
[ok] RowSpill: handler redirected via RASC DLTA OOB write
[RowSpill] hijacked handler reached via RASC DLTA OOB write
[RowSpill] file write: /tmp/pwned (PWNED marker + vuln name)
[RowSpill] file read: first 5 lines of /etc/passwd:
  | root:x:0:0:root:/root:/bin/bash
  | daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
  | bin:x:2:2:bin:/bin:/usr/sbin/nologin
  | sys:x:3:3:sys:/dev:/usr/sbin/nologin
  | sync:x:4:65534:sync:/bin:/bin/sync

/tmp/pwned after the run:

PWNED
Vulnerability: RowSpill
Component: FFmpeg libavcodec/rasc.c decode_dlta()
Primitive: 32-bit bitstream-controlled heap OOB write
Geometry: 128x1 PAL8 (128-byte plane allocation)
Run type: 13 (per-iteration 32-bit fill from bitstream)

The fix. Bikini's writeup described the shape a fix would need to take. I wrote the actual patch, tested it against both geometries and all four run types, confirmed a legitimate packet at the row boundary (x=124, w=4 on a 128-wide frame, and x=60, w=4 on a 64-wide frame) still decodes correctly under the patched build, and sent it to ffmpeg-security. Michael Niedermayer opened the public PR the next day, and it merged as-is a few hours after review.

Best Use of Google AI

I did the root cause analysis and the fix by hand first, then used the same problem to see how far Gemini 3.1 Pro (High), running inside Google's Antigravity IDE, could get on its own.

The prompt was deliberately narrow: analyze the codebase, focus on decode_dlta(), find the root cause. No hints about run types, no mention of the 128-byte question. It walked the same path I had: found the NEXT_LINE macro, flagged that the boundary check runs after the access instead of before, and identified the same four run types as affected. It then built its own reproduction, validated it statically against the source and dynamically against an ASAN build, and independently landed on the 128-byte scaling case, the exact detail bikini's original report never tested. Static reasoning and a real ASAN run agreeing with each other is a much stronger signal than either one alone, and getting there took about 40 minutes end to end, prompt to a working PoC plus a fix patch.

I still read every line before anything went near ffmpeg-security. That is not optional on a patch touching a decoder this widely deployed, no matter who or what wrote the first draft. But as a way to compress the distance between "I have a bug report" and "I have an independently reproduced, independently root-caused, independently scaled confirmation of it," it earned its place in this workflow. I have used a lot of coding assistants for security work. Gemini 3.1 Pro, pointed at a specific function with a specific question, is the first one that gave me a second opinion worth taking seriously.

Vulnerability: RowSpill, CVE-2026-58049, CWE-787
Component: libavcodec/rasc.c, decode_dlta()
Fix: PR #23628, merged as commit 7ae64dba3c
Original finding credit: bikini (bikini/exploitarium)