Streaming Master Uploads to Disk: Preventing Memory Exhaustion in Video Pipelines

Master uploads are where video tools stop behaving like ordinary web apps. A profile image can survive a form parser that buffers the whole payload. A high-bitrate ad master cannot. In the AI ad/video platform, the section-swap workflow from Variant Multiplier exposed the failure: uploads over 10 MB returned "Expected a 'video' file field", even though the user had selected the right file.

The root cause was not the field name. The route was buffering multipart data before the application code owned the stream. Once the request crossed the framework limit, the parser truncated or rejected it and the validation layer reported the symptom it could see: no complete video field. PR #64 (fix/upload-body-truncated-at-10mb) made the failure reproducible. PR #66 (perf/stream-upload-to-disk) changed the ingest architecture so bytes went directly to disk instead of through process heap. The same pattern connects to Streaming AI Video to Cloud Storage, but this pass was intentionally local-disk first: remove the memory cliff without forcing the whole upload flow to become a signed-URL product change.

In-memory file buffering scales linearly with upload size. Streaming to disk turns memory consumption into a flat constant.

The 10 MB failure mode

The broken behavior looked like a form-validation bug because the UI saw a clean error string. Underneath, the shape was a transport problem:

• Body parser limit: the request crossed the default body cap before the route handler could validate the multipart boundary.
• Heap pressure: concurrent 50-200 MB uploads created large buffers that competed with render jobs, thumbnails, and FFmpeg subprocesses.
• Misleading error surface: the parser dropped the file part, then the application reported that the video field was missing.
• Retry waste: users retried the same upload and reproduced the same failure, tying up worker slots with no useful artifact.

That last point matters in creative tooling. Upload is the front door to expensive generation. If ingest is flaky, the rest of the reliability stack — Pre-Spend Guards, cost preview, Drive mirroring, splice repair — never gets a chance to help.

Refactoring to disk streaming

The fix was to treat the HTTP body as a stream from the first byte. The route creates a temporary file, converts the Web Stream to a Node stream, pipes chunks to disk, and returns a file handle to the downstream validation path. No Buffer.concat(). No "read the whole upload, then decide." The route becomes a byte transport boundary.

import { createWriteStream } from 'fs';
import { mkdir, rm } from 'fs/promises';
import { tmpdir } from 'os';
import { join } from 'path';
import { Readable } from 'stream';
import { pipeline } from 'stream/promises';

type StreamedUpload = {
  path: string;
  cleanup: () => Promise<void>;
};

export async function streamUploadToTempFile(
  req: Request,
  filename: string,
): Promise<StreamedUpload> {
  if (!req.body) throw new Error('Missing request body stream');

  const uploadDir = join(tmpdir(), 'video-master-uploads');
  await mkdir(uploadDir, { recursive: true });

  const safeName = filename.replace(/[^a-z0-9._-]/gi, '_');
  const path = join(uploadDir, `${Date.now()}-${safeName}`);
  const output = createWriteStream(path, { flags: 'wx' });

  await pipeline(Readable.fromWeb(req.body as any), output);

  return {
    path,
    cleanup: () => rm(path, { force: true }),
  };
}

The important design choice is that cleanup is returned with the artifact handle. The caller that validates, probes, uploads, or fails the master file owns the final cleanup decision. That prevents a common bug in streaming refactors: moving memory pressure out of Node and creating a disk-fill incident instead.

Next.js route gotchas

PR #223 in the sibling video-generation service exposed the second half of the problem: legacy Next.js API-route config keys do not protect App Router handlers the way teams expect during a Next.js 15 migration. A route can look configured while the framework still applies the wrong body behavior. I cover the audit angle in Systematic Cross-Tool Defect Auditing; for upload routes, the rule is simple: the handler must own the raw stream and the route must avoid hidden parser work.

export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;

export async function POST(req: Request) {
  const upload = await streamUploadToTempFile(req, 'master.mp4');

  try {
    await assertVideoProbePasses(upload.path);
    const result = await ingestMasterVideo(upload.path);
    return Response.json(result);
  } finally {
    await upload.cleanup();
  }
}

Reliability PR cluster

Where streaming fits in the video pipeline

Streaming upload is not the whole durability story. The rest of the pipeline still needs ffprobe validation, Drive or object-storage mirroring, job-scoped artifact keys, and retry-safe state transitions. But upload streaming removes the most expensive early failure: losing the request before there is an artifact to reason about.

The same discipline appears elsewhere in the toolchain. Seam-Free Video Splicing treats generated sections as files with explicit boundaries, not blobs hidden inside UI state. Drive-Backed Asset Catalogs separates media storage from runtime configuration. Streaming uploads are another version of that rule: keep large media as addressable artifacts as early as possible.

What I'd do differently

I would move direct browser-to-object-storage uploads earlier once the UX stabilizes. Local disk streaming is the right tactical fix when users are blocked by 10 MB payload failures. Signed URL upload is the strategic shape because it removes the app server from the byte path entirely.

I would also add upload-size chaos tests before launch: 9 MB, 11 MB, 100 MB, aborted connection, duplicate filename, and slow client. The bug here was not subtle once tested at the right payload size. The missing piece was having those payload sizes in CI instead of discovering them through real master videos.

The pattern: stream everything over 5 MB

Any endpoint that accepts video, audio, model artifacts, or high-resolution frames should default to streaming. In-memory buffers are convenient until they become the scaling limit for the whole product. Treat upload bytes as a stream, validate the resulting artifact, and clean up explicitly. That keeps the rest of the video system available for the work users actually care about: generating, comparing, and shipping creative.