Building Stable Video Portrait Outlines in the Browser with MODNet, SlimSAM, MediaPipe, and Optical Flow

A portrait outline effect looks simple: find a person and draw a line around them. In real video, however, the problem quickly expands into subject identity, fine matting, temporal stability, occlusion, model latency, and memory pressure.

This article walks through the portrait-outline pipeline currently being developed for Timeline Studio, an open-source, local-first video editor that runs directly in the browser. The implementation combines YOLOS tiny, MODNet, MediaPipe, SlimSAM, and OpenCV Farneback optical flow. Source footage stays on the user's device during analysis and preview.

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live editor: https://video-editor.ai-creator.top/

Why not run MODNet on every frame?

The first prototype was straightforward: run portrait matting on every sampled frame, extract the alpha boundary, and render an outline. It worked for still frames, but an editor-quality implementation exposed four problems:

Full inference at 25 or 30 FPS is expensive in a browser.
Independent predictions make hair and clothing edges flicker.
Nearby people, sofas, and cushions can leak into the portrait mask.
Style changes should not require the AI pipeline to run again.

The solution was to split the job into subject locking, high-quality anchor matting, fast subject gating, guarded segmentation recovery, and temporal propagation.

Pipeline overview

Video decoding
  ↓
YOLOS tiny detects and locks one person
  ↓
MODNet produces an alpha inside the person ROI
  ↓
MediaPipe constrains the subject and removes spill
  ↓
SlimSAM repairs difficult anchors only when needed
  ↓
Farneback optical flow propagates alpha between anchors
  ↓
Area, position, and motion quality checks
  ↓
SVG filters render a material-aware outline

Each component has one primary responsibility:

Component	Responsibility
YOLOS tiny	Decide which person is the effect target
MODNet	Produce a soft portrait alpha with fine edges
MediaPipe	Refresh the person region quickly and suppress spill
SlimSAM	Recover difficult segmentation anchors
Farneback optical flow	Propagate accepted alpha masks through time

1. Lock the target person with YOLOS tiny

Portrait matting answers which pixels look like a person, but it does not necessarily answer which person the user wants in a crowded scene.

The pipeline starts with YOLOS tiny and ranks person detections using confidence, box area, body completeness, and the previous tracked location. Once a primary person is selected, the downstream MODNet, MediaPipe, and SlimSAM stages operate around that person's ROI.

This identity-first step is important when the main person is touching a sofa, sitting beside another person, or surrounded by background people.

2. Use MODNet as the alpha-quality baseline

A binary segmentation mask is usually not enough for a polished outline. Hair, clothing, and motion-blurred boundaries contain partially transparent pixels. If those values are collapsed to only zero and one, the rendered outline becomes jagged and unstable.

Timeline Studio runs MODNet on an expanded person ROI rather than the full source frame. On browsers with WebGPU, the worker uses the WebGPU path; otherwise it falls back to quantized WASM.

Processing an ROI lowers the amount of work while preserving more useful resolution around the target person.

3. Use MediaPipe as a fast soft gate

MediaPipe person segmentation is fast and mobile-friendly, but it does not replace MODNet for every fine-edge case. Instead, the pipeline uses it as a fast subject constraint:

refresh the approximate person region between expensive anchors,
remove sofa or background spill from the MODNet alpha,
reduce contamination from nearby people.

The MediaPipe confidence mask is slightly dilated and blurred before it is applied to MODNet:

fusedAlpha[i] = Math.round(modnetAlpha[i] * mediaPipeGate[i] / 255);

A soft gate is deliberate. A hard binary intersection would easily remove hair strands, fingers, and translucent clothing edges. MediaPipe also has a GPU-to-CPU fallback, while the rest of the pipeline continues if MediaPipe is unavailable.

4. Run SlimSAM only as a guarded fallback

SlimSAM is not part of every frame. It is invoked only when an anchor fails quality validation—for example, when MODNet produces an incomplete person, the connected component no longer matches the detector box, or the mask includes a large spill region.

The SlimSAM request includes:

the YOLOS person box,
a positive point inside the selected person,
negative points derived from other detected people,
negative points derived from suspicious MODNet spill.

The current worker pins Xenova/slimsam-77-uniform to a fixed revision. It uses FP16 on WebGPU and quantized WASM when WebGPU is not available.

Depending on the frame, the SlimSAM mask can be used as the selected subject or as a soft gate over MODNet. That preserves MODNet's smooth alpha edges while benefiting from SAM's prompt-guided subject separation.

5. Connect frames with Farneback optical flow

Instead of running a heavy model on every sampled frame, an OpenCV.js worker calculates Farneback dense optical flow and warps the previous alpha toward the current frame.

Optical flow is limited to short temporal windows. The propagated mask is never trusted blindly. The pipeline monitors:

sudden mask-area expansion or collapse,
large position changes relative to tracking history,
abnormal mean motion,
ROI fallback to the full frame,
loss of the selected person.

When a check fails, the propagated alpha is rejected and a new model anchor is scheduled early. This prevents one tracking error from contaminating the rest of the clip.

Current analysis profile

The editor currently calls the portrait pipeline with a profile similar to this:

const analysis = await analyzePersonOutlineVideo({
  src: videoSource,
  duration,
  flowFps: 8,
  anchorFps: 1 / 3.5,
  maxSamples: 360,
  maxDimension: 360,
  onProgress,
  onSample,
});

That means:

effect samples are generated at roughly 8 FPS,
routine model anchors are approximately 3.5 seconds apart,
the analysis frame's longest side is capped near 360 pixels,
one run produces at most 360 samples,
degraded tracking can trigger an earlier anchor.

A 30 FPS source does not need 30 full model executions per second for an editor preview. Preview and export resolve the cached analysis sample for the current source timestamp.

Optimizing more than inference

Model inference is only one part of browser video processing. Seeking, decoding, canvas reads, worker copies, and transparent-image encoding can all become bottlenecks.

The current implementation includes several engineering optimizations:

Prefer sequential WebCodecs decoding.
Fall back to precise video seeking for unsupported codecs.
Warm YOLOS/MODNet, MediaPipe, and the optical-flow worker in parallel.
Transfer RGBA and alpha buffers with transferable ArrayBuffer objects.
Keep initialized models and workers alive for reuse.
Commit each completed sample to the UI instead of waiting for the whole video.

The last point matters for product experience. Users can see the current cutout, outline, timeline position, processed-frame count, and tracking state advance together. The analysis can also be canceled without leaving a hidden task running.

Turning alpha into material-aware outlines

The simplest outside border is:

Outline = Dilate(Alpha) - Alpha

Timeline Studio builds on that ring with an SVG filter pipeline using feMorphology, feComposite, feTurbulence, feDisplacementMap, texture images, feBlend, feDropShadow, and feGaussianBlur.

The current material presets are intentionally different from one another:

crumpled paper,
heavy frost,
layered light halo,
liquid chrome,
impasto paint,
ink bleed.

Users can tune outline color, width, opacity, grain, texture scale, irregularity, relief, diffusion, ring count, ring spacing, and glow radius.

The important architectural detail is that the person alpha is cached separately from the visual style. Changing a paper outline to chrome does not rerun YOLOS, MODNet, or optical flow.

Delivering browser AI models reliably

The first model download is another practical challenge for local browser AI. Timeline Studio mirrors model artifacts across Hugging Face and ModelScope. Chinese and domestic sessions can prefer ModelScope, with Hugging Face as a fallback, while equivalent files share the same cache identity.

ModelScope: https://www.modelscope.cn/models/martindelophy/timeline-studio-onnx-models
Hugging Face: https://huggingface.co/haixin/timeline-studio-onnx-models

Models are downloaded on first use and reused from browser storage afterward.

Run the project locally

Timeline Studio is open source under the MIT License:

git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run dev

Use Node.js 20+ and a modern Chromium-based browser. A WebGPU-capable device improves the MODNet and SlimSAM paths.

For a production build:

npm run build
npm run preview

Current limitations and next steps

The hybrid pipeline improves speed and temporal consistency, but difficult cases remain:

people fully crossing in front of each other,
a target leaving and re-entering the frame,
fast shot changes,
severe motion blur,
low contrast between hair and background,
complex overlap between a person and a held object,
long clips on low-memory mobile devices.

Next steps include shot-boundary detection, stronger person-ID continuity, manual subject selection, a larger repeatable test set, and extending the same effect system to object outlines.

The portrait-outline workspace is still under active development, so the repository README, releases, and roadmap remain the source of truth for current availability.