Written for the Gemini All Agentic AI Hackathon.
A game stream can last four hours. The moments worth turning into a video might last only a few minutes.
Finding those moments usually means rewatching the recording, marking timestamps, cutting clips, and rebuilding the story manually. Existing AI clipping tools can reduce that work, but they often return disconnected clips or a finished video with little explanation or control.
That is the problem I’m exploring with FrameMill.
FrameMill turns long game streams into editable video projects. It analyzes what happened, uses AI to assemble a first cut, and gives the creator a real Timeline they can inspect, change, and render again.
The idea: AI should make the first cut
I did not want FrameMill to be a black box that decides what a “good highlight” is.
A combat event is a fact. Whether it belongs in an intense montage, a tactical breakdown, or a funny recap depends on the creator’s goal.
That led to the central ownership model:
Processor observes.
Gemini interprets, plans, edits, and reviews.
Server persists and executes capabilities.
Client presents and changes the Timeline.
Renderer derives video from a Timeline revision.
The media processor detects factual gameplay Activities with timestamps, confidence, and source evidence. It does not assign universal highlight scores.
Gemini receives those Activities alongside bounded pieces of source media. It can map the relevant parts of a game, plan an edit, select footage, and construct a revisioned Timeline.
The Timeline—not the generated video—is the canonical result.
How FrameMill works
FrameMill currently supports Call of Duty: Warzone DMZ as its first game.
The local workflow looks like this:
- FFmpeg and libav decode the source footage.
- A deterministic Go processor detects game boundaries and gameplay Activities.
- The processor publishes structured evidence instead of editorial decisions.
- Google ADK agents use Gemini to map the game and plan an edit.
- The Editor agent constructs a validated Timeline with provenance.
- FrameMill renders a real preview from that Timeline.
- The creator can revise the Timeline and request another render.
The backend is written in Go and contains the local server, processing core, and agent runtime. The editing workspace uses Vue, TypeScript, Vite, Tailwind CSS, and Pinia.
The hosted architecture is designed around MediaMTX, Cloud Storage, Eventarc, Cloud Run, and Firestore. A relay can forward the creator’s stream to its publishing destination while recording small, immutable segments for asynchronous analysis.
Processing hours of footage efficiently
One of the hardest technical problems has been processing long recordings without treating the entire video as one giant request.
FrameMill performs sparse checks across the footage, then uses denser 100 ms refinement only near promising regions. Persistent decoder contexts and bounded parallelism avoid repeatedly reopening the same media.
An important lesson was that decoding architecture mattered more than micro-optimizing individual detectors. If every detector independently reads the video, small optimizations elsewhere do not help much.
The better design is to decode once and let multiple detectors consume the same frame evidence.
Making AI editing recoverable
Video analysis, model calls, and rendering can all take longer than a browser session. They can also fail independently.
FrameMill therefore treats recovery as part of the product:
- Media objects have immutable, generation-qualified identities.
- Agent tasks have stable IDs and explicit lifecycle states.
- Duplicate events are handled idempotently.
- Successful child-agent results survive sibling failures.
- Timeline changes use base revisions instead of silent overwrites.
- Rendered videos can be reproduced from a specific Timeline revision.
If the browser closes, accepted work should continue. When the creator returns, the client should reconstruct the latest durable state rather than starting over.
Building with Codex
I used Codex as an engineering collaborator throughout the project.
It helped me inspect unfamiliar code, investigate real gameplay footage, create tests, compare performance measurements, and coordinate changes across isolated Git worktrees.
The most valuable part was not generating code quickly. It was being able to challenge architectural decisions, trace ownership across the system, and verify whether an implementation actually matched the intended product.
I still retained control over the product model, system boundaries, tradeoffs, and final implementation decisions.
Where the project is today
The current MVP proves a real local vertical slice: source analysis, Activity mapping, Timeline validation, and FFmpeg preview rendering.
The browser workspace demonstrates evidence review, Timeline editing, provenance, revision-conflict recovery, and the render workflow. Some of that workspace still uses canonical fixtures while the remaining server-backed Timeline and control endpoints are completed.
The hosted path is also still being integrated end to end. I would rather label those boundaries clearly than present a demo fixture as production infrastructure.
What I learned
The biggest lesson is that AI does not need to own the entire system to be useful.
Deterministic software is better at observing facts, validating contracts, tracking state, and reproducing outputs. Gemini is better used where interpretation and editorial judgment are actually required.
I also learned that editability is part of AI trust.
A creator should be able to see why a clip was chosen, change the decision, and render a different result. The goal is not to remove the creator from editing. It is to remove the hours of searching that happen before the creative work can begin.
FrameMill is my attempt to make that distinction real.
