Building an offline speech-to-text app with Flutter, FFI, whisper.cpp and FFmpeg

Cloud transcription is convenient until you have a recording you'd rather not upload: a lecture, a client call, a voice note. So I built Extracter, an open-source Flutter app that turns any audio or video file into text entirely on the device. No account, no analytics, and no network at all once a model is downloaded.

It runs on Android, iOS and macOS today (Windows and Linux build scripts exist but haven't been tested yet). This post walks through how it works, and the one native bug that taught me more about Dart isolates than any documentation.

The architecture

audio/video file → FFmpeg (decode-only) → 16 kHz mono PCM → whisper.cpp → segments → Flutter UI

The heavy lifting lives in a small FFI plugin, extracter_native, with one C ABI shared by every platform:

• ex_job_start(media, model, language, translate, threads) starts a job on a native worker thread and returns an id.
• ex_job_cancel(id) is idempotent.
• The worker posts events back to Dart: progress, a segment with start and end timestamps, logs, and exactly one DONE as the last event.

Dart wraps that in a repository and a controller. The UI only sees a stream of segments, which is why the transcript appears line by line while a file is still being processed.

FFmpeg: decode anything, ship nothing extra

"Any format" was the requirement, so FFmpeg was the obvious choice. But I didn't want encoders, network protocols or GPL code in the app. The build is decode-only and LGPL, dynamically linked, with flags like:

--disable-programs --disable-encoders --disable-muxers
--disable-network --disable-protocols --enable-protocol=file
--disable-gpl --disable-nonfree --disable-hwaccels
--disable-decoders --enable-decoder=<all ~200 audio decoders>

For video files we simply pick the best audio stream. Playlist-style demuxers (HLS, concat, DASH) are disabled so a file can never make the app open other files or URLs.

Feeding whisper.cpp long audio

Whisper works on 30-second windows internally, but you still have to decide how much audio to hand it at once. Extracter feeds chunks of about three minutes and cuts each one at the quietest point in its last 15 seconds (by RMS energy), so a sentence is rarely split across chunks. Timestamps stay monotonic across chunks, and long silent gaps advance the clock without wasting time transcribing silence.

On Apple devices whisper.cpp uses Metal and Accelerate; elsewhere it uses the optimised CPU path.

The bug: NativeCallable and dying isolates

The first version delivered events with NativeCallable.listener: native code got a C function pointer and called it from the worker thread. It worked perfectly in every test, until I generated store screenshots on Android.

The app crashed with SIGSEGV, and the backtrace pointed at the line where the worker called the callback. Here's what happened:

1. A transcription is running on a native thread.
2. The user presses Back on the main screen. Android finishes the activity, and the Flutter engine shuts down the Dart isolate.
3. The isolate takes the NativeCallable trampoline with it.
4. The worker thread is still running and posts its next event, jumping to memory that no longer exists.

Even worse, when the app was reopened, the new isolate called ex_init again, and the native side refused to replace the callback, so new jobs never reported back.

The fix was to stop calling into Dart and post to a port instead. Dart passes NativeApi.postCObject and a RawReceivePort's native port to ex_init, and the worker posts each event's address as an int64:

ex_cobject message{};
message.type = kDartCObjectInt64;
message.value.as_int64 = reinterpret_cast<intptr_t>(ev);
if (!post(job.port, &message)) ex_event_free(ev);  // dead isolate: native frees it

Posting to the port of a dead isolate just returns false, so nothing crashes. Each job remembers the port of the isolate that started it, and a new ex_init cancels jobs left over from an old isolate. I reproduced the crash with a tiny flutter run target (start a job, SystemNavigator.pop(), relaunch), and after the fix the same process survived three isolates in a row.

Models

Models come from a pinned Hugging Face revision of ggerganov/whisper.cpp and are verified by size and SHA-256 before they're installed. Base (60 MB) is the recommended default and Large v3 Turbo (574 MB) is the most accurate; models that need more memory than the device has are locked in the UI.

The rest of the app

• History is stored in SQLite (Drift) with FTS5 full-text search across all transcripts.
• Export to TXT, SRT and VTT, or copy as clean paragraphs, with or without [00:00:12 → 00:00:18] timestamps.
• Store screenshots and the demo GIF above are generated by integration tests, run with flutter drive on a real emulator with real transcription, then composed with a small Python script.

Try it

• Code (MIT): https://github.com/sb-dor/Extracter
• Android APK: https://github.com/sb-dor/Extracter/releases/latest (Google Play is in review)

Contributions are very welcome, especially Windows and Linux builds and translations. There are a few good first issue tickets to start with. If you have questions about the FFI side, ask in the comments!

This article was written with AI assistance and reviewed by me.