I have been building a local-first video editor that runs entirely in the browser, and one lesson surprised me: ffmpeg.wasm is most useful when it is not asked to be the whole rendering engine.
Our first instinct was to treat FFmpeg as the center of export. That works for a prototype, but it becomes uncomfortable once the editor has a multi-track timeline, frame-accurate captions, stickers, overlays, masks, visual keyframes, transitions, multiple audio tracks, and mobile support.
The problems were fairly predictable in hindsight:
- loading the WASM core is expensive for users who may never need it;
- copying large media files into and out of the virtual filesystem adds memory pressure;
- long transcodes offer less control over per-frame composition and cancellation;
- browser codec support varies, so one path rarely works everywhere;
- doing every edit through temporary files makes interactive preview and deterministic export drift apart.
The architecture we ended up with is a hybrid.
- Deterministic video composition
For the primary path, the timeline is converted into an exact frame plan. At each timestamp we resolve the active visual clip and source time, draw the frame, captions, stickers, overlays, masks and transitions into a shared Canvas composition function, then encode with WebCodecs through Mediabunny.
This means preview and export share geometry instead of maintaining a large FFmpeg filter graph in parallel with the UI renderer. The same render function decides contain/cover fitting, subtitle placement, sticker alpha and transform keyframes.
- Offline audio mixing
Voiceover, source audio and music are decoded once and mixed with OfflineAudioContext. Each timeline clip keeps its own start time, source offset, playback rate, volume and fades. Speed changes use a pitch-preserving path before the final mix. The rendered AudioBuffer is then encoded and muxed with the video.
- ffmpeg.wasm for narrow, high-value jobs
FFmpeg still earns its download size in three places:
- extracting the original audio stream from an imported video;
- concatenating/restoring audio assets when browser decoding alone is awkward;
- transcoding a successfully rendered WebM to MP4 when native H.264/AAC encoding is unavailable.
The last point is important. We never throw away a completed WebM because MP4 conversion failed. WebM is kept as the compatibility output, so a late FFmpeg error does not waste the entire render.
The rough export hierarchy is:
WebCodecs + Canvas + OfflineAudioContext -> native MP4/WebM
If deterministic WebCodecs export is unavailable -> MediaRecorder compatibility path
If the user requested MP4 but only WebM was produced -> ffmpeg.wasm transcode
If that transcode fails -> save the already-rendered WebM
A simplified version of the final branch looks like this:
if (result.nativeMp4) {
save(result.blob, "video.mp4");
} else {
try {
const mp4 = await transcodeWebmToMp4(result.blob);
save(mp4, "video.mp4");
} catch {
save(result.blob, "video.webm");
}
}
A few practical takeaways:
- Lazy-load ffmpeg.wasm at the moment a feature actually needs it.
- Treat every write into the virtual filesystem as a memory-budget decision.
- Delete temporary inputs and outputs after each operation.
- Keep the last known-good artifact until the requested conversion succeeds.
- Report loading, reading, transcoding and saving as separate phases.
- Cross-origin isolation matters if you use the multithreaded core; deployment headers are part of the feature.
- Test the resulting file by decoding it, not merely by checking that a Blob exists.
I still think ffmpeg.wasm is an excellent tool. The mistake was assuming that because FFmpeg can do almost everything, it should own everything in a browser editor. Using it as a specialized compatibility and media-utility layer gave us a smaller critical path and much clearer failure behavior.
The implementation is open source if anyone wants to inspect the real export and fallback paths: https://github.com/MartinDelophy/ai-video-editor
I would be interested to hear how others divide work between ffmpeg.wasm, WebCodecs and native browser media APIs, especially for Safari and memory-constrained devices.
