How to build a video editor in the browser: the complete architecture

If you want to learn how to build a video editor in the browser, start with one idea: the editor is a document plus a renderer. The document is a plain data structure that says which clips sit where on a timeline and which effects they wear. The renderer reads that document and draws pixels, frame by frame, onto a web page. Everything else, including the undo button, the autosave and the export file, is a set of helpers around those two parts.

This guide is the map for a whole series on browser video editing. It names each layer of a web based video editor, explains it from the bottom up, and links to a deeper article for every layer. The examples come from the Azelify editor, a browser-native non-linear editor that we build, but the general technique does not depend on our code. Where we describe our own design, we describe the decision we made and the trade-off it carried.

What is a browser video editor?#

A browser video editor is a web application that lets someone import video files, arrange them, change them and export a new video file, all inside one browser tab. Nothing is installed on the computer. The work happens in JavaScript, with help from the media and graphics APIs that every modern browser already ships.

Most professional editors are non-linear. Wikipedia describes non-linear editing as a form of offline editing in which dedicated software specifies and alters the edits, while the original content is not modified.^[1] "Non-linear" means you can reach any moment of any clip at once, without copying footage into one fixed sequence. Your cuts are instructions. The source file never changes.

Four terms carry most of the weight in this series, so define them now:

• A codec is a method for compressing video into fewer bytes, such as H.264 or VP9. Encoding means compressing; decoding means turning the compressed bytes back into pictures.
• A container is the file wrapper, such as MP4 or WebM. It holds the compressed streams and the timing that says when each picture should appear. A codec and a container are different things, and one container can hold more than one codec.
• A frame is one still picture in the sequence. A clip at 30 frames per second shows 30 of them every second.
• A keyframe in video compression is a frame that can be decoded without the frames around it. Most other frames store only the difference from earlier pictures. This is different from the animation keyframes covered in the keyframe animation guide.

Here is the whole pipeline in one picture. Read it left to right for playback. Export reuses the same middle steps, which is the most important property of the design.

Fig. 01

1. 01 → ImportFile to media record
2. 02 → DecodeBytes to frames
3. 03 → ComposeDocument picks frames
4. 04 → DisplayCanvas shows the frame
5. 05 → EncodeSame frames to a file

Illustrative, not data[mdn-webcodecs][nle]

How to build a video editor in the browser, layer by layer#

Build from the bottom up, because each layer assumes the one beneath it is correct. You cannot compose frames you have not decoded, and you cannot decode a file you have not admitted. The list below follows the order you should build in. Each heading links to the article that goes deeper.

Layer 1: the project document#

The project document is a plain JavaScript object that describes the whole edit: the settings such as width, height and frame rate, the media library, and the timeline. Because it is plain data, you can save it, compare two versions and send it over a network. The Azelify editor's document is serializable for the same reason. The project file design guide covers the schema, versioning and autosave rules.

Layer 2: commands and undo#

Every change to the document should go through a command. A command knows how to apply an edit and how to reverse it, and a history stack keeps the reversals in order. Without that discipline, undo becomes guesswork and a half-finished drag can leave the document in a state nobody meant to create. The undo and redo guide builds the stack from zero.

Layer 3: the timeline#

The timeline is the data model that places clips on tracks. A clip has two ranges: where it sits on the timeline, and which part of its source file it plays. Trims, ripples and overlap rules all operate on those ranges. The timeline data model article covers them in detail. The figure below shows two clips on one track.

Fig. 02

Clip A0–4s

Clip B4–12s

0s5s10s

4s · Cut

1. 1Clip A 0–4sin 2s · out 6s
2. 2Clip B 4–12s

Illustrative, not data

Notice that Clip A's source starts at two seconds while its timeline position starts at zero. Those are two different numbers, and the difference between them is the whole point of a trim.

Layer 4: media import#

Import turns a file the user picked into a media record. The first steps are admission, which decides whether the file can be used at all, and probing, which works out what the file contains before any heavy work begins. The import guide explains how to probe a file, and why phone footage needs extra care.

Layer 5: decoding#

Decoding turns compressed bytes into frames. Browsers offer two routes. The built-in video element is simple but hides its frames from you. The WebCodecs API hands you each frame as an object you can draw, hold or release. The WebCodecs decoding guide builds a full decoder.

Layer 6: the playback clock#

A playback clock decides which frame should be on screen right now. It needs one master time source, usually the audio clock, so that picture and sound stay aligned. The frame-accurate playback guide covers the sixty-hertz frame budget and the moments when frames must be dropped.

Layer 7: compositing#

Compositing means drawing several layers into one picture: a background, a video clip, a title, a blend mode, an effect. The 2D canvas handles the simple cases, and WebGL handles heavier per-pixel work. The canvas versus WebGL guide compares them. Motion built from keyframes feeds this layer the transforms it draws, as the keyframe animation guide explains.

Layer 8: text and captions#

Text is drawn into the picture, not typed onto the video. The canvas does not wrap words for you, so a caption needs its own line-breaking code, fonts that have finished loading, and a safe area inside the frame. The text and captions guide covers those steps.

Layer 9: transform handles#

Handles let a person drag, resize and rotate a clip on the preview. They need coordinate conversions between the screen, CSS pixels and the canvas grid, and they must respect the device pixel ratio, which is the number of physical screen pixels behind each CSS pixel. The transform handles guide works through that geometry.

Layer 10: audio#

The audio layer mixes every audio clip, applies gain and fades, and produces the waveform shown on the timeline. The Web Audio API builds this from nodes linked into a graph. The Web Audio guide explains the graph and offline rendering.

Layer 11: export#

Export runs the composition and audio steps again, without the playback clock, and feeds the results to an encoder and a muxer. The muxer writes the container. The MP4 export guide shows the pipeline end to end.

Layer 12: proving the picture#

Testing a renderer is hard because the output is pixels. Golden images, property-based tests and independent oracles let you check the picture without trusting a single screenshot. The renderer testing guide shows how to set these up.

The rule that keeps preview and export honest#

If you remember one principle from this guide, make it this: the preview and the export should draw through the same function. When they use two code paths, the picture a viewer approves on screen and the file you deliver will drift apart, and the resulting bugs are hard to reproduce.

Here is how the drift happens. Suppose preview draws a caption with the browser's text engine, while export uses a separate routine that wraps lines differently. A creator sees a tidy two-line caption on screen and then receives a file where the words run off the edge of the frame. Nothing crashed. The two code paths simply disagree, and nobody wrote a test that compared them.

In our editor, preview and export call one rendering path with different settings for resolution and quality. The trade-off is that the render function must be pure enough to run in two contexts: a live browser tab during editing, and an offline pass during export. We accepted that discipline early, because writing the drawing code once is cheaper than keeping two renderers in agreement for years.

Shared drawing code also changes how you test. If one function draws both outputs, a test of that function covers both. The renderer testing guide returns to this idea.

A minimal end-to-end sketch: file to canvas#

The smallest useful editor does three things: it takes a file, it shows one frame, and it draws that frame. The sketch below uses the browser's built-in video element for decoding, because it needs no extra library. Read it top to bottom. The comments explain each step.

// 1. Find a canvas and get its 2D drawing context.
const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");

// 2. Turn a File (from an <input type="file"> element) into a URL
//    that a media element can read.
const video = document.createElement("video");
video.src = URL.createObjectURL(file);
video.muted = true; // browsers are stricter about autoplay with sound

// 3. Wait until the browser knows the duration and the first frame.
await new Promise((resolve) =>
  video.addEventListener("loadeddata", resolve, { once: true })
);

// 4. Seek to 2.5 seconds, then wait for the decoder to reach that frame.
video.currentTime = 2.5;
await new Promise((resolve) =>
  video.addEventListener("seeked", resolve, { once: true })
);

// 5. Draw the current frame so it fills the whole canvas.
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

// 6. Release the temporary URL once the file is no longer needed.
URL.revokeObjectURL(video.src);

Each step teaches one idea. Step 2 uses URL.createObjectURL, the standard way to let a media element read a local file. Step 4 is the first real lesson in editing: seeking is asynchronous. Setting currentTime starts the work, and the seeked event tells you the decoder has landed on a picture. Step 5 is compositing in its simplest form: one image drawn into one rectangle.

The video element is not frame-accurate. The browser decides which picture to show and when, which is fine for watching but not enough for an editor that must know exactly which picture it drew. That is why the WebCodecs guide replaces steps 2 to 5 with a decoder that hands over each frame as an object.

Which decoding path should you choose?#

Choosing how to decode is a small decision tree. Start from the question you must answer: do you need to know exactly which frame you are holding, or is the browser's own playback good enough?

Fig. 03

1. Need exact frame access?
   1. yes → Is the codec supported?
      1. yes → WebCodecs decoder
      2. no → Convert first, then decode
   2. no → Video element and canvas

Illustrative, not data[mdn-webcodecs]

Each outcome has a cost. The video element is the least work. WebCodecs gives exact frames but expects you to split the container into chunks yourself, or to use a library that does. Converting a file first adds time and storage before the user sees anything. MDN describes the WebCodecs API as the way to turn EncodedVideoChunk objects into VideoFrame objects and back again.^[2] The VideoDecoder interface is the piece that performs the decoding.^[3]

MDN also says decoder support is broad, while hardware encoder support is still limited.^[2] That asymmetry matters for export planning, which is why the export guide returns to encoding. Chrome's introduction describes WebCodecs as "a new, low level way to access built in audio and video codecs," which is the right mental model: you get the codec, and you do the rest.^[4]

Browser support at a glance#

A browser video editor depends on several APIs at once. Each one has a job, and each one has its own support story. Check the live tables before you commit, because support changes between browser versions and this article is a snapshot.^[5]

MDN describes OffscreenCanvas as a canvas that can be rendered off screen, and it notes that the feature is available in Web Workers.^[6] web.dev explains the reason to bother: heavy pixel work moved into a worker does not freeze the page while it runs.^[7]

The Web Audio API works with an audio context that holds a graph. Sources feed nodes, the nodes pass sound along links, and the chain ends at a destination such as the speakers.^[8] The origin private file system is private to the page's origin and is not visible to the user. It sits under the same storage quotas as other browser storage, so a large project can run into a limit that you must plan for.^[9][10]

Build or buy? Libraries you can start from#

Not every layer needs to be written from scratch. Three open-source projects show the range.

• mediabunny describes itself as a TypeScript toolkit for reading, writing and converting media in the browser. Its README says it reads and writes formats such as MP4, MOV, WebM and MKV, covers more than 25 codecs through WebCodecs, and has zero dependencies.^[11]
• ffmpeg.wasm is a port of FFmpeg compiled to WebAssembly, so a large part of FFmpeg's command set can run inside a tab. Its JavaScript wrapper is MIT-licensed.^[12]
• Remotion makes videos programmatically with React. It is a code-first renderer, not an editor with a draggable timeline.^[13]

A useful rule is to buy the commodity parts, such as container parsing, and build the parts that define your product, such as the timeline and the renderer. WebCodecs gives you encoded chunks and decoded frames, but it does not write a finished file. The muxer, which packs encoded chunks into a container, is a separate job. mediabunny's README says its multiplexers and demultiplexers are connected through abstractions around the WebCodecs API, which is one clear picture of that split.^[11] The WebCodecs, ffmpeg.wasm and MediaRecorder comparison sets out the trade-offs for each route.

Build order: how to build a video editor in the browser without rework#

The order below is the order we recommend. Each step produces something you can run before the next step starts, so you never have a half-built pipeline that only works on paper.

Step 1: write the document#

Describe one project as a plain object with settings, a media list and a timeline with one track. Write a function that saves it and loads it back, and check that the round trip is identical. Nothing else needs to exist yet.

Step 2: add commands and undo#

Write one command, such as "add a clip", that records how to reverse itself. Then add a history with an undo stack and a redo stack. Doing this before the picture means every later feature inherits a history it can trust.

Step 3: draw one frame#

Use the sketch from earlier to draw a single frame from a single file onto a canvas. Keep the drawing in one function, so that you can later reuse it for export.

Step 4: add decoding and playback#

Replace the video element with a decoder once you need exact frames. Add a playback clock with one master time source. Keep the playback loop separate from the drawing function, so the same drawing code serves both.

Step 5: add the timeline and export#

Place several clips on tracks, add trims and then an export path that runs the same drawing code without a clock. Export is last only in the build order; in your tests, it should be checked from the start.

Mistakes to avoid while you build#

Starting with the interface. A timeline that looks right is not a data model. Build the document and the commands first, then draw controls over them.

Loading whole files into memory. Video files are large. Read the metadata first, then decode only the frames you need.

Trusting the video element for exact frames. It is fine for watching and wrong for any editing decision that depends on an exact frame.

Planning export last. Export touches every other layer. If the renderer is not pure, you will discover that the first time someone asks for a file.

How the Azelify editor handles the whole stack#

The Azelify editor is built the way this guide describes, and it has a few deliberate choices you can copy or reject.

The interactive path runs entirely in the viewer's browser tab. There is no server-side rendering while someone edits, so every preview picture is produced on the viewer's own device. The benefit is instant feedback and no render queue. The cost is that a weaker phone gets smaller previews and fewer heavy features, and the interface says so plainly rather than hiding it.

The project is one plain document. The renderer, the autosave loop and the export all read that one object. The cost is that the document has to stay small enough to save and reopen on every change, so the editor sets a size limit and refuses documents above it.

Undo is built on commands with inverses. Each edit is refused unless it can be reversed, which keeps the history trustworthy. The cost is that a new kind of edit needs its inverse before it can ship.

Browser storage is split into data that can be rebuilt and data that exists nowhere else. Cached media may be evicted under storage pressure, while recovery drafts are never evicted. The cost is careful bookkeeping about which store owns which data.

We chose these trade-offs because a browser editor fails in quiet ways. A lost draft or a preview that silently disagrees with the export damages trust faster than any slow feature would.

Frequently asked questions#

Can I build a video editor in JavaScript without a server?#

Yes, for the editing path. The browser can decode, draw, mix and encode media through built-in APIs, so the interactive editor needs no server. A server can still help with heavy jobs, such as exports on devices that cannot keep up, but it is optional for the core experience.

Is WebCodecs supported in every browser?#

No. Support differs by browser and by version, so check caniuse and MDN before you rely on it. MDN notes that decoder support is broader than hardware encoder support, so plan a fallback path for both.

Do I need FFmpeg to build a browser video editor?#

No, not for the core editing loop. ffmpeg.wasm is one option for conversion and for tasks that WebCodecs does not cover. It is a larger download, and you must check the licence of the compiled core before you ship it.

Why does the preview look different from the export?#

Usually because two code paths draw the same frame. Route both through one rendering function, then compare their output with golden images, as the renderer testing guide describes.

Should I build the timeline or the renderer first?#

Build the document and its commands first. The renderer needs a document to read, and the timeline needs commands to edit. A minimal renderer that draws one frame is a sensible first visible milestone.

Where to go next#

The project file format guide is the best next read if you want to design the document, and the WebCodecs decoding guide is the best next read if you want code that decodes frames.

Sources

1. Wikipedia, “Non-linear editing,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

2. MDN Web Docs, “WebCodecs API,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2]

3. MDN Web Docs, “VideoDecoder,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

4. Chrome for Developers, “New in Chrome 94,” developer.chrome.com (opens in a new tab) (accessed 2026-10-13). ↩

5. Can I use, “WebCodecs API,” caniuse.com (opens in a new tab) (accessed 2026-10-13). ↩

6. MDN Web Docs, “OffscreenCanvas,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

7. web.dev, “OffscreenCanvas—speed up your canvas operations with a web worker,” web.dev (opens in a new tab) (accessed 2026-10-13). ↩

8. MDN Web Docs, “Basic concepts behind Web Audio API,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

9. MDN Web Docs, “Origin private file system,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

10. MDN Web Docs, “Storage quotas and eviction criteria,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

11. Vanilagy, “Vanilagy/mediabunny,” GitHub, github.com (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2]

12. ffmpegwasm, “ffmpegwasm/ffmpeg.wasm,” GitHub, github.com (opens in a new tab) (accessed 2026-10-13). ↩

13. Remotion, “remotion-dev/remotion,” GitHub, github.com (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2]