WebCodecs tutorial: decoding video frames in the browser without a video tag

This webcodecs tutorial shows how to decode video frames in the browser with the VideoDecoder interface, without putting a video element on the page. WebCodecs is the browser API that gives JavaScript direct access to the browser's own encoders and decoders, and it hands you each decoded frame as an object you can draw, measure or keep.

The tutorial starts from the bottom. It defines the terms you need (codec, container, keyframe, chunk and frame), shows the steps for turning a file into decoded pictures, and explains why each step exists. We have built a decoder path for our own browser editor, so the later sections describe the decisions we made and what they cost. Everything here is a general technique first, with our own editor as one worked example. For where decoding sits among the other parts, see the browser video editor architecture overview.

What is WebCodecs, and what does it leave out?#

WebCodecs is a set of interfaces for encoding and decoding audio and video. The decoding side has a VideoDecoder that takes encoded chunks and produces video frames. An encoded chunk is one compressed picture, stored as bytes. A video frame is one uncompressed picture, with pixels you can read. The decoder is the bridge between the two.^[1]

The API is deliberately narrow. It does not read files. The MDN documentation states that WebCodecs does not provide a built-in way to read encoded chunks from a video file, and that you need a demuxing library for that job.^[1] A demuxer splits the container into its separate streams and produces the chunks. The import article in this series covers how to probe a file before you decode it, and the mp4box.js library is one choice for the demuxing step.^[2] Another is Mediabunny, a JavaScript toolkit for reading and writing media files in the browser.^[3]

Why bother with an API that leaves the file reading to you? Because the two halves have different costs and different jobs. The chunks are small: MDN notes that an encoded chunk typically stores ten to a hundred times less data than the raw frame it represents.^[1] The frames are large, and holding many of them at once can exhaust memory. Keeping those two concerns separate lets you choose how much of the file to read and how many frames to keep.

Fig. 01

1. 01 → MP4 filecontainer bytes
2. 02 → Demuxsamples in order
3. 03 → Encoded chunkEncodedVideoChunk
4. 04 → VideoDecodercompressed to raw
5. 05 → VideoFrameraw pixels; close() it
6. 06 → CanvasdrawImage, then close

Illustrative, not data

The figure shows the whole path. Read it left to right and you have the plan for the rest of this tutorial: the demuxer produces chunks, the decoder produces frames, and your code draws each frame and then releases it.

What is a codec, a container and a keyframe?#

Three terms carry most of the weight, so define them before any code. A codec is a hardware or software component that encodes or decodes a data stream, and the name is short for coder and decoder.^[4] H.264 and VP9 are codecs. A container is the wrapper that holds the encoded streams together with their timing and metadata. MP4 and WebM are containers.^[5] A video file is almost always a container holding one or more codec streams, and the decoder only ever sees the codec part.

A keyframe is a frame that can be decoded on its own, with no reference to any earlier frame. Wikipedia describes a group of pictures as the arrangement of intra-coded and inter-coded frames in a stream. Each group starts with a frame that decodes alone, and a decoder that reaches a new group needs no earlier frames to continue.^[6] The frames between two keyframes depend on the ones before them, so to show any one of them you must start from the keyframe and decode forward.

This is the single fact that shapes every decoder design. Playing a video from the beginning is cheap, because each frame follows the one before it. Jumping to a random time is expensive, because you must find the keyframe before the target and decode everything between. The rest of this tutorial is about working with that fact rather than against it.

Step-by-step: decode frames from an MP4 file#

The five steps below build a working decoder from an MP4 file. Each step ends with something you can check, and the code is written so that each step can be tested before the next one is added.

Step 1: Demux the container into encoded chunks#

Use a demuxer to read the track list, select the first video track, and turn each compressed sample into an EncodedVideoChunk. The code below uses mp4box.js, whose extraction API takes a track id, a user value and options, and calls back with batches of samples.^[2]

import MP4Box from "mp4box";

// Demux the first video track of an MP4 file into EncodedVideoChunk objects.
function demux(mp4, { onTrack, onChunk }) {
  mp4.onReady = (info) => {
    const track = info.tracks.find((t) => t.video); // the first video track
    onTrack(track);
    mp4.setExtractionOptions(track.id, null, { nbSamples: 1000 });
    mp4.start();
  };

  mp4.onSamples = (id, user, samples) => {
    for (const s of samples) {
      onChunk(new EncodedVideoChunk({
        type: s.is_rap ? "key" : "delta",              // random access point = keyframe
        timestamp: Math.round((s.cts * 1e6) / s.timescale), // presentation time, microseconds
        duration: Math.round((s.duration * 1e6) / s.timescale),
        data: s.data,                                  // the compressed bytes
      }));
    }
  };
}

The cts value is the composition time, which is when the frame should appear on screen. Use it rather than dts, which is the decode time, because the decoder outputs frames in presentation order. The conversion to microseconds is the step most people get wrong. Container timestamps are counted in ticks, and the timescale says how many ticks make one second. Multiply by a million and divide by the timescale, and the result is the microsecond value that WebCodecs expects.

The file is fed to the demuxer in pieces, using the same read loop as the import article. When the track list arrives, you know the codec string and the picture size that the decoder needs.

Step 2: Check support, then configure the decoder#

The decoder must be configured with the codec, the picture size and, for H.264 in MP4, a codec-specific description. The description is the binary record that the Chrome for Developers article calls the AVCC form, and it lives in the sample's description box.^[7] The helper readAvcRecord below is one you write: it extracts the payload of that box from the first sample.

// Ask first, then configure. A decoder that cannot handle the config throws later.
async function startDecoder(track, samples, onFrame, onError) {
  const config = {
    codec: track.codec,                 // e.g. "avc1.42c00d", from the demuxer
    codedWidth: track.video.width,
    codedHeight: track.video.height,
    description: readAvcRecord(samples[0]), // your helper: the avcC bytes
    hardwareAcceleration: "prefer-hardware",
  };

  const { supported } = await VideoDecoder.isConfigSupported(config);
  if (!supported) throw new Error(`This browser cannot decode ${track.codec}`);

  const decoder = new VideoDecoder({ output: onFrame, error: onError });
  decoder.configure(config);
  return decoder;
}

The isConfigSupported call is the honest check. It asks the browser about this exact config and resolves with a supported flag and a copy of the config.^[8] The hardwareAcceleration preference asks for a hardware decoder when one exists, which is faster and uses less battery, while still allowing a software decoder if hardware is missing.^[8]

Step 3: Feed chunks and watch decodeQueueSize#

The next step is the part that most tutorials skip. You cannot send a whole file to the decoder at once and wait. The decoder keeps a queue of pending work, and if the queue grows without limit, the decoder can fail. The MDN guide warns that a decoder given thousands of chunks at once might close or fail, and that the application must keep decodeQueueSize from growing without bound.^[9]

The W3C specification defines decodeQueueSize as the number of pending decode requests, which decreases as the codec accepts more input. It also fires a dequeue event whenever that number drops.^[10] A feeder that waits for the event keeps the queue bounded.

const MAX_QUEUE = 10; // a limit we chose, not a platform constant

// Feed chunks in order, pausing whenever the decoder's queue is full.
async function feed(decoder, chunks) {
  for (const chunk of chunks) {
    if (decoder.decodeQueueSize >= MAX_QUEUE) {
      await new Promise((resolve) =>
        decoder.addEventListener("dequeue", resolve, { once: true }));
    }
    decoder.decode(chunk);
  }
  await decoder.flush(); // emit the frames the decoder is still holding
}

The first chunk you send must be a keyframe. The specification says that decoding a delta chunk when no key chunk has been seen throws a DataError.^[10] Your demuxed list starts with a keyframe if the file does, so the rule is easy to meet here, but it matters when you start decoding from the middle of a file, as the seeking section does.

Notice that the feeder does not wait for each frame before sending the next chunk. The MDN guide says the decoder may need several chunks before it produces its first frame, and that the minimum depends on the device.^[9] Waiting for one frame per chunk would stall the pipeline.

Step 4: Close every frame you receive#

The output callback gives you a VideoFrame. A frame is a decoded picture, and it holds a reference to a media resource owned by the decoder. The specification encourages you to call close() as soon as the frame is no longer needed, and it warns that failing to release frames, or waiting for garbage collection to do it, can cause decoding to stall.^[10] MDN goes further and says that video frames are large enough for an application to crash with fewer than a hundred active frames in memory.^[9]

const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");

// Draw one decoded frame, then release it immediately.
function onFrame(frame) {
  ctx.drawImage(frame, 0, 0, canvas.width, canvas.height); // copy pixels to the canvas
  frame.close();                                           // free the decoder's memory now
}

Step 5: Flush and close the decoder#

When the last chunk is sent, call flush() to receive the final frames, and then call close() on the decoder to release its resources. The MDN guide notes that the decoder may hold the last few frames until you flush.^[9] Closing a decoder you no longer need is the matching cleanup, and it should happen whenever a clip is removed or a source changes.

async function finish(decoder) {
  await decoder.flush(); // the last frames are emitted here
  decoder.close();       // release the decoder's hardware session
}

How do you seek to a frame in a video?#

Seeking is the case where the keyframe fact matters most. To show the frame at a target time, you find the last keyframe at or before that time, reset the decoder, decode forward from that keyframe, and keep only the frames at or after the target. The keyframe search is a simple scan over the chunk list, which is in decode order, so this sketch assumes that presentation order matches decode order. Real files with reordered frames need the sample tables, which is a detail the demuxer handles for you.

// Index of the last keyframe whose timestamp is at or before the target (microseconds).
function keyframeAtOrBefore(chunks, targetUs) {
  let found = 0;
  for (let i = 0; i < chunks.length; i++) {
    if (chunks[i].timestamp > targetUs) break;
    if (chunks[i].type === "key") found = i;
  }
  return found;
}

// Decode from the keyframe before `seconds`. Frames before the target are closed unshown.
let targetUs = 0;
function seekTo(decoder, chunks, seconds) {
  targetUs = seconds * 1e6;
  decoder.reset();                                  // drop state from the old position
  const start = keyframeAtOrBefore(chunks, targetUs);
  for (const chunk of chunks.slice(start)) decoder.decode(chunk);
}

function onSeekFrame(frame) {
  if (frame.timestamp < targetUs) { frame.close(); return; } // decoded only to reach the target
  ctx.drawImage(frame, 0, 0);
  frame.close();
}

The reset() call matters. It clears the decoder's state so that the next chunk, which is a keyframe, starts a clean decode. Without it, the decoder would try to apply the new keyframe on top of the state from the old position.

The cost of a seek depends on how far the nearest keyframe lies from the target. The figure below shows a timeline cut into groups, each beginning with a keyframe, and a scrub target that sits in the middle of one group.

Fig. 02

GOP 10–2s

GOP 22–4s

GOP 34–6s

GOP 46–8s

0s5s

5s · Seek to 5s

1. 1GOP 1 0–2skey at 0s
2. 2GOP 2 2–4skey at 2s
3. 3GOP 3 4–6skey at 4s
4. 4GOP 4 6–8skey at 6s

Illustrative, not data[gop]

The shorter the groups, the less work each seek costs, and the bigger the file becomes. That is the central trade-off of keyframe placement. Encoders that place keyframes often make scrubbing cheap and files large. Encoders that place them rarely make files small and scrubbing slow. A video editor often cannot choose the encoder of the source file, so it may need a second, easier copy for scrubbing. The section on our editor below describes that choice.

Hardware acceleration, workers and fallbacks#

Three practical questions come up once the basic loop works. The first is whether to run the decoder on the main thread or in a worker. The MDN page for VideoDecoder notes that the interface is available in dedicated workers.^[11] Moving decoding into a worker keeps the page responsive while frames are produced, which matters most for export and for background analysis, where a long decode would otherwise freeze the interface.

The second question is hardware. The hardwareAcceleration value in the config asks for a hardware path when one exists. The general idea is that a dedicated hardware unit performs a specific task faster than a general-purpose processor running the same work in software.^[12] A decoder that falls back to software still works, but it is slower and draws more power, so frame budgets must allow for it.

The third question is what to do when isConfigSupported says no. The honest answer is a fallback chain, and the right chain depends on the product. One option is the HTML video element, which can play many formats the decoder cannot, though it offers less frame-level control. Another is a software decoder compiled to WebAssembly, which is slower but covers more codecs. The decision tree below shows the order to try them in.

Fig. 03

1. Does isConfigSupported say yes?
   1. yes → Decode with WebCodecs
   2. no → Can a video element play it?
      1. yes → Seek the video element
      2. no → Software decoder, slower

Illustrative, not data[isconfig]

A fallback is a promise to the user, and the promise should be stated. If the decoder path is unavailable, the user should see a message or a slower but correct result, never a silent black frame. The section below explains why that distinction matters to us.

How the Azelify editor handles decoding#

The Azelify editor uses a warm decoder per source. A warm decoder is one that is already open and already positioned near the last frame that was requested. When the next request is close to the current position and forward, the decoder simply pulls the next frames. The editor treats a short forward gap, about a second and a half, as close enough to pull through, because pulling a few frames is cheaper than rebuilding. A backward request or a long forward jump restarts the iterator with a single keyframe seek. Rebuilding the decoder from nothing is the most expensive path, so the editor avoids it wherever it can.

The access costs make the reasoning concrete. We measured each access pattern on a warm decoder in our editor, and the table below gives those figures.

The last row explains why scrubbing needs help. A 4K source with long groups of pictures makes a random jump expensive. Our editor builds a lighter, lower-resolution copy in the background, with more frequent keyframes, and uses that copy for scrubbing. Playback starts on the original immediately, so a background job never delays the first frame.

Several smaller decisions follow from the same thinking. The editor keeps a decoder budget per device class, and it closes an old decoder before it opens a new one, because opening first would briefly exceed the limit. A decoder that is no longer needed is retained until its pending output has drained, because dropping the reference too early can let frames be reclaimed without being closed. When the page is frozen by the browser, the warm decoders are released, since a frozen page holding hardware sessions is the page the browser is most likely to discard.

The editor also copies frames it intends to keep. A decoder's pooled canvas is overwritten by the next decode, so a cached frame that is not copied silently becomes whatever was decoded last. And when a fallback is needed, the editor tries the decoder again after a short delay, then falls back to the video element, and finally shows a clearly labeled placeholder rather than a black frame. The short delays are deliberate: the whole fallback ladder costs roughly 150 milliseconds of waiting, and that is the most a preview can absorb without a visible stall.

Our editor also decodes animated GIFs differently from video. A GIF is a video clip on the timeline, but the demuxing library cannot read GIF files, so the editor decodes every frame up front and keeps them. The lesson is the same as the one for seeking: know which decode path a file needs, and choose it from the bytes rather than from the label.

Frequently asked questions#

Why does my decoder return no frames at first?#

The decoder may need several chunks before it can output its first frame, and the number depends on the device and the stream. Feed chunks in batches rather than waiting for one frame per chunk. Call flush() at the end, because some frames may still be held until then.

Does WebCodecs work in every browser?#

No. Support depends on the browser, the operating system and the codec. Call VideoDecoder.isConfigSupported with the exact config you need, and build a fallback for the cases where it returns false.

Can I use WebCodecs without a video element?#

Yes. The decoder works without any element on the page, which is why it suits frame-by-frame work such as export, analysis and thumbnails. You are responsible for drawing the frames yourself, usually to a canvas.

What happens if I forget to close a frame?#

The decoder's memory is held until the frame is closed or the garbage collector reclaims it. Holding many unclosed frames can stall decoding or crash the page. Close each frame as soon as you have drawn or copied it.

Is WebCodecs faster than a video element for seeking?#

It depends on the job. A video element is simpler and handles playback well. WebCodecs gives you frame-level control and lets you decide what to keep, but you must write the seek logic yourself, and a bad keyframe plan can make seeking slower than the element.

Which codec string should I pass to the decoder?#

Use the string the demuxer reports for the track, such as avc1.42E01E for a baseline H.264 stream. The string names the codec and its profile, and the browser needs it to choose a decoder. Do not guess it from the file extension.

Where to go next#

For the step before decoding, read how to import video files in the browser, which covers the file signature, the container probe and the storage of the bytes. Once frames arrive, frame-accurate playback in JavaScript explains how to pace them against a clock. If you are choosing between decoding approaches, the comparison of ffmpeg.wasm, WebCodecs and MediaRecorder sets out the trade-offs.

Sources

1. MDN Web Docs, “WebCodecs API,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2] ↩^[3]

2. GitHub, “gpac/mp4box.js,” github.com (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2]

3. Mediabunny, “Mediabunny — A complete JavaScript media toolkit for the browser,” mediabunny.dev (opens in a new tab) (accessed 2026-10-13). ↩

4. Wikipedia, “Codec,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

5. Wikipedia, “Container format,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

6. Wikipedia, “Group of pictures,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

7. Chrome for Developers, “Video processing with WebCodecs,” developer.chrome.com (opens in a new tab) (accessed 2026-10-13). ↩

8. MDN Web Docs, “VideoDecoder: isConfigSupported() static method,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2]

9. MDN Web Docs, “Using the WebCodecs API,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2] ↩^[3] ↩^[4]

10. W3C, “WebCodecs,” www.w3.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2] ↩^[3]

11. MDN Web Docs, “VideoDecoder,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

12. Wikipedia, “Hardware acceleration,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩