Compositing video layers with Canvas and WebGL: transforms, blends and effects

Video compositing is the job of drawing several layers, such as video clips, still images and text, into one picture for a single moment in time. In the browser you can do it with Canvas 2D or with WebGL, and the choice between canvas vs WebGL for video is usually the first technical decision a team makes after picking a timeline. This article explains what compositing is, how the two APIs differ, and how to draw a stack of layers in the right order.

We build from the bottom up. First we define a layer, a transform, an alpha channel and a blend mode, then show the smallest code that draws a frame with Canvas 2D. After that we move to WebGL, write a small fragment shader, and compare the two with a table and a decision tree. Where the Azelify editor, a browser-native video editor we build, makes a useful trade-off, we say so near the end.

You need to know basic JavaScript and what a pixel is. Nothing else is assumed, and every term is defined the first time it appears.

What is video compositing, and why does order matter?#

A layer is one thing you want to see in the final picture: a decoded video frame, a photograph, a caption or a shape. Compositing is the act of combining all visible layers at one moment, so that the result looks like a single image. The word comes from the photographic practice of combining several exposures or elements into one picture.^[1]

Order matters because later drawing covers earlier drawing. The simplest correct method is the painter's algorithm: sort the things to be drawn from farthest to nearest, then paint them in that order, so that the nearer object overwrites the farther one where they overlap.^[2] A painter paints the sky first and the trees last, so the trees sit in front of the sky without any extra bookkeeping. Compositing uses the same idea, with layers instead of objects. The bottom layer is painted first and the top layer last.

// Layers are listed bottom to top. Index 0 is drawn first, so it sits at the back.
const layers = [
  { draw: (ctx) => { ctx.fillStyle = "#111"; ctx.fillRect(0, 0, 1920, 1080); } },
  { draw: (ctx) => ctx.drawImage(videoFrame, 0, 0, 1920, 1080) }, // decoded frame
  { draw: (ctx) => { ctx.fillStyle = "white"; ctx.fillText("Hello", 100, 900); } },
];

// Later draws cover earlier ones where they overlap.
for (const layer of layers) layer.draw(ctx);

The important property is that the whole stack is computed for one moment. If you draw the video at one time and the caption at another, you have made a collage, not a composite. Every layer must be evaluated at the same time value before anything is painted.

Here is the first figure. It shows the pipeline for one frame. We return to each box in the sections that follow.

Fig. 01

1. 01 → Decode frameone picture per layer
2. 02 → Apply transformmatrix, not new pixels
3. 03 → Apply effectscolour, blur, mask
4. 04 → Blend onto stackbottom to top
5. 05 → Output framesame steps for export

Illustrative, not data[wiki-painter][mdn-composite]

What is a transform matrix?#

A transform is a rule that moves, turns, stretches or skews a layer before it is drawn. The mathematical name for the family of rules that keep straight lines straight and parallel lines parallel is an affine transformation.^[3] Canvas 2D uses six numbers to describe one such rule, conventionally written a, b, c, d, e and f.

MDN gives the rule for a point. A point at (x, y) lands at (a times x plus c times y plus e, b times x plus d times y plus f).^[4] The numbers e and f move the layer. The numbers a and d scale it, and b and c skew or rotate it. A rotation by an angle theta uses cosine and sine, so the matrix is cos, sin, minus sin, cos.

// Rotate a layer 30 degrees, then move it 200 px right and 100 px down.
// The six numbers are a, b, c, d, e, f.
const angle = Math.PI / 6;
const cos = Math.cos(angle);
const sin = Math.sin(angle);

ctx.setTransform(cos, sin, -sin, cos, 200, 100);
ctx.drawImage(videoFrame, 0, 0, 640, 360); // drawn in the moved, turned space

ctx.setTransform(1, 0, 0, 1, 0, 0); // reset to the identity before the next layer

Two details trip up beginners. First, setTransform replaces the current matrix, while transform multiplies it into the current one. Second, the order of operations matters: rotating and then moving is not the same as moving and then rotating. A common pattern is to save the context, set the matrix for one layer, draw it, and restore the saved state, so that no transform leaks into the next layer.

How do blend modes and alpha work?#

Alpha is the fourth channel of a pixel, the opacity. A pixel with alpha 0 contributes nothing and a pixel with alpha 1 covers whatever lies beneath. Porter and Duff described the mathematics of combining these partly transparent pixels in 1984, and their over operator is the rule that most people mean when they say a layer is drawn on top of another.^[5]

The W3C Compositing and Blending specification defines the general formula. It separates two questions: which pixels of each layer survive, which is the compositing operator, and how the colours mix where they overlap, which is the blend mode.^[6] Canvas exposes both through one property, globalCompositeOperation.^[7] The default value, source-over, places the new layer on top. The value lighter adds colour values together, which is how a glow or a dissolve can be built. The value multiply darkens by multiplying the two colours, so white leaves the lower layer unchanged and black produces black.

// Source-over is the default: the new layer covers what is below it.
ctx.globalCompositeOperation = "source-over";
ctx.globalAlpha = 0.5; // half opacity for this layer only
ctx.drawImage(videoFrame, 0, 0);

// Multiply darkens: each colour channel is multiplied by the one beneath.
ctx.globalCompositeOperation = "multiply";
ctx.drawImage(gradientLayer, 0, 0);

// Restore the defaults so later layers are not affected.
ctx.globalCompositeOperation = "source-over";
ctx.globalAlpha = 1;

Premultiplied alpha is the storage format that makes this arithmetic reliable. Instead of storing red, green, blue and alpha separately, each colour channel is multiplied by alpha before it is stored. The Wikipedia article on alpha compositing explains that this avoids colour bleeding at the edges of transparent regions and makes filtering correct.^[1] You do not need to convert anything by hand in Canvas, because the browser handles the storage format. You do need to know it when you write a shader, which is why it comes up again below.

Canvas 2D vs WebGL: which should you use?#

Canvas 2D is a set of drawing commands that the browser executes for you, often on the GPU. WebGL is a lower-level interface to the GPU. You write small programs called shaders, which run once for every pixel, and you upload the pictures they read.^[8] WebGL 2 adds features such as 3D textures and sampler objects, and it is exposed as its own interface.^[9] WebGPU is the newer successor, with general-purpose compute and access to more modern GPU features.^[10][11]

The practical comparison is easier to see in a table.

The decision tree below turns that table into a sequence of questions. It is a rule of thumb for one team, not a benchmark, and the answers change as browsers improve.

Fig. 02

1. Need custom shader effects?
   1. yes → Need GPU compute as well?
      1. yes → WebGPU, with caveats
      2. no → WebGL2
   2. no → Many layers or heavy blends?
      1. yes → WebGL2, measure first
      2. no → Canvas 2D

Illustrative, not data[mdn-webgl][mdn-webgpu][mdn-composite]

The honest answer for most first versions is Canvas 2D. It is easier to make correct, it is easier to test, and browsers accelerate its common operations. You move to WebGL when you need an effect that Canvas cannot express, or when a composite with many layers is too slow to keep up with a playback budget.

Build a Canvas 2D compositor in five steps#

The following steps build a compositor that draws one frame from a list of layers. It is deliberately small, but it uses the same structure a larger editor needs.

Step 1: Create one reusable target#

Create one canvas and reuse it for every frame. Creating a new canvas for each frame allocates memory and slows the loop. An OffscreenCanvas can be drawn off the main page, and it can run in a worker, which is how many editors keep rendering away from the user interface.^[12]

// One reusable target for every frame. Create it once, outside the loop.
const target = new OffscreenCanvas(1920, 1080);
const tctx = target.getContext("2d");

Step 2: Stack the layers in order#

Every frame begins by clearing the target, so that pixels from the last frame cannot show through transparent areas. Then the layers are visited from the bottom up, exactly as the painter's algorithm requires.

async function renderFrame(time, layers) {
  // Clear first, or last frame's pixels show through transparent areas.
  tctx.clearRect(0, 0, target.width, target.height);

  for (const layer of layers) {          // bottom to top
    const bitmap = await layer.decodeAt(time); // the picture for this moment
    drawLayer(tctx, bitmap, layer);
    bitmap.close();                      // release memory held by the decoder
  }
  return target.transferToImageBitmap(); // the finished picture, as one object
}

The loop awaits each decode before drawing. In a real editor, decodes run ahead of the playhead and the await mostly resolves at once, which is the decoding discipline covered in the WebCodecs article.

Step 3: Place each layer with a transform#

Each layer carries its own six numbers. The function below saves the context, applies the matrix, draws, and restores the saved state. Restoring is the step beginners forget, and forgetting it makes every later layer inherit the rotation.

function drawLayer(ctx, bitmap, layer) {
  ctx.save();                                   // remember the current state
  ctx.setTransform(...layer.matrix);            // six numbers: a, b, c, d, e, f
  ctx.globalAlpha = layer.opacity;              // 0 to 1
  ctx.globalCompositeOperation = layer.blend;   // for example "multiply"
  ctx.drawImage(bitmap, 0, 0, layer.width, layer.height);
  ctx.restore();                                // undo transform, alpha and blend
}

Step 4: Apply opacity and blend#

Opacity and blend are both properties of the context, so they are set before drawing and reset by restore. The same drawing call produces a different picture when the blend changes, which is why blends are part of the layer's data rather than a hard-coded behaviour.

Step 5: Hand back the finished frame#

The last step returns the finished picture. transferToImageBitmap creates an ImageBitmap from the most recent frame and leaves the canvas blank for the next one. The caller owns that bitmap and must close it when it is finished, because a bitmap holds graphics memory until it is released. MDN is explicit that dropping the reference is not enough.^[12]

How do video frames reach the GPU with WebGL?#

Before a shader can read a video frame, the pixels must be copied into a texture, which is the GPU's name for a two-dimensional image it can sample. WebGL provides texImage2D for this copy. Its source can be an ImageBitmap, an HTML video element, an OffscreenCanvas or a VideoFrame, among others.^[13] MDN's video texture example shows the same idea with a video element.^[14] The figure traces the path a frame takes from decoder to output.

Fig. 03

1. 01 → VideoFramedecoder output
2. 02 → Upload texturetexImage2D copy
3. 03 → Sample pixeltexture() lookup
4. 04 → Scale by opacitypremultiplied rgba
5. 05 → Write outputframebuffer or canvas

Illustrative, not data[mdn-teximage][mdn-video-tex][wiki-alpha]

// Assume gl is a WebGL2 context and frame is a VideoFrame from a decoder.
const tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, tex);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);

// Copy the pixels to the GPU. The VideoFrame is the source.
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, frame);
frame.close(); // the copy is complete, so release the decoder's memory

Three points matter for an editor. The copy is the expensive step, so each frame should be uploaded once, not once per effect. The clamp-to-edge wrap mode stops a blurred edge from sampling the opposite side of the picture. And the frame must be closed after it is copied, or the decoder will run out of buffers.

Write a tiny fragment shader for one layer#

A fragment shader is a small program that runs once for each pixel of the output, and returns that pixel's colour. The shader below reads the uploaded frame and multiplies every channel by an opacity value. Because the stored colours are premultiplied, scaling all four channels together gives the correct fade.^[1]

#version 300 es
precision mediump float;

uniform sampler2D u_layer; // the uploaded video frame
uniform float u_opacity;   // 0.0 (invisible) to 1.0 (full)
in vec2 v_uv;              // where this pixel sits on the layer, 0 to 1
out vec4 outColor;         // the colour written to the target

void main() {
  vec4 c = texture(u_layer, v_uv); // read one texel
  outColor = c * u_opacity;        // scale rgb and alpha together
}

The shader is useful because it shows where the work happens. A Canvas 2D context hides this step. A WebGL program makes you decide how each layer is sampled, how it is scaled and how it is written, which gives you control of effects such as a colour grade, a blur or a mask. The cost is that you must also decide how the output is blended with what lies beneath, and get the premultiplied arithmetic right.

Layers are usually drawn into their own framebuffers first, and the framebuffers are then combined. That gives the same stacking order as Canvas 2D, with the blend written in a second shader instead of chosen from a list.

How do transitions and nested layers fit in?#

A transition, such as a dissolve between two clips, needs both clips at the same moment, so the renderer draws each clip into its own full-size layer and blends the two. A nested layer, sometimes called a compound clip, is a small timeline rendered as one layer inside another. The clean way to support both is to use the same routine that draws an ordinary clip for each child, so that a child behaves identically whether it is at the top level or inside a nest.

If the transition code uses a different path from the ordinary clip code, the two will drift apart over time. Effects will apply in one place and not the other, and a bug fix in one path will not reach the second. Sharing one routine is the simplest way to avoid that drift.

How do you know a composite is correct?#

A composite is easy to make look right and hard to prove right. A reviewer can see that a caption sits in front of a video, but not that it sits exactly where its matrix says it should. Two kinds of check help. The first compares the output with a saved reference picture, often called a golden image, and reports the pixels that differ. It proves that nothing changed, which is valuable during a refactor, but it cannot prove that the saved picture was correct in the first place.

The second kind computes the expected answer independently. For example, you can calculate where a corner of a transformed rectangle must land, then check that the renderer draws that corner there. It takes longer to write, but it catches a mistake that was saved into a golden picture long ago. The testing article covers both approaches in detail, including the trap of saving a wrong picture as the reference.

A useful habit is to check the compositor at two output sizes. A transform that is correct at 1920 by 1080 can still be wrong at a 960 by 540 preview if a scale is applied to some of its numbers but not others. Rendering the same frame twice and comparing the results is a cheap way to catch that class of error before a user does.

How the Azelify editor handles compositing#

The Azelify editor composites every frame through one renderer. Given a project, a time and a target size, it produces one picture by painting the active clips from the bottom track to the top track. Preview and export both call that renderer, so the steps described above run in the same order for both. The decision has a clear benefit: a fix to compositing reaches the preview and the exported file at the same time.

Three other decisions shape the renderer. Placement is decided once, in the project's own coordinate space, before anything is painted. The size of a decoded picture never decides where a clip lands, so decoding at a smaller size changes sharpness and nothing else. Painters draw content into a box that already exists. Second, the renderer treats a failed text clip as a visible fault tile rather than an error that blanks the whole frame. A black frame with no explanation is harder to debug than one bad clip marked in place. Third, effects run on the GPU only when their output has been measured to match the CPU reference. A chain runs entirely on one backend, because every crossing between the GPU and the CPU costs a copy back to memory, roughly five milliseconds at full HD.

The table summarises the choices and their costs.

A WebGPU compositor exists in the editor's code, but it is deliberately not wired in yet. The renderer still composites through Canvas 2D composite modes, which the browser already accelerates. Replacing that path is a conscious decision, not an accident, and it is recorded so that the team does not switch by habit. Motion blur is handled by summing several sub-frames across the shutter, and a playing preview draws only one sample to keep the frame rate steady.

Common mistakes#

Forgetting to clear the target. A reused canvas keeps last frame's pixels in every transparent area. Clear it at the start of every frame.

Leaking a transform. Without save and restore, a rotation applied to one layer rotates everything drawn after it. Wrap each layer in save and restore.

Drawing layers at different times. A composite is one moment. Evaluate every layer at the same time value before you draw any of them.

Uploading the same frame twice. Each upload is a copy. Upload once per frame and reuse the texture for every effect that reads it.

Frequently asked questions#

Is Canvas 2D fast enough for video compositing?#

For a handful of layers with standard blends, usually yes, because the browser accelerates the common operations. It becomes a limit when you need many layers, custom per-pixel effects or colour-accurate grading. Measure with a real project before you move to shaders.

When should I use WebGL instead of Canvas?#

Use WebGL when you need a shader effect that Canvas cannot express, or when a stack of layers no longer fits the frame budget. The shader gives you control over every pixel, at the cost of more code and more careful memory handling.

What does premultiplied alpha mean in simple terms?#

Each colour channel is stored already multiplied by the pixel's opacity. A transparent pixel therefore carries no colour, which prevents dark or bright fringes at the edges of transparent shapes. Shaders must scale all four channels together when they fade a layer.

Why does my composite look different in preview and export?#

The two paths probably do not share one renderer, or they use different sizes or colour handling. Route both through a single function, and test the output at more than one target size. A small difference in scale can look like a large difference in edges.

Can I run compositing in a web worker?#

Yes. An OffscreenCanvas can be transferred to a worker and drawn there, so the main thread stays free for the interface.^[12] You still need to move decoded frames to the worker, and you must close any bitmap you pass along.

What is the difference between a transform and a blend mode?#

A transform decides where a layer is drawn and how it is shaped. A blend mode decides how its colours combine with the layers already on the target. You can change one without changing the other, and both are set per layer.

Where to go next#

If you want the timing side of this loop, read frame-accurate playback in JavaScript. It explains which frame each composite should draw and why the clock matters.

Once a layer can be placed on the canvas, the next question is how a person moves it. The transform handles article covers hit testing, pointer capture and rotated bounds, which are the same transform numbers seen from the user's side.

Sources

1. Wikipedia, “Alpha compositing,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2] ↩^[3]

2. Wikipedia, “Painter's algorithm,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

3. Wikipedia, “Affine transformation,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

4. MDN Web Docs, “CanvasRenderingContext2D: setTransform() method,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

5. Thomas Porter and Tom Duff, “Computer Graphics Volume 18, Number 3, July 1984,” keithp.com (opens in a new tab) (accessed 2026-10-13). ↩

6. W3C, “Compositing and Blending Level 1,” www.w3.org (opens in a new tab) (accessed 2026-10-13). ↩

7. MDN Web Docs, “CanvasRenderingContext2D: globalCompositeOperation property,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

8. MDN Web Docs, “WebGL: 2D and 3D graphics for the web,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

9. MDN Web Docs, “WebGL2RenderingContext,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

10. MDN Web Docs, “WebGPU API,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2]

11. Chrome for Developers, “Overview of WebGPU,” developer.chrome.com (opens in a new tab) (accessed 2026-10-13). ↩

12. MDN Web Docs, “OffscreenCanvas,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2] ↩^[3]

13. MDN Web Docs, “WebGLRenderingContext: texImage2D() method,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩

14. MDN Web Docs, “Video textures,” developer.mozilla.org (opens in a new tab) (accessed 2026-10-13). ↩