How to test a video renderer: golden images, oracles and property-based tests

Visual regression testing for canvas is the practice of checking that a drawn picture has not changed by accident. A canvas (opens in a new tab) is the browser's drawing surface, and a video renderer paints every frame onto one, so a mistake usually appears as a wrong pixel rather than a wrong return value. That makes the testing problem harder than it first looks, because a suite can be fully green while the renderer it protects is quietly broken.

This guide explains how to test a video renderer properly. It starts with the simplest kind of check, shows why that check is weak, and builds toward the tools that catch what it misses. Three ideas carry the article. A golden image test proves that the output did not change, not that it was right, so a wrong picture saved as the golden stays green forever. An oracle computes the expected answer independently, such as where a corner should land, and catches errors that a golden cannot. Property-based tests generate thousands of inputs and check rules that must always hold, which finds edge cases nobody would think to write. The Azelify editor uses all three, and the later sections show where each one earns its place. For the drawing side of the same problem, read canvas vs WebGL for video compositing.

Visual regression testing for canvas: why a renderer is hard to test#

A renderer is the code that turns a description of a project into pixels. A pixel is one coloured dot, and a single 1920 by 1080 frame holds about two million of them. Each pixel carries four numbers, for red, green, blue and alpha, where alpha is transparency. A test is code that runs the renderer and then asserts that something about the result is right.

That last word is the difficulty. For a function that returns a number, you can write the expected number by hand: the sum of 2 and 2 is 4. A frame is different. Nobody can write two million expected values by hand, and a person glancing at a picture will miss a shift of two pixels. The question of what the correct output is for a given input has a name in software testing, the oracle problem (opens in a new tab). An oracle is any source of the right answer, such as a formula, a reference implementation, a saved picture or a human judgement.^[1]

Consider a function that crops a frame to a vertical 9:16 region. A test that runs the crop and checks that the result is smaller than the original proves very little. It would pass if the crop used the wrong corner, or the wrong size. We need a test that knows where the crop should land, and the rest of this article is about building that knowledge into tests.

Choosing between a golden, an oracle and a property#

Three kinds of check cover most renderer questions. Use a golden image when the expected picture is fixed and you can store it. Use an oracle when you can compute the expected answer independently of the renderer. Use a property when a rule must hold for every input in a range. If none of those fits, the test is only a change detector, and that is the weakest kind of test.

Fig. 01

1. Does the test know the right answer?
   1. yes → Is the right answer one fixed picture?
      1. yes → Golden image with tolerance
      2. no → Is it a rule for all inputs?
         1. yes → Property-based test
         2. no → Write an oracle first
   2. no → Change detector only: weak

Illustrative, not data[wiki-oracle]

Change detectors: the test that stays green when the picture is wrong#

A change detector asserts that output differs from a previous run. It carries one bit of information, which is yes or no, something changed. Our editor's suites once showed the trap clearly. Someone deliberately broke the crop in the compositor so that it read the wrong region of the source image. The full render suite still reported 15 passed and 0 failed. Its assertions had the shape below.

expect(result.cropped).not.toBe(result.full); // "the output changed"

That line passes for every crop that is not a literal no-op. A crop is described by four numbers, but the assertion carries one bit, so a wrong crop passes as long as it differs from the uncropped frame. The same blind spot hid a larger defect. Text rendered 2.13 times oversize in the preview for months, under a fully green suite. Nothing asserted the actual rendered size. The suite only asserted that something had rendered.

The lesson concerns the value of an assertion. A test that proves only change can be fully green while the output is simply wrong. So the first rule for renderer tests is that every test makes at least one value assertion, one that states what the output is.

Three review rules that keep tests honest#

Our editor holds its render tests to three review rules, and each one answers a failure we actually saw. The first requires a value assertion, as described above. The second forbids loosening a render assertion in the same commit as a change to render code. A developer under pressure, or an AI coding assistant, can make a failing test pass by weakening the assertion rather than fixing the code. Splitting the two into separate commits makes any weakening visible on its own. The third rule concerns units. Layout measurements such as font size are expressed in project units, and using them directly against a canvas measured in device pixels is the mistake that made the preview text too large. Convert first, and keep the two spaces apart in the code.

These rules do not make a test correct. They make it harder for a weak test to pass unnoticed, which is the realistic goal for a team.

What is a test oracle, in plain words?#

Wikipedia defines a test oracle as a provider of information that describes correct output for the input of a test case. Testing with an oracle means comparing the actual result of the system with the expected result that the oracle supplies.^[1] The oracle can be a separate computation that runs at test time, or the expected value can be written into the test in advance. Some oracles are heuristic, meaning they give approximate answers across a whole class of inputs rather than one exact answer.^[1]

A saved picture is an oracle too, but a weak one. It says the answer is whatever was saved last time, which is a claim about the past, not about correctness. That distinction drives the rest of this article. Goldens are useful, but they need a companion that knows the answer independently.

Golden images for visual regression testing: save, then compare#

Golden master testing, also called approval testing, records how existing code behaves and compares later runs against that record. The idea goes back to Michael Feathers and is described on Wikipedia as characterization testing.^[2] For images, the procedure is simple. Render the frame once, save it as the golden, and on later runs render it again and compare. The method depends on repeatability, so volatile values such as timestamps must be masked before the comparison.

Exact equality is usually too strict for pictures. Different graphics drivers, GPUs and encoders can produce tiny differences in colour at anti-aliased edges, and a test that fails on a one-level change in one pixel is noisy. Comparing with a tolerance solves that. The pixelmatch library compares two buffers of RGBA values and counts the pixels that differ.^[3]

import pixelmatch from 'pixelmatch';

// actual and golden are RGBA buffers of the same width and height.
const diff = new Uint8ClampedArray(width * height * 4);
const mismatched = pixelmatch(actual, golden, diff, width, height, { threshold: 0.1 });
if (mismatched > 0) {
  throw new Error(`${mismatched} pixels differ from the golden image`);
}

The threshold of 0.1 is the library's default, and lower values are stricter. By default, pixelmatch ignores anti-aliased edge pixels, which is what you want when a small difference in an edge is expected. Its windowSize option changes the count to the largest number of differing pixels inside any square window, so a scatter of single noisy pixels does not fail a test while a cluster still does. Perceptual metrics take a different route. The structural similarity index, for example, compares an image against an uncompressed reference and returns a value that is 1 for identical images.^[4]

The trap is that a golden saved from a broken render is a perfect golden. The first time someone accepts a broken picture as the new baseline, the test turns green for the wrong reason. That is why goldens need the next kind of check.

Placement oracles: the pixels must sit where the box says#

Our gizmo, the handles around a selected clip, once drifted away from the picture. The outline was drawn in one place and the pixels in another. A test that compared two numbers could not see that, because both numbers were correct in their own system. The bug class was never "a number is wrong". It was "the picture and the box disagree".

The fix is an oracle that reads the pixels back. The editor's conformance suite renders each kind of clip alone on a transparent base, using the real painters. It then reads the canvas, measures the bounding box of the pixels that have alpha above zero, and asserts that this box matches the bounding box of the clip's expected quadrilateral within a tolerance of 2.5 target pixels. The tolerance is 1.5 pixels for the resampler and one more for the anti-aliased edge. The oracle is the pixels. Nothing the painter reports about itself is trusted.

The core of such a check is small. The code below is an illustration of the idea, not the editor's source.

// Oracle: the bounding box of the pixels that are not transparent.
function alphaBounds(rgba: Uint8ClampedArray, width: number, height: number) {
  let minX = width, minY = height, maxX = -1, maxY = -1;
  for (let y = 0; y < height; y++) {
    for (let x = 0; x < width; x++) {
      const alpha = rgba[(y * width + x) * 4 + 3];
      if (alpha === 0) continue; // transparent: not part of the clip
      minX = Math.min(minX, x); maxX = Math.max(maxX, x);
      minY = Math.min(minY, y); maxY = Math.max(maxY, y);
    }
  }
  return { minX, minY, maxX, maxY };
}

// Compare one edge of the measured box with the expected edge.
function expectWithin(actual: number, expected: number, tolerancePx: number, label: string): void {
  if (Math.abs(actual - expected) > tolerancePx) {
    throw new Error(`${label}: got ${actual}, expected ${expected} ± ${tolerancePx}`);
  }
}

Fig. 02

1. 01 → Render alonetransparent base
2. 02 → Read pixelscanvas back to bytes
3. 03 → Measure boxnon-transparent bounds
4. 04 → Compare quadexpected edges
5. 05 → Within 2.5 pxpass or fail

Illustrative, not data[wiki-oracle]

The judge must also be tested. The editor's suite includes a row that shifts every draw by a few pixels on purpose. The clean render must agree with the expected box, and the shifted render must disagree. Then the shift is switched off again. A conformance test that a deliberate mutation cannot turn red is a change detector in disguise, so this row is what proves the oracle can fail.

Property-based testing with fast-check#

A property is a rule that should hold for every valid input, not for a handful of examples that a person chose. A property-based testing library generates many inputs, runs the rule against each one, and reports the first counterexample it finds. QuickCheck, the Haskell library that gave the technique its name, was first released in 1999 and works by searching for a falsifying input and then reducing it to a minimal example.^[5]

The JavaScript library fast-check follows the same pattern. Its generators are called arbitraries, each test runs many times with a default of 100 runs, and a failing case is reduced by counterexample shrinking so that you see a small example rather than a random one. Arbitraries draw from a seeded random number generator, so a seed reproduces the same values on every run.^[6] For an asynchronous rule, fast-check provides asyncProperty.^[7]

Here is a rule that a timeline needs. Splitting a clip at any cut strictly inside it must leave two pieces that touch each other and together cover the original.

import fc from 'fast-check';

type Clip = { start: number; end: number }; // timeline milliseconds, end exclusive

// The unit under test: the split operation the editor would perform.
function splitClip(clip: Clip, at: number): [Clip, Clip] {
  return [{ start: clip.start, end: at }, { start: at, end: clip.end }];
}

test('a split leaves two contiguous pieces that cover the original', () => {
  fc.assert(
    fc.property(
      fc.integer({ min: 0, max: 60_000 }), // clip start
      fc.integer({ min: 2, max: 60_000 }), // clip length
      fc.integer({ min: 1, max: 59_999 }), // offset of the cut
      (start, length, offset) => {
        fc.pre(offset < length); // skip cuts that fall outside the clip
        const clip = { start, end: start + length };
        const [a, b] = splitClip(clip, start + offset);
        return a.start === clip.start && a.end === b.start && b.end === clip.end
          && a.end > a.start && b.end > b.start;
      },
    ),
  );
});

A few hand-written examples would probably check a cut in the middle and miss a cut at the first millisecond. Generated inputs reach those edges without anyone thinking of them. Our editor's suites use this style for rules that change continuously, such as keyframe interpolation, speed changes, transitions and transforms, because a fixed golden frame can sample only a few points along such a rule.

Parity tests: two paths must agree#

A parity test renders the same content through two paths that must produce the same result. The editor has several. Preview must match export. A WebGL2 path must match the Canvas2D path. Realtime audio must match offline audio. A cache that is cold must match a cache that is warm. A frame list in shuffled order must match the frame list in order. Each comparison guards against a different kind of drift.

One parity suite makes two assertions on purpose: fidelity and alignment. Fidelity asks whether the frames look the same. Alignment asks whether each frame sits at the correct timestamp. The two fail in different ways. An over-aggressive encoder breaks fidelity. A timestamp that is off by one frame can still look fine on slow-moving content, so only the alignment assertion catches it. Asserting both is what makes the parity test a real oracle rather than a visual impression.

Mutation testing: break the code on purpose#

Mutation testing is a way to test the tests. You make a small deliberate change to the code, called a mutant, run the suite, and check that at least one test fails. The score is the share of mutants the suite catches.^[8] A suite that passes against a broken program is hiding a gap.

The editor keeps some standalone checks that must fail when specific faults are injected. The table lists three of them. Each check is an oracle that runs in a small page, and each one is designed to go red.

If one of those injected faults passes, the check is a change detector and needs a value assertion added before anyone relies on it.

Choosing the first harness when a bug appears#

When a new editor bug arrives, the first question is which kind of failure it is. Running the wrong harness wastes time, so the editor's routing notes map a symptom to the harness most likely to reproduce it. The table names five classes.

The sequence in the figure shows the same habit as a reusable procedure, so the routing does not depend on memory.

Fig. 03

1. 01 → Symptom reportwhat the user saw
2. 02 → Classify C1 to C5match the failure class
3. 03 → Pick harnessfrom the routing table
4. 04 → Reproduce redfail before any fix
5. 05 → Fix and rerunsame harness, now green

Illustrative, not data

The table also states a limit, and the limit matters more than the table. Golden frames alone cannot catch freezes, placement errors or memory creep. They catch text drift only when the catalogue is covered exhaustively, and they catch nondeterminism only when the harness actually varies warmth and frame order. Writing those limits down is what stops a green suite from being mistaken for proof.

How the Azelify editor proves its picture#

The editor's render harness runs the real renderer inside a headless Chromium browser. It has to, because the normal unit-test environment has no canvas and no WebCodecs (opens in a new tab), so only a real browser can draw the picture. That harness holds pixel goldens, parity suites for preview and export, runtime checks for the preview under stress, and checks for typography, export and media. Several dozen spec files cover those areas as of this writing.

Node-side tests need pixel values too, so the editor includes a software 2D context. It implements a small, deliberately incomplete set of drawing operations, enough to assert a transition's endpoint and coverage pixel for pixel without a browser. The incompleteness is intentional. An operation the software context does not support is simply absent, so a test cannot pass by quietly exercising something the software version gets wrong. If a test needs an operation that is missing, the author either extends the software context on purpose or moves the assertion into the real-browser harness.

Tests in the golden harness are held to the three review rules described earlier. The routing table and the explicit limits sit beside them, because a harness is only useful when the team knows what it cannot see.

Frequently asked questions#

What is visual regression testing for canvas?#

It is a family of tests that check whether drawn pictures have changed, usually by comparing a new render with a saved image. On its own it only detects change. It becomes a real test when the expected picture is correct and the comparison has a stated tolerance, and when a separate oracle checks what the picture should contain.

How do I write a golden image test?#

Render the frame in a fixed environment, save the output as the golden, and on later runs render it again and compare the two with a tolerance. Mask volatile values such as timestamps before the comparison. Review every change to a golden as carefully as a change to the code that produced it.

How much pixel difference should a test allow?#

Allow as little as the environment requires. Start with the library default threshold, measure the real variation across the machines you run on, and set the tolerance just above that variation. A tolerance that is set too wide will hide the bugs you most need to see.

Why did a snapshot test pass after a real bug?#

Most likely the bug was saved into the snapshot when someone accepted a new baseline. A snapshot records the output of the moment it was taken, so it cannot tell whether that output was correct. An independent oracle or a value assertion is what catches a bug that the snapshot has already absorbed.

Is a property-based test better than a unit test?#

It is different, and each has a place. A unit test pins one example that you chose. A property test states a rule and tests it against many generated inputs, which is better for continuous ranges and edge cases. Keep unit tests for examples that matter in their own right, and use properties for the rules that must hold everywhere.

Can a test be wrong about its own expectation?#

Yes, and that is the reason for mutation testing. Inject a deliberate fault and check that the test fails. If the test still passes, its expectation is too weak or aimed at the wrong thing, and it needs a stronger assertion.

Where to go next#

If the renderer you are testing draws layers and effects, read canvas vs WebGL for video compositing to see what those layers are doing. If your tests cover exported files, how to export MP4 from the browser explains the export path that parity tests have to match.

Sources

1. Wikipedia, “Test oracle,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩ ↩^[2] ↩^[3]

2. Wikipedia, “Characterization test,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

3. Mapbox, “pixelmatch,” github.com (opens in a new tab) (accessed 2026-10-13). ↩

4. Wikipedia, “Structural similarity index measure,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

5. Wikipedia, “QuickCheck,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩

6. fast-check, “What is property based testing?,” fast-check.dev (opens in a new tab) (accessed 2026-10-13). ↩

7. fast-check, “Introduction,” fast-check.dev (opens in a new tab) (accessed 2026-10-13). ↩

8. Wikipedia, “Mutation testing,” en.wikipedia.org (opens in a new tab) (accessed 2026-10-13). ↩