Visual regression testing when an agent changed the UI

Stefan-Iulian Tesoi · 11 October 2026 · 6 min read

Compare screenshots of the screens it touched against approved baselines, and have a person look at every image that differs. The dangerous step is the update: an agent that regenerates a baseline to make a failing comparison pass has deleted the evidence, so a baseline change needs the same review as code.

That is visual regression testing applied to agent work. The tooling is mature; what has changed is who runs the update command.

What is visual regression testing?

A test that renders a screen, takes a screenshot and compares it pixel by pixel against a baseline image somebody approved earlier. A difference beyond the tolerance fails the test.

Playwright builds it in as expect(page).toHaveScreenshot(), which is what most teams mean by Playwright screenshot testing. The first run writes the reference images, later runs compare against them, and the default tolerance is a per-pixel colour threshold of 0.2 on a scale from 0 to 1. Storybook takes the idea down to components: each story is one component in one state, and Chromatic, made by the Storybook maintainers, screenshots every story in each build. Percy, from BrowserStack, does the same for pages and components through its own SDKs.

The tools differ most in where the baseline lives, and so in who can change it:

The last column is the one that matters once a coding agent is doing the work.

What does a unit test miss in a UI change?

Everything that only exists once the page is laid out. A unit test can assert that a component renders the right text and calls the right handler. It cannot see that the text now wraps onto three lines, that a button has moved below the fold, or that a modal opens behind the header.

Our own marketing site had a page that scrolled sideways on a 320-pixel phone, and nothing in the build noticed. The document was 326 pixels wide: one button label, "Connect Slack in Laimonade", could not wrap, and six pixels of overflow made the whole page pan. A script that drives Chrome and compares scrollWidth with clientWidth on every route caught it, but only when someone ran it, because it needs Chrome and was never part of the gate.

Agents make this class of defect more common. A coding agent changing a shared component, such as a button or a spacing token, changes every screen that uses it, then checks the one screen the item named. The tests for that screen pass. The regression sits on a page the item never mentioned.

Why is updating the baseline the risky step?

Because the update command turns a failing check into a passing one without touching the code, and an agent told to make the suite green will find it.

In Playwright it is one flag: npx playwright test --update-snapshots rewrites the images that differ. Chromatic has --auto-accept-changes, usually reserved for the main branch. Both are legitimate, and both are the shortest path from red to green. An agent updating snapshot tests to get a pass has changed what the test means, not just its result, and every later run compares against the regression.

A baseline is a record of what a person approved. An agent that regenerates it has not passed the test. It has replaced the person.

Three rules keep the update honest:

1. The agent does not run the update command. Say so in the rules file, and enforce it where you can: a CI check that fails when baseline images change without an approval label.
2. Baseline changes ship in their own commit. A reviewer sees the image diff beside the code that caused it, not folded into a 40-file change.
3. Every changed baseline is listed in the hand-back with its reason. "Header spacing changed on 14 screens because the item changed the spacing token" can be reviewed. "Updated snapshots" cannot.

A green run can also mean nothing rendered. While our admin app's overflow script was being written, an early version passed an error screen as a page that fitted: it had six elements against a liveness floor of five. The script now fails when fifteen routes render only two distinct pages between them, because an error screen fits any viewport.

Which screens are worth a baseline?

The screens where a regression costs the most, and the shared components behind them, not every route in the application.

A suite of 400 full-page screenshots fails on every intended change, and a team that sees 60 diffs per pull request stops reading them. Start with a short list:

• Sign-up, sign-in and checkout, or whatever the money path is.
• Each shared component in its main states: empty, loading, error, overflowing text.
• The narrowest supported viewport, usually 320 or 375 pixels, for every page with a desktop baseline.
• Dense screens such as tables and dashboards, where overflow hides.

Mask whatever changes on its own. Timestamps, avatars and charts fed by live data produce a diff on every run, and Playwright's mask option paints a box over a locator before comparing.

How do you review image diffs without reading every pixel?

Read the list of changed screens first, then ask one question of each image: did the item ask for this?

A change the item asked for, on a screen the item named, is approved quickly. A change on a screen the item never mentioned is the regression the suite exists to catch, and it gets read carefully. Most tools show the baseline, the new image and a highlighted overlay side by side, which makes ten seconds per screen realistic. That is how a person can verify UI changes at the pace an agent produces them.

The definition of done for agent work applies unchanged: the agent's report is a claim, the image diff is evidence, and someone who did not make the change decides. An agent can look at its own screenshots, but its reading of whether they look right is the same reading that produced the change, which is why tests an agent writes for its own work prove less than they appear to.

In Laimonade, an agent handing work back attaches its screenshots to the item and lists, in a separate field, what a person must still look at. What evidence a coding agent should return covers the rest of that hand-back.

Frequently asked questions

Do screenshots count as evidence in an agent's hand-back?

Yes, for any criterion about how something looks, and they are often the only evidence available. A screenshot proves the screen rendered and shows what it rendered. It does not prove the data behind it was real or that other screens are unchanged, so pair it with the state that produced it and a comparison run across the shared components.

How do you stop fonts and animations producing false diffs?

Fix the environment and freeze the page. Run comparisons in the same container image every time, because font rendering differs between macOS, Linux and Windows, which is why Playwright names each baseline by browser and platform. Playwright's toHaveScreenshot disables animations and hides the text caret by default. Mask timestamps and live data, and let the threshold absorb anti-aliasing.

Should baseline images live in the repository?

For Playwright, yes. Committed images give every change a reviewable diff and tie each baseline to the code that produced it. Chromatic and Percy keep baselines on their side and track them against git branches, which moves approval into a web app where a person accepts each change.