AI Visual Testing: What It Actually Does (and Doesn't)

Every visual testing tool on the market now has AI on its homepage. Ours included.

That makes the category hard to shop for, because “AI” is being used for at least four different things: the algorithm that finds differences, the layer that describes them, the assistant that writes your config, and the agent that reads the results. Some of those are real and useful. One of them is not what it sounds like.

We ship two AI features and we run them in production, so we have a reasonably close view of where a vision model helps and where it quietly costs you time. This article is that view — what AI visual testing actually does, what it does not do, and what to ask a vendor before you believe the number on their pricing page.

The part that was never AI

Start with the thing visual regression testing is built on: deciding that two screenshots are not the same. That is arithmetic, and in most tools it still is.

In Diffy, a comparison runs in two deterministic stages before any model is involved. The first downloads both screenshots, normalises them to the same dimensions, applies a sensitivity threshold to suppress noise, and runs an ImageMagick comparison that produces a numeric score. If that score is zero, the comparison is finished — nothing changed. If it is above zero, a second stage groups the changed pixels into rectangular regions and returns their bounding-box coordinates and the percentage of the image affected.

Other tools describe their core the same way. Chromatic’s documentation states that its default threshold is .063, that it “measures the color distance in a 3D color space (YIQ)”, and that “by default, Chromatic detects anti-aliased pixels and ignores them to prevent false positives” (Chromatic docs). How that stacks up against Diffy’s pipeline is in Diffy vs Chromatic.

Applitools, which built its positioning around the term Visual AI, is the interesting case. It still ships a match level called Exact that “causes the Eyes matching engine to do a pixel to pixel comparison of the checkpoint and baseline images” — and in the same breath tells you not to use it: “It is not recommended for ordinary verification purposes,” because it catches rendering anomalies invisible to the human eye (Applitools docs). Raw pixel equality is there, and the vendor has deliberately moved its customers off it.

What replaced it is worth reading in Applitools’ own words, because it is more precise than the marketing around it: “Applitools Visual AI is made up of a network of hundreds of algorithms implemented using different tools and approaches ranging from hand-coded rule based algorithms to deep learning” (Applitools). Hundreds of algorithms, some hand-coded rules, some machine learning. That is computer vision, and it is a real engineering achievement. It is not a language model looking at your homepage and forming an opinion. We go through the practical differences in Diffy vs Applitools.

The distinction matters for one practical reason: deterministic detection is reproducible. Run the same two screenshots through a pixel comparison twice and you get the same score twice. That is what lets a comparison be a CI gate. The more detection depends on a generative model, the more your build status inherits that model’s variance — and nobody wants a pull request that passes on the second try for no reason.

Where AI actually earns its place in visual testing

So if detection stays deterministic, what is the model for? Three jobs, in our experience.

1. Triage — telling you what the changes mean

This is the big one, and it is a genuinely different problem from detection.

A comparison comes back and tells you the truth: 34 screenshots changed. What it cannot tell you is whether that is a broken header on every page or a blog listing that picked up a new article. Both look identical in a results grid. You find out by opening screenshots one at a time.

Diffy’s AI Summary does that first pass. It runs in two stages. For every page and breakpoint where the pixel comparison already found a change, it sends the baseline and comparison images to a vision model and gets back structured findings — each with a location in pixel coordinates, a type, a severity and a description. A second call then condenses those findings into one brief at the top of the comparison, split into Critical and Minor, with each bullet linked to the URLs it came from and, where a finding does not apply to every breakpoint, to the specific widths where it does.

Note the order of operations, because it is the point: the model never decides whether your build passes. It only looks at pairs a deterministic comparison already flagged, and nothing it writes changes the comparison’s own result. Pages that came back identical are never sent to a model at all.

Severity is not a fresh judgement call each run. The type determines the severity, from a fixed list:

A whole section disappearing is always Critical. Text changing is always Minor. That is deliberately blunt, and it is blunt because a model that re-derives its own severity scale on every run gives you a summary you cannot compare to last week’s.

Percy is making the stronger bet here, and it is worth knowing about. BrowserStack’s baseline pipeline is deterministic — “Percy captures DOM snapshots and assets, renders and compares them to previously generated snapshots to detect visual changes” (Percy features) — but its Visual Review Agent adds an AI diff that goes beyond pixel-by-pixel comparison, and the standard diff becomes the fallback: “AI diff may not be available when a comparison contains a very large number of visual changes or when one or more changes are extremely large. In such cases, Percy falls back to standard visual diffs” (BrowserStack docs). On top of that it labels changes “Irregular” (likely bugs) or “Valid” (likely intended). That is a different design from ours, and the fallback is the part to note: a Percy comparison can be produced by two different mechanisms depending on its size. The rest of the comparison with Diffy is in Diffy vs Percy.

2. Checking the screenshot, not the comparison

The second job is upstream of everything else: is this screenshot even a fair picture of the page?

A cookie banner fires on the new capture and not the baseline. Cloudflare decides the screenshot bot looks suspicious and serves a challenge page. A hero image never resolves. In each case the comparison is technically correct and completely useless — it is about the capture, not your deployment.

This is a good fit for a vision model because the questions are closed. Diffy’s screenshot check asks exactly five: is a popup or banner blocking the content, are elements stretched or squashed, is content rendered outside its layout, is there a CDN challenge page, did media fail to load. Fixed questions, structured answers. It is a checklist, not an open-ended “what’s wrong with this page”, and that is why the output is consistent enough to act on.

It is also deliberately small: the check runs at most once per project per calendar day, on a single screenshot. It is there to catch systematic problems — the CDN that started challenging us, the banner that appeared after a deploy — not to audit every page in every run. Of its five findings, only two (popup detected and CDN protection) surface to customers as recommendations; the other three are recorded and reviewed on our side.

3. Explaining the result to another machine

The newest job, and the one that will probably matter most in a year: AI coding agents write UI code, and something has to tell them what the UI now looks like.

A visual comparison is an unusually good feedback signal for an agent, because it is grounded — it is not a model’s opinion of the code, it is a screenshot of the rendered result. We wrote about running Diffy from Claude Code, where the agent makes a change, triggers a comparison, and gets the changed pages back. Here the AI is not judging the pixels; it is the consumer of a deterministic result.

What AI does not do

This is the half of the category that tends to get skipped.

It does not know what is correct. A screenshot contains no information about intent. A model can tell you the hero background went from #fff to #f0f0f0. It cannot tell you whether that was last week’s approved rebrand or someone’s stray CSS variable, because the answer lives in a Jira ticket it has never seen. Every severity label in every tool, ours included, is a way of ordering a review queue, not a verdict. The approve button stays yours.

It is not free, and it is not instant. This is the cost nobody puts on the pricing page. Our AI summary generates an ETA from measured averages, and those averages are: roughly 36 seconds for a comparison with up to 10 changed screenshots, about seven minutes at 20, and around twenty minutes at 80. Above 80 changed screenshots we skip the summary entirely rather than run it — a comparison that size is usually telling you something structural broke, and a bulleted list is the wrong tool for that. Percy documents a comparable ceiling: its AI diff “may not be available when a comparison contains a very large number of visual changes,” in which case “Percy falls back to standard visual diffs.” Every vendor has a cliff. Ask where it is.

It is not consistent. Same two screenshots, two runs, two slightly different sentences. Fine for a summary you read once. Not fine as the thing that decides whether a build passes.

It is worse at fine-grained UI defects than the marketing suggests. This is measurable now. UI-Lens, a benchmark presented at CVPR 2026, evaluated general multimodal models against 4,759 expert-annotated Chinese-language interfaces across six defect categories, and reported that “for tasks requiring fine-grained element boundary understanding, performance is near random, with task-average F1 scores of 20.36% and 31.21% on Text Overflow and Container Overlap, respectively”, and that for sequential interface semantic consistency “the task-average F1 score is only 10.61%, indicating severe underperformance” (UI-Lens, CVPR 2026). Two caveats worth stating: that benchmark tests general-purpose models rather than any specific commercial product, and it is built around Chinese-language UI scenarios rather than Western web pages. A tuned pipeline with a fixed checklist does better than an open-ended prompt. But it is a useful corrective to “AI sees your UI the way a human does” — text overflowing its container is exactly the kind of bug a person spots instantly, and it is where these models score close to chance.

The vendors say so too, in the docs if not on the homepage. BrowserStack’s documentation for the Percy Visual Review Agent states plainly: “AI may occasionally miss or misinterpret changes. Please review before approving,” and, separately, “Classification accuracy may vary if a snapshot has a significant number of changes.” That is a vendor telling you, in its product documentation, that the AI is an aid and not an authority. Believe the docs over the landing page.

How to read an AI claim on a pricing page

Four questions will tell you what you are actually buying.

On that third row: accuracy percentages are everywhere in this category, and I have not found one published with a methodology attached. Treat them as marketing until a vendor shows the benchmark. That includes any number you might one day see from us.

A workflow that uses AI without trusting it

What works, in practice:

1. Keep detection deterministic. Pixel comparison plus configurable sensitivity decides what changed. That is the number your CI gate reads.
2. Fix false positives at the source, not with a model. Mask the carousel, stabilise the dynamic elements, exclude the ad slot, pin the test data. A model asked to ignore your ticker on every run is a recurring cost; a mask is a one-time fix. We wrote a whole guide on avoiding false positives.
3. Let AI order the queue. Read the summary first to decide where to look, then look. It changes the order you open screenshots in; it does not change what you approve.
4. Give it the context a screenshot can’t carry. Both of our AI features take a per-project instruction. For AI Summary it is a field you edit yourself, under Project Settings → Advanced → AI, once the feature is switched on for the project; for the screenshot check, ask support and we will add it. In our experience it is the highest-leverage change you can make: “the right rail is a live feed”, “the brand colour changed deliberately last week”.
5. Keep a human on approve. Always. Every vendor we checked says the same thing in its documentation, whatever the homepage says.

Frequently asked questions

Does AI replace pixel comparison in visual regression testing?

In most tools, no. Chromatic documents a pixel comparison with a colour-distance threshold. Applitools ships a pixel-exact match level alongside its Visual AI, though it recommends against using it. Diffy’s AI Summary runs ImageMagick first and only calls a model on screenshots that already changed. Percy is the exception worth knowing about: its Visual Review Agent produces an AI diff in place of the standard comparison, with the deterministic diff as a documented fallback.

Can AI decide whether a visual change is a bug?

No tool can, because the screenshot does not contain the intent. AI can help order a review queue — Percy classifies changes as Irregular or Valid, which is a guess at intent, while we sort by change type into Critical and Minor, which is not a guess at anything. Neither is a verdict. The decision stays with the person who knows what was supposed to ship.

Is AI visual testing more accurate than traditional visual testing?

It is accurate at a different task. Deterministic comparison is more reliable at detecting that something changed; a model is better at describing what changed in words. Published accuracy figures in this category generally come without a methodology, and the independent benchmark evidence on fine-grained UI defect detection is not flattering.

Does Diffy’s AI cost extra?

Screenshot checks run on all projects at no extra cost. AI Summary is enabled per project on request, because it costs real money per comparison and we would rather turn it on for teams who want it. Email support@diffy.website with the project.

Will AI make visual regression testing unnecessary?

It is pushing the other way. AI agents generate more UI code, faster, with fewer people reading every diff — which makes an independent, deterministic check on what the rendered page actually looks like more valuable, not less.

If you want the version of this you can run rather than read: start a free Diffy trial, point it at your staging and production environments, and see what a comparison turns up. The pixel comparison does the finding. Decide for yourself whether the summary on top of it saves you time.