A comparison finishes and tells you the truth: 34 screenshots changed. What it does not tell you is whether that means a broken header on every page, or a blog listing that picked up a new article since the last run.
Both look identical in the results list. You find out which by opening screenshots one at a time.
AI Summary does that first pass for you. When a comparison completes, it looks at every screenshot that actually changed, then writes one short brief at the top of the comparison — split into Critical and Minor — so you know what you are walking into before you open a single image.
It is available on request. Skip to how to turn it on if that is all you need.
What it actually does
The summary is built in two stages.
Stage one: every changed screenshot, individually. For each page and breakpoint where the comparison found a difference, Diffy sends the baseline image and the comparison image to a vision model and asks it to describe what changed. It comes back with structured findings, not prose — each one carrying a location on the page in pixel coordinates, a type, a severity, and a description that includes before and after values where they can be read off the image.
Only screenshots that changed are analysed. Pages that came back identical are never sent.
Stage two: one brief for the whole comparison. Those per-screenshot findings are then condensed into a single summary, with two rules that matter in practice: repeated patterns are grouped into one line rather than repeated per page, and each bullet is kept to a short phrase. A cookie banner that appeared on five pages reads as one item, not five.
How it classifies a change
Every finding is tagged with a type, and the type determines the severity. This is a fixed taxonomy, not a judgement call the model makes freshly each time:
| Severity | Types | What they mean |
|---|---|---|
| Critical | section-change | A whole section or component appeared or disappeared |
padding | Spacing or padding around elements changed | |
color | A background, text or border colour changed | |
breadcrumb | Breadcrumb navigation changed in content, structure or visibility | |
| Minor | layout | Elements moved, resized or reflowed |
content | Text, images or data changed | |
style | Font or shadow changes | |
rendering | Anti-aliasing or sub-pixel differences |
The split is deliberately blunt. A section that vanished is something you need to look at now; text that changed on a page that publishes new text every day usually is not. Sorting those two into different halves of a short list is most of the value.
The output follows a fixed shape. One issue in the whole comparison gets a sentence or two, no headings. Two or more get a Critical: section followed by a Minor: section. If nothing critical was found, the summary says so explicitly — a line reading “Critical: no issues found.” — rather than leaving you to infer it from an absence. If there is nothing minor, that section is simply left out.
Giving it context about your site
Once AI Summary is switched on for a project, an AI tab appears in that project’s settings, under Advanced. It contains one field you can edit: Diff AI summary instruction. If the project does not have the feature yet, the tab is not there at all — so if you go looking for it and find nothing, that is why.
Whatever you write in that field is passed to the model twice: once when it looks at each individual screenshot, and again when it writes the final brief.
This is where you tell it the things that are obvious to you and invisible in a screenshot. That the right rail is a live feed and always changes. That this client’s brand colour was deliberately updated last week. That the numbers in the dashboard are seeded from a demo database. Context that would otherwise get reported to you as a finding, every run, forever.
The field is plain text, there is no syntax to learn, and anyone on your team who can edit the project can change it — turning the feature on is on our side, but the instruction is yours.
What it does not do
Worth being straight about the boundaries.
It does not replace the comparison. The summary is an extra panel at the top of the comparison, not a substitute for it. The screenshots, the diff overlay, the approve buttons and the vertical-shift-aware comparison algorithm all work exactly as they did. The summary tells you where to look; you still look.
It does not decide what is a bug. Severity here is a sorting mechanism, not a verdict. A color change is classified Critical because colour changes are usually intentional-or-alarming rather than noise — but if you just shipped a rebrand, every one of them is expected. You approve or reject; the summary just orders the queue.
It skips very large comparisons. If a comparison contains more than 80 changed screenshots, it is skipped rather than summarised. A run that size is usually telling you something structural went wrong — a failed deploy, a login that stopped working, a stylesheet that did not load — and a bulleted list is not the tool for that.
It only runs when there is something to summarise. No changes, no summary.
It is a language model reading images. It can miss things and it can describe something imprecisely. That is why every summary carries a thumbs up and thumbs down — “Was it helpful?” — directly in the panel. We read that feedback; it is how the feature gets better. You can also dismiss a summary once you have acted on it.
Where it appears
Once enabled for a project, it is automatic. Run a comparison as usual — manually, from the CLI, from CI, or from scheduled monitoring — and if the comparison finds changes, the summary appears at the top of the comparison view once it has been generated. There is no button to press and nothing to configure per run.
It is generated after the comparison completes rather than as part of it, so on a large comparison the panel can show up a little after the results do.
How to get it
AI Summary is enabled per project, by us, on request. The toggle that turns it on lives behind our admin, not in your project settings — so there is nothing for you to switch on yourself today.
Email support@diffy.website with the project — or projects — you want it on, and we will switch it on for you. If you are not sure whether it fits your workflow, say what your comparisons usually look like and we will tell you honestly whether it will help.
We are running it this way on purpose. It is a new feature, it costs real money to run per comparison, and we would rather turn it on for people who want it and hear what they think than ship it everywhere and guess. The feedback from those projects is what decides how it ships more broadly.
If you are interested in the other side of the same problem — screenshots that are wrong before any comparison happens — we wrote about AI screenshot checks, which catch cookie banners, CDN challenge pages and unloaded media before they reach your results.