Looking into a possible compare UI here: https://webcomponents.dev/edit/yXxe3Z8mQ6uCfZXTFulA/src/index.ts

Not sure how we'd want to attach "approve" workflows, especially for tests that run in CI, but would love to hear more of your thoughts on that!

uh that looks really nice 💪

for approval, I could see a workflow similar to how we use changeset now. That could look something like this

1. PR with changes to the UI => so tests for it fail
2. You add a file like imagechange/something-something.md
3. In that md file you say which images/urls/imagestates are allowed to change and you add a description of the change
4. PR runs again... all UI changes are approved by the added file => CI green
5. Optionally: This could also enable a changelog for all visual changes (no images or only thumbnails to save size)

There are couple of things that can go awry why working with in repo image caches:

1. Screenshot churn can increase the size of a repo and make it slow to work with. Still unsure whether this is the personal opinion of the colleague that brought it up or worth worrying about for real. I also work with multiple GB repos that make even the 4-500MB of screenshots that I've gotten up to feel like not the part of a project that would make this an issue.
2. If you check in the images, the it's possible that you actually commit the images. Visual regression is very brittle to things like OS (sometimes even version), browser version, font availability, processor/RAM availability, so they can shift and change in a local environment. If you commit those changes then it can be next to impossible to recreate them. It's very important that you always build in the same context (whether that's CI, requires OS build, docker, etc.) and allowing those images to the repo increases the possibility that that's not true.
3. Being in repo means that any tooling that you leverage around those screenshots also needs to have access to that repo. Maybe more a reality than an issue as these tools will need access to these images where ever they live, but ensuring they don't live next to your code reduces the access those tools require to give you value.

these are awesome points - would have never thought about them 🙈
thx for bringing them up 🤗

removing scrollbars and using a transparent caret I think are awesome solutions 👍

the others are really tricky 🤔

• system font => we could force a system font that is available on both systems (e.g. processing all css and then have a map of fonts to replace)
• font aliasing/smoothing => puh yeah that is going to be tough... we could try to force -webkit-font-smoothing: none; 🤔

I would recommend to check how existing libraries handle some of the problems listed above.
One good example is looks-same used by gemini and hermione.

In particular, it has antialiasingTolerance and ignoreCaret options.

I used this library for comparison and by experience the ignoreCaret option is not always working. But maybe the antialiasing tolerance is better

i've already been through the pain of finding a good VRT solution so you might be interested in my findings so you don't have to:

• pixelmatch with my own script launching puppeteer - font smoothing is a problem, and there's a lot of manual setup for this solution
• backstopjs - has its own report system, its an all-in-one solution really but yet again, font smoothing is a problem and there's a lot of manual setup
• looks-same with my own script launching puppeteer - slightly better than pixelmatch alone, unsurprisingly even with its "antialiasing tolerance" it has the font smoothing problem
• any of the above containerised - no font smoothing problem, everything works fine, seems overkill and a lot of manual setup to have a container just for consistent font smoothing though

i tried fiddling with chrome flags, matching OS versions, browser versions, etc. but there will always inevitably be some minor difference between two machines, especially a CI server and a local machine. so the font smoothing mismatch has proven to be a problem every time.

in the end, i used percy (now owned by browserstack) with a storybook plugin for it. this means it just uploads a static storybook to percy, percy does the visual diffing and produces a report in their dashboard.

this is one case where doing it myself wasn't worth it tbh.

of course it could still be worth implementing a looks-same solution here to give users the flexibility, but in the end i think a service is the right way to go and sits outside your test runner anyway.

thanks for sharing your experience here

we should definitely look into percy. I like reusing storybook stories inside tests, but limiting this to only tests is unfortunate

What do you mean by image data? The actual images? Tests are executed in the browser while the screenshots are taken server side.

What do you mean by image data? The actual images?

Yes. pdf is generated client side (pdfkit/pdfmake), converted to image (png) using pdf.js and compared using jest.

Here's one example:

https://github.com/foliojs/pdfkit/blob/master/tests/visual/helpers.js#L23-L25

pdf2png returns an array of images as buffer. In case of using browser, the image would be a Blob or Uint8Array

I've added the following PR to further the discussion around the possible API changes/additions that would be associated with the above: #1339

With a tool like Screener.io, new screenshots are marked as "new" and not accepted by default, which requires manual acceptance to add them to the baseline.

Which I think is the right path, something not being nothing, and new results should fail. If you want to hard update the baseline (which happens..., new project, major upgrade, etc.) then you can do so with the update flag.

The PR above would make that the default here, too.

@daKmoR do you know of any work being done on this feature?

Yeah, I pushed a branch with a real test that works except for the visualDiff part:
https://github.com/melink14/rikaikun/blob/538c7508b876cf14428bd8d55566eb8c1f155571/extension/test/rikaicontent_test.ts#L139

(configs are all at the top level if they're important)

I had uploaded it to see if it would work in the Github action environment in case the problem was my WSL2 local env being weird. (However, for some reason even more tests failed though it's hard to tell given #1549 log spam)

As I said though, it fails with just the code sample too so I could probably make a branch for that as well.

I did some more debugging and got it to successfully not hang in a Github action though it fails with:
There was no baseline image to compare against.

I could probably figure out how to make an action which can also update images but I'm surprised since documentation implied that the absence of baseline image would lead to baseline creation:

When you run a diff test for the first time, a baseline image is saved to screenshots/baseline/${browser}/${name}.png. Afterward, every time you do a diff it is compared to this baseline images.

Locally, it times out but the error I get is:
Error: Node is either not visible or not an HTMLElement

which probably is due to whatever limitations WSL gives me. It might be nice to fail fast with good error message when that happens though.

On the first run it will fail when there is no baseline to compare to, even as it creates a baseline for the next test. This is to clarify whether the new image is approved to enter the cache or not. Once approved and saved to your cache the test will work against that baseline going forward.

Re: "The Cache", the plugin in un-opinionated as to how you manage your own cache. A naive approach is to run once, commit the baseline images to your repo and then run again for regressions, though that means that your golden images are saved to your repo, which can, over time, increase the size of your repo, the speed at which this occurs depending on the size of your test suite. Other options, like saving in a separate repo, saving in an archive in your CI provider, etc. need to be handled by you in your project.

The Error: Node is either not visible or not an HTMLElement seems to be coming from Puppeteer, which means it's not something we could fail early on as it's not resolved till quite late in the test process. Are you sure you're building the test context correctly on local? I'm not sure that platform should effect this, however I don't know much about the file ownership situation between Windows and a related WSL install.

Thanks; that makes sense; though I would have expected some kind of success message if it also saved the screenshots (like No baseline detected, new baselines saved at /file/location).

Re 'error handling', I had assumed it was a matter of wrapping the calls to the pupeeteer page object since it seemed to be called from the handler. Either way, WSL2 is known to have various problems when it comes to puppeteer and browsers so I'll either wait for more updates to the windows linux kernel or figure something else out. Given it seems to work on regular linux in Github CI environment it's probably something out of scope for the plugin at least.

Thanks for the help and chiming in.

I worked on this some more and got it working locally as well:
I found that even running chrome --headless from the command line would cause chrome to hang so I worked on solving that problem. I unfortunately did several things before testing to see if it worked so I haven't isolated the fix but essentially I installed an X server and all the GUI libs which were needed to work with it. After that everything worked normally (and I guess as a bonus headful mode will also work!)

My project isn't so big so storing the images in Github is probably fine but reviewing diffs sounds like it will be a pain in Github; my next plan is to run this tool with update flag always and then use percy as my remote cache. I think this could be done by just adding a command for saveBaseline which will essentialy capture screenshots and upload them to percy for diff processing.

I actually tried to integrate percy directly but it does a weird thing where it tries to reload the test page causing many things to break. 😅

Thanks again!

Hi, currently it is possible by passing import.meta.url

Here is a code that we use to get path to a test file from it. The test can look like this:

it('disabled', async () => {
  element.disabled = true;
  await visualDiff(div, `${import.meta.url}_disabled`);
});

@web-padawan interesting! This has given me an idea...

Since testFile is available in WTR plugins, I wonder whether this could be added to GetNameArgs? That way you wouldn't need to remember to pass import.meta.url in each test, it could be calculated entirely within the extension points like getBaselineName etc.

export interface GetNameArgs {
  browser: string;
  name: string;
+ testFile: string;
}

I would be happy to make a PR for this.

Also, your linked wtr-utils is full of useful snippets! I'm going to dive into that. Thanks

Made a PR here #1657

here is a snippet I've done to get a clip of around 2 elements (selector and endClipSelector) with a 8px margin:

async function getClip({ page, selector, endClipSelector }) {
  const clip = await page.evaluate(
    ([sel, endSel]) => {
      const el = document.querySelector(sel);

      if (!el) {
        throw new Error(`An el matching selector \`${sel}\` wasn't found`);
      }

      const range = document.createRange();

      range.setStartBefore(el);
      range.setEndAfter(endSel ? document.querySelector(endSel) : el);

      const rect = range.getBoundingClientRect();
      const margin = 8;

      const x = rect.left - margin < 0 ? 0 : rect.left - margin;
      const y = rect.top - margin < 0 ? 0 : rect.top - margin;

      return {
        x,
        y,
        width: rect.width + (rect.left - x) * 2,
        height: rect.height + (rect.top - y) * 2,
      };
    },
    [selector, endClipSelector]
  );

  return clip;
}

        visualRegressionPlugin({
            // update is set to false so that the test fails for the first time. The modification to getFailedName causes the file to get overwritten even though update is set to false
            update: false,
            baseDir: 'test-screenshots',
            getBaselineName: (args) => {
                return getTestFileName(args, '');
            },
            getDiffName: (args) => {
                return getTestFileName(args, 'diff');
            },
            getFailedName: (args) => {
                return getTestFileName(args, '');
            },
        }),

And then getTestFileName is this:

function getTestFileName(args, type) {
    // most of this is just picking a screenshot structure that I like
    const screenshotDir = relative(
        join(rootDir, 'src'),
        args.testFile.replace(/\.[jt]sx?$/, ''),
    );
    // the real magic is here, where extension, and ultimately screenshotName, are identical for both failed and baseline files 
    const extension = `${type ? `${type}.` : ''}png`;
    const screenshotName = `${
        args.name
    }.${process.platform.toLowerCase()}.${args.browser.toLowerCase()}.${extension}`;
    const dirs = [
        screenshotDir,
        args.name,
        type === 'diff' ? 'failure-diff' : '',
    ].filter((a) => a);
    return join(...dirs, screenshotName);
}

Here's my source code: https://github.com/electrovir/tdd-html-challenges/blob/main/.virmator/web-test-runner.config.mjs

That's a great insight. having baseline and failed names be the same is an easy way to get this functionality! Given source control makes diffs easy to see and I never want to check-in failed screenshots it seems like there's no downside with this unless you need to do offline processing.

Could you guys also share the github actions you are running?

Mine is a simple one that runs tests and then if there's a failure uploads new screenshots for comparison. Here's the relevant section: https://github.com/melink14/rikaikun/blob/5b835f5f40bcb01f50a47cfcc936a6eb6d35707c/.github/workflows/presubmit.yml#L42

Thank you very much. That is enough inspiration to get me going :)