21 Commits

Author SHA1 Message Date
Pavel Feldman
20699c36ba
chore: simplify types handling in toHaveScreenshot (#29374) 2024-02-05 19:07:30 -08:00
Max Schmitt
edd003c230
chore: validate expected image buffer when comparing images (#23030)
Fixes https://github.com/microsoft/playwright/issues/23012
2023-05-15 23:32:16 +02:00
Andrey Lushnikov
fdcd7b549d
chore: mark comparator option back as experimental (#20816)
This reverts commit 303c5998f8b747e94ac80e2e02c53b47ded11322.

Reason for revert: I tried enabling `ssim-cie94` by default on
ionic-framework test suite, and it proves to be overly strict for their
usecase.
2023-02-13 11:11:44 -08:00
Andrey Lushnikov
303c5998f8
feat: release "comparator" option from experiment (#20720)
The option defines a comparator to be used to compare images.
Possible values are `"pixelmatch"` and `"ssim-cie94"`.

Note: This reverts commit 8167f8bf548308ad8c6f1188508aadee84f26023.
2023-02-07 08:51:48 -08:00
Dmitry Gozman
b700c08dc5
feat(screenshots): when actual and expected have different sizes, pad and produce the diff image (#20208)
Also show sizes in the html report to easier spot the size mismatch
issue.

<img width="1030" alt="diff"
src="https://user-images.githubusercontent.com/9881434/213327632-b8fcd69c-8d08-460c-9de1-b5f4f8c56359.png">

Fixes #15802.
2023-01-20 19:41:43 -08:00
Andrey Lushnikov
8167f8bf54
chore: hide "comparator" option from documentation and types (#19441) 2022-12-14 09:58:19 -08:00
Andrey Lushnikov
cac67fb94f
chore: do not use "threshold" option for the ssim-cie94 comparator (#19314) 2022-12-06 17:03:13 -08:00
Dmitry Gozman
d294c5da33
chore: roll stable-test-runner to 1.28.1 (#19310)
This required `allowJs: false` in our `tsconfig.json` due to the
following error:

```

Error: Cannot find module './utilsBundleImpl'
Require stack:
- <playwright>/packages/playwright-core/src/utilsBundle.ts
- <playwright>/packages/playwright-test/lib/reporters/html.js
- <playwright>/tests/playwright-test/reporter-html.spec.ts
- <playwright>/tests/playwright-test/stable-test-runner/node_modules/@playwright/test/lib/loader.js
- <playwright>/tests/playwright-test/stable-test-runner/node_modules/@playwright/test/lib/runner.js
- <playwright>/tests/playwright-test/stable-test-runner/node_modules/@playwright/test/lib/cli.js
- <playwright>/tests/playwright-test/stable-test-runner/node_modules/playwright-core/lib/cli/cli.js
- <playwright>/tests/playwright-test/stable-test-runner/node_modules/playwright-core/cli.js
- <playwright>/tests/playwright-test/stable-test-runner/node_modules/@playwright/test/cli.js

   at ../../packages/playwright-core/src/utilsBundle.ts:20

  18 | import path from 'path';
  19 |
> 20 | export const colors: typeof import('../bundles/utils/node_modules/colors/safe') = require('./utilsBundleImpl').colors;
     |                                                                                   ^
  21 | export const debug: typeof import('../bundles/utils/node_modules/@types/debug') = require('./utilsBundleImpl').debug;
  22 | export const getProxyForUrl: typeof import('../bundles/utils/node_modules/@types/proxy-from-env').getProxyForUrl = require('./utilsBundleImpl').getProxyForUrl;
  23 | export const HttpsProxyAgent: typeof import('../bundles/utils/node_modules/https-proxy-agent').HttpsProxyAgent = require('./utilsBundleImpl').HttpsProxyAgent;

    at Object.<anonymous> (<playwright>/packages/playwright-core/src/utilsBundle.ts:20:83)
    at Object.<anonymous> (<playwright>/packages/playwright-test/src/reporters/html.ts:17:1)

```
2022-12-06 15:46:19 -08:00
Andrey Lushnikov
6d82460a02
feat: implement a new image comparison function (#19166)
This patch implements a new image comparison function, codenamed
"ssim-cie94". The goal of the new comparison function is to cancel out
browser non-determenistic rendering.

To use the new comparison function:

```ts
await expect(page).toHaveScreenshot({
  comparator: 'ssim-cie94',
});
```

As of Nov 30, 2022, we identified the following sources of
non-determenistic rendering for Chromium:
- Anti-aliasing for certain shapes might be different due to the
  way skia rasterizes certain shapes.
- Color blending might be different on `x86` and `aarch64`
architectures.

The new function employs a few heuristics to fight these
differences.

Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`:
1. If the [CIE94] metric is less then 1.0, then we consider these pixels
   **EQUAL**. (The value `1.0` is the [just-noticeable difference] for
   [CIE94].). Otherwise, proceed to next step.
1. If all the 8 neighbors of the first pixel match its color, or
   if the 8 neighbors of the second pixel match its color, then these
   pixels are **DIFFERENT**. (In case of anti-aliasing, some of the
   direct neighbors have to be blended up or down.) Otherwise, proceed
   to next step.
1. If SSIM in some locality around the different pixels is more than
   0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them
   as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be
   very close to 1.0).

[CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94
[just-noticeable difference]:
https://en.wikipedia.org/wiki/Just-noticeable_difference
2022-12-02 15:22:05 -08:00
Andrey Lushnikov
dfb4ad388a
feat: support custom png comparator (#18689)
This way we might experiment with different custom PNG comparators
for VRT.
2022-11-10 16:16:05 -08:00
Ross Wollman
5f03bd9477
chore(test-runner): increase jpeg-js max mem allowance (#15381)
Resolves #14255.

PNGs should be preferred as they are deterministic, but increasing this
limit should be fine for JPEG users. (Looking through jpeg-js source code, the actual memory allocation is based on the size of the image—so unless a user is hitting this limit already—this should not impact the memory consumption of Playwright.
2022-07-05 13:25:54 -07:00
Pavel Feldman
7989427ea6
chore: bundle core deps (#13621) 2022-04-18 20:20:49 -07:00
Pavel Feldman
63ea81ec54
chore: make bundles typed (#13620) 2022-04-18 17:50:25 -07:00
Pavel Feldman
5f843c347d
chore: bundle pixelmatch, it has not changed for 2 years (#13585) 2022-04-15 16:58:31 -07:00
Andrey Lushnikov
7f51336068
fix: allow toMatchSnapshot to use text comparator for text data (#12934)
This was regressed awhile ago.

In v1.17 we shipped the following code: 30e15ad36f/packages/playwright-test/src/matchers/golden.ts (L122-L131)

`toMatchSnapshot` should fallback to text comparator in case of
unknown extension and string data.

Fixes #12862
2022-03-21 16:42:21 -07:00
Andrey Lushnikov
c18077c0de
feat(toHaveScreenshot): align screenshot generation & comparison (#12812)
This patch aligns the strategies that are used to generate new
screnshot expectations and to compare screenshot expectations against
baseline.

With this patch, `toHaveScreenshot` will:
- when generating a new expectation: will wait for 2 consecutive
  screenshots to match and accept the last one as expectation.
- when given an expectation:
  * will compare first screenshot against expectation. If matches,
    resolve successfully
  * if first screenshot doesn't match, then wait for 2 consecutive
    screenshots to match and then compare last screenshot with the
    expectation.

An example of a new detailed call log:

```
  1) a.spec.ts:3:1 › should work ===================================================================

    Error: Screenshot comparison failed:

      20000 pixels (ratio 0.03 of all image pixels) are different

    Call log:
      - expect.toHaveScreenshot with timeout 5000ms
      -   verifying given screenshot expectation
      - fast-path: checking first screenshot to match expectation
      - taking page screenshot
      -   disabled all CSS animations
      -   waiting for fonts to load...
      -   fonts in all frames are loaded
      - fast-path failed: first screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
      - waiting for 2 consecutive screenshots to match
      - waiting 100ms before taking screenshot
      - taking page screenshot
      -   disabled all CSS animations
      -   waiting for fonts to load...
      -   fonts in all frames are loaded
      - 2 consecutive screenshots matched
      - final screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
      - 20000 pixels (ratio 0.03 of all image pixels) are different

    Expected: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-expected.png
    Received: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-actual.png
        Diff: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-diff.png

      3 | test('should work', async ({ page }) => {
      4 |   await page.goto('file:///Users/andreylushnikov/prog/playwright/tests/assets/rotate-z.html');
    > 5 |   await expect(page).toHaveScreenshot();
        |                      ^
      6 | });
      7 |
```
2022-03-21 15:10:33 -07:00
Andrey Lushnikov
d7648d390c
fix(toHaveScreenshot): round ratio up (#12540)
This way the ratio value could be used right away in configuation.
2022-03-04 18:55:48 -08:00
Andrey Lushnikov
1f5cfcaeec
fix(to-have-screenshot): show number of bad pixels in mismatched error (#12502)
Drive-by: do not show empty call log.

References #12441
2022-03-04 11:01:05 -08:00
Andrey Lushnikov
279e579097
chore: address pixelCount and pixelRatio review comments (#12501)
- documentation fixes
- rename pixelCount to maxDiffPixels
- rename pixelRatio to maxDiffPixelRatio
2022-03-03 23:17:31 -08:00
Andrey Lushnikov
396d920145
feat(test-runner): implement expect(pageOrLocator).toHaveScreenshot (#12242)
Fixes #9938
2022-02-28 12:25:59 -08:00
Andrey Lushnikov
0682672242
chore: move comparator logic to playwright-core (#12232)
This will enable implementation of `toHaveScreenshot` on the
server-side.

Drive-by: drop blink-diff

References #9938
2022-02-18 14:39:17 -08:00