When merging blob reports test ids are patched to make sure there is no
collision when merging reports that might have overlapping test ids.
However, even if you were merging reports that had no overlapping ids,
all test ids will be modified, which is an undesirable side effect.
This PR only modify test ids when the same test id has already been used
in a previous blob report.
----
This change is also part of
https://github.com/microsoft/playwright/pull/30962
The spirit of this change is reverting #23153. Since that time, we have
moved tracing and `artifactsDir` lifetime into the test runner, so the
reason for revert is mitigated.
Fixes#30287, fixes#30718, fixes#30959.
This includes two major changes:
- reuse `SerializedFS` for live test runner tracing;
- merge scheduled `appendFile` operations into a single `fs` call.
In some cases, this improves performance of UI mode by 61% and
performance of `trace: on` mode by 38%. Note that performance
improvement on the average test will not be as noticeable.
References #30875, #30635.
- Documents `PLAYWRIGHT_FORCE_TTY` and `FORCE_COLOR` across terminal
reporters.
- New `PLAYWRIGHT_LIST_PRINT_STEPS`. Removes undocumented test-only
`PW_TEST_DEBUG_REPORTERS_PRINT_STEPS`.
- Replaces `PLAYWRIGHT_HTML_REPORT` with `PLAYWRIGHT_HTML_OUTPUT_DIR`
and `PW_TEST_HTML_REPORT_OPEN` with `PLAYWRIGHT_HTML_OPEN` for
consistency, supports older versions for backwards compatibility.
- New `PLAYWRIGHT_HTML_HOST`, `PLAYWRIGHT_HTML_PORT` and
`PLAYWRIGHT_HTML_ATTACHMENTS_BASE_URL`.
- New `PLAYWRIGHT_JUNIT_STRIP_ANSI` and
`PLAYWRIGHT_JUNIT_INCLUDE_PROJECT_IN_TEST_NAME`.
- Removes `PW_HTML_REPORT` that was set for unknown reason.
Previously, terminal reporters consulted `process.stdout.isTTY`. Now it
is possible to control the tty behavior:
- `PLAYWRIGHT_FORCE_TTY=0` or `PLAYWRIGHT_FORCE_TTY=false` to disable
TTY;
- `PLAYWRIGHT_FORCE_TTY=1` or `PLAYWRIGHT_FORCE_TTY=true` to enable TTY,
defaults to 100 columns when real columns are unavailable;
- `PLAYWRIGHT_FORCE_TTY=<number>` to force enable TTY and set the
columns.
Fixes#29422.
Previously, terminating worker always had a 30 seconds force exit.
Now, regular worker termination assumes that process will eventually
finish tearing down all the fixtures and exits. However, the
self-destruction routine keeps the 30 seconds timeout to avoid zombies.
Fixes#30504.
Implements feature requested in
https://github.com/microsoft/playwright/issues/30457
The test runner treats flaky tests as failures when the flag is enabled,
but still reports flaky tests as flaky in the reporting interface. It
feels like something worth discussing as this behaviour makes sense to
me, but looked a bit odd to @BJSS-russell-pollock when I ran this past
him.
Closes#30457.
Matching bu `apiName@wallTime` fails when two actions start at the same
time, e.g. two parallel api requests. Moreover, it results in trace
actions that have parent set to themselves, which in turn causes
infinite loop in the trace viewer. To avoid this problems we write
stepId explicitly to the library trace and use those step ids to find
corresponding test runner steps.
The stepId is passed via zone in case of expect, because the protocol
step is quite deep in the call chain after or explicitly in case of API
call steps created by the test runner instrumentation.
Pre Node.js `18.20.0`:
- `assert` is supported
Post Node.js `18.20.0`
- `assert` and `with` is supported.
Before https://github.com/microsoft/playwright/pull/30482 we kept
`asserts` in the JS code, Node.js was interpreting them. The `with`
keyword was not supported, this was what the PR was fixing.
After https://github.com/microsoft/playwright/pull/30482 Babel is
converting `assert` (deprecated) into `with` (successor) since we use
the `deprecatedAssertSyntax` option.
This means, that the minimum Node.js version we support in order to use
import attributes is now `18.20.0` where they added the `with` support.
This follows our principle of supporting only the latest minor release
for Node.js versions.
See here for the 18.20 changelog:
> #### Added support for import attributes
>
> Support has been added for import attributes, to replace the old
import
> assertions syntax. This will aid migration by making the new syntax
available
> across all currently supported Node.js release lines.
>
> This adds the `with` keyword which should be used in place of the
previous
> `assert` keyword, which will be removed in a future semver-major
Node.js
> release.
>
> For example,
>
> ```console
> import "foo" assert { ... }
> ```
>
> should be replaced with
>
> ```console
> import "foo" with { ... }
> ```
Fixes https://github.com/microsoft/playwright/pull/30482 - the tests
were a noop before, since they were tree-shaked by Babel.
Similarly to page.close, we pass test-runner specific reason to
facilitate better error messages.
```
1) a.test.ts:10:11 › test
Error: apiRequestContext.fetch: Fixture { request } from beforeAll cannot be reused in a test.
- Recommended fix: use a separate { request } in the test.
- Alternatively, manually create APIRequestContext in beforeAll and dispose it in afterAll.
See https://playwright.dev/docs/api-testing#sending-api-requests-from-ui-tests for more details.
9 |
10 | test('test', async () => {
> 11 | await context.fetch('http://example.com');
| ^
12 | });
13 |
```
Closes#29260.
When comparing `outputDir` and html-reporter `outputFolder`, we now make
sure that both paths end with a forward-slash.
Fixes#28677
---------
Co-authored-by: Georg Unterholzner <georg.unterholzner@dynatrace.com>
There are plenty of edge cases in this area:
- interrupted test run;
- did not run because of serial mode failure;
- failed before `test.skip()` call (e.g. in `beforeEach`) in one of the
retries;
- and more...
Related issues: #28322, #28321, #27455, #17652.
Prior changes: #27762, #26385, #28360, probably more.
There is still some duplication between `outcome()` and similar logic in
`base.ts`, which might be deduped in a follow-up.
Fixes#28322.
Generate 'global timeout' and 'max failures' errors in the runner. Avoid
reading `config.globalTimeout` and `config.maxFailures` in the base
reporters.
Reference https://github.com/microsoft/playwright/issues/29768
Instead of filtering tests assuming there are no two projects with same
name we always rebuild test tree from scratch and restore previos test
results in the list mode.
Fixes https://github.com/microsoft/playwright/issues/30396
