* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"
This reverts commit 73dee69558110b00916708dafaa99e121c6e7c94.
* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"
This reverts commit d744a87aeefe9c131ed10cedfff201efa1b5c156.
* Revert "chore(test runner): run hooks/modifiers as a part of the test (#12329)"
This reverts commit 47045ba48dd84d38bde17541c8a0a05cdfed7ffc.
chore(test runner): run hooks/modifiers as a part of the test
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
This way we control the timeout error message from the runner,
so that later on we can differentiate between test timeout, fixture
timeout and hook timeout.
Soft expects will still fail the test, but will not abort it's execution. As a consequence of this:
- `TestResult` now might have multiple errors, which is reflected with a new `testResult.erros: TestError[]` field.
- `TestInfo` now might have multiple errors as well, which is reflected with a new `testInfo.errors: TestError[]` field.
Fixes#7819
In several of the Playwright APIs, falsey values were not handled correctly. This changeset adds tests (and some fixes):
- route.continue: If options.postData was the empty string, the continue failed to override the post data.
- page.post (application/json with options.data: false|''|0|null): Raw falsey values were getting dropped (i.e. you can't do the equivalent of curl --header application/json … -d 'false'). This has been fixed with most values across all browsers, but an additional fix is needed for 'null' which the channel serializer treats extra specially.
- testInfo.attach: This didn't get reported as an error when options.path was the empty string, but should have been.
#11413 (and its fix#11414) inspired this search as they are the same
class of bug.
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
Two bug fixes:
- Do not use the worker that is being shutdown for a new job.
- Report unhandled errors during "expected to fail" tests as
fatal errors.
In some circumstances, dispatcher was waiting for all exisitng jobs
to finish before scheduling a new one. This leads to unneded stalls.
Instead, we can schedule jobs right away, if we have a worker
available.
