53 Commits

Author SHA1 Message Date
Dmitry Gozman
d214778548
chore(test runner): move timeout handling to the top, stop inheriting runnable (#29857) 2024-03-08 15:19:36 -08:00
Dmitry Gozman
f61e445f2b
Revert "chore(test runner): remove fake skipped test results (#27762)" (#28360)
This reverts commit 210168e36db026448f419a53e86865335f1332be.

Fixes #28321.
2023-11-27 16:08:20 -08:00
Dmitry Gozman
210168e36d
chore(test runner): remove fake skipped test results (#27762)
Fixes #27455.
2023-10-24 09:31:30 -07:00
Dmitry Gozman
d67515f6c1
chore(test runner): do not produce some of the fake skipped test results (#27730) 2023-10-20 17:01:46 -07:00
Jim Hays
dcc8dcca73
Fix various typos (Fixes #27396) (#27391)
Fixes https://github.com/microsoft/playwright/issues/27396
2023-10-04 19:56:42 -07:00
Dmitry Gozman
49fd9500fe
fix: handle worker process start failure (#27249)
Worker process start failure is reported as a test error and skips other
tests from the group.
If happened during stop (e.g. from a Ctrl+C) before worker has fully
initialized, this error is ignored.

Drive-by: send SIGINT in tests to the whole tree, to better emulate
Ctrl+C behavior.
2023-09-22 10:57:35 -07:00
Dmitry Gozman
70589c0113
test: unflake "beforeAll and afterAll should have a separate timeout" (#23263) 2023-05-24 13:46:28 -07:00
Dmitry Gozman
2858ca95d8
tet: unflake some test runner tests (#23240)
- `unhandled rejection during beforeAll should be reported and prevent
more tests` had timeout of 100ms that is too short on busy system;
- `runTSC` is always slow, so mark as such.
2023-05-23 13:31:23 -07:00
Dmitry Gozman
8f09935e81
fix(test runner): separate test fixture scope for beforeAll/afterAll hooks (#22746)
There was a single test fixture scope that covers all hooks, modifiers
and test function. Now beforeAll-like modifiers, beforeAll and afterAll
hooks get a scope each.

Fixes #22256.
2023-05-02 11:04:51 -07:00
Dmitry Gozman
91da67fab1
test: remove magic headers in ttest (#20867)
Instead, explicitly import from '@playwright/test'.
2023-02-14 19:20:56 -08:00
Pavel Feldman
1b941bcf2e
chore: simplify ttests (#20733) 2023-02-07 15:11:44 -08:00
Dmitry Gozman
388a3e1f37
fix(test runner): make sure to run afterAll after skipped tests (#19878)
Fixes #19745.
2023-01-04 14:13:49 -08:00
Pavel Feldman
675f0eb4a0
chore: report error location for fatal errors (#19610) 2022-12-21 09:36:59 -08:00
Dmitry Gozman
23e02dd006
test: unflake some timeout-dependent tests (#19139) 2022-11-29 12:54:53 -08:00
Dmitry Gozman
9fe72a1da8
fix(test runner): remove unused type imports during transform (#18157)
Fixes #18117.
2022-10-19 13:06:11 -07:00
Dmitry Gozman
b64457d8ce
fix(locators): make regex escape work when multiple spaces are present (#17893) 2022-10-06 16:23:40 -07:00
sand4rt
72a18754ef
core(ct): rename tests (#17216) 2022-09-12 09:27:53 -07:00
Dmitry Gozman
919b1cbb22
fix(test runner): do not swallow afterEach errors after skipped tests (#16276)
Drive-by: add more test runner logging.
2022-08-04 18:49:23 -07:00
Dmitry Gozman
a94fe361a2
test: unflake a few tests (#15959) 2022-07-27 11:27:53 -07:00
Dmitry Gozman
22ead7b5ab
Revert "fix(test runner): handle istty in line reporter (#15455)" (#15633)
This reverts commit 767babc3a225f9dec7efbcfef1bd073bcb053d93.
2022-07-13 15:11:56 -07:00
Dmitry Gozman
51c3ea55ed
Revert "feat(line reporter): show testing progress (#15547)" (#15631)
This reverts commit e76d9b3b2828620e36827047dd714ef037b75fdc.
2022-07-13 13:54:26 -07:00
Dmitry Gozman
64353c4b96
fix(test runner): make sure worker cleans up if last test was skipped (#15552) 2022-07-11 21:33:56 -07:00
Dmitry Gozman
e76d9b3b28
feat(line reporter): show testing progress (#15547)
Line reporter now shows stats in addition to the test name:
```
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
[21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
2022-07-11 20:44:50 -07:00
Dmitry Gozman
767babc3a2
fix(test runner): handle istty in line reporter (#15455)
When running without tty, line reporter outputs a line for each
percent of the tests, thus limiting the output to ~100 lines.

In addition, reporters now support PLAYWRIGHT_LIVE_TERMINAL
env variable to force tty mode.
2022-07-07 09:55:12 -07:00
Dmitry Gozman
b86926e691
fix(test runner): update timeout error messages (#15117)
Some of the current timeout error messages are confusing, because they do not suggest that the issue is most likely a slow test. This PR updates timeout messages as follows:
- Test timeout of 30000ms exceeded.
- Test timeout of 30000ms exceeded while setting up "browser".
- Test timeout of 30000ms exceeded while tearing down "context".
- Test timeout of 30000ms exceeded while setting up "playwright configuration".
- Test timeout of 30000ms exceeded while running "beforeEach" hook.
- Test timeout of 30000ms exceeded while running "afterEach" hook.
- "beforeAll" hook timeout of 30000ms exceeded.
- "afterAll" hook timeout of 30000ms exceeded.
- Worker teardown timeout of 30000ms exceeded.
- "skip" modifier timeout of 30000ms exceeded.
- Fixture "myCustomFixture" timeout of 5000ms exceeded.
2022-06-30 17:05:08 -07:00
Dmitry Gozman
237954212c
Revert "feat(line reporter): show stats, handle tty" (#12735)
This reverts commit be817d1a53c2f2e31a29dd2b11df5d928fea3805, PR #12695.
Reason: found issues with stdout messing with stats.
2022-03-14 10:37:43 -07:00
Dmitry Gozman
be817d1a53
feat(line reporter): show stats, handle tty (#12695)
- Line reporter now shows stats in addition to the test name:
  ```log
  [chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
  23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
  ```

- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.

- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.

- Updated tests to the golden style.
2022-03-11 18:19:33 -08:00
Dmitry Gozman
e8ce5d0258
feat(test runner): on beforeAll failure, precisely skip the tests (#12417)
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.

Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
2022-03-08 20:29:31 -08:00
Dmitry Gozman
682eeb4ae0
reland(hooks): separate test timeout from beforeAll/afterAll timeouts (#12609)
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-03-08 19:05:23 -08:00
Dmitry Gozman
9cd704220c
reland(test runner): run hooks/modifiers as a part of the test (#12518)
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-03-08 16:35:14 -08:00
Pavel Feldman
6a663ef54f
chore(test-runner): revert recent changes to fix tests (#12439)
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"

This reverts commit 73dee69558110b00916708dafaa99e121c6e7c94.

* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"

This reverts commit d744a87aeefe9c131ed10cedfff201efa1b5c156.

* Revert "chore(test runner): run hooks/modifiers as a part of the test  (#12329)"

This reverts commit 47045ba48dd84d38bde17541c8a0a05cdfed7ffc.
2022-03-01 09:11:17 -08:00
Dmitry Gozman
73dee69558
fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-02-28 15:09:04 -08:00
Dmitry Gozman
47045ba48d
chore(test runner): run hooks/modifiers as a part of the test (#12329)
chore(test runner): run hooks/modifiers as a part of the test

This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-02-28 11:42:47 -08:00
Dmitry Gozman
4399623f9f
chore(test runner): make timeout a separate error in TestInfo.errors (#12315)
This way we control the timeout error message from the runner,
so that later on we can differentiate between test timeout, fixture
timeout and hook timeout.
2022-02-23 12:32:12 -08:00
Dmitry Gozman
d3c4323021
fix(test runner): improve error message for unexpected calls (#12240) 2022-02-18 18:25:18 -08:00
Dmitry Gozman
5881a46ecf
fix(test runner): skip beforeAll/afterAll when all tests are skipped (#11952)
There is a corner case where tests were skipped like this:

```js
test.skip('title', () => {});
```
2022-02-08 16:36:30 -08:00
Andrey Lushnikov
2336692e8a
feat: support clarification message for expect (#11735)
The clarification message is displayed in the HTML report as the name of the step:

![image](https://user-images.githubusercontent.com/746130/151852652-48194140-5ea4-439d-afee-12583a8caf71.png)

It is also shown in terminal output:

![image](https://user-images.githubusercontent.com/746130/151852666-5c956ef1-6e94-4bc2-8e55-b58688dfc7e0.png)

Fixes #7816
2022-01-31 17:14:59 -08:00
Dmitry Gozman
14fd837e94
fix(test runner): hide beforeAll/afterAll hooks from the reporter api (#11306)
This api is not ready yet.
2022-01-10 12:09:51 -08:00
Dmitry Gozman
2d00836f0e
fix(test runner): show the location of afterAll timeout (#11007) 2021-12-18 09:32:41 -08:00
Dmitry Gozman
0d54afab9c
feat(test runner): show beforeAll/afterAll hooks similar to tests (#10923)
Reporters now get notified about hooks start/end/steps.
2021-12-15 10:39:49 -08:00
Dmitry Gozman
8f43f4c98f
feat(serial): better errors from beforeAll (#10419)
When beforeAll hook times out or fails with an exception, we now
close the context and show a nice error.
2021-11-18 14:36:55 -08:00
Dmitry Gozman
9622704a8a
fix(test runner): update TestInfo.duration before running afterEach hooks (#10228) 2021-11-10 16:02:27 -08:00
Dmitry Gozman
13ed1dee50
fix(test runner): do not mask uncaught error in beforeEach (#9764) 2021-10-25 14:17:27 -07:00
Dmitry Gozman
23aa0be15b
docs: update afterEach docs with an example (#9727)
Also add a test for TestInfo.status in afterEach.
2021-10-22 16:32:22 -07:00
Dmitry Gozman
37a897c9b4
fix(test runner): report beforeAll timeout instead of hanging (#8529)
We used to not report fatal error and hang forever because worker
did not run any tests but also did not report any errors.

Also properly show stack-less errors.
2021-08-28 07:19:45 -07:00
Dmitry Gozman
f9b87268a7
fix(test runner): afterAll error should not mask beforeAll error (#8358) 2021-08-23 09:21:40 -07:00
Dmitry Gozman
d32d50a906 fix(test runner): make sure we always teardown all fixtures (#8158)
Even if one of the fixtures throws, we should teardown all of them
so that we can run afterAll hooks.
2021-08-12 09:08:56 -07:00
Dmitry Gozman
9d6c7cdf20
fix(test runner): ensure we run after hooks after failures (#8102) 2021-08-10 10:54:05 -07:00
Dmitry Gozman
e638c4597f
fix(test runner): do not swallow afterAll failure (#8099) 2021-08-09 14:21:53 -07:00
Dmitry Gozman
87548f94c1
feat(test runner): support test fixtures in beforeAll/afterAll (#8082)
Each hook gets its own test scope. This is not too useful for
object fixtures like `page` (although one can use a page in
`beforeAll` to save storage state), but much more useful for option
fixtures like `viewport`.
2021-08-09 13:26:33 -07:00