52 Commits

Author SHA1 Message Date
Andrey Lushnikov
05b019f1ba
reland: testrunner: make environment a simple class (#2812)
This re-lands PR https://github.com/microsoft/playwright/pull/2769

It was reverted before in https://github.com/microsoft/playwright/pull/2790
because it was breaking the new CHANNEL bot.
2020-07-02 11:05:38 -07:00
Pavel Feldman
e467ea57e7
revert: testrunner: make environment a simple class (#2769) (#2790) 2020-07-01 10:56:19 -07:00
Andrey Lushnikov
1605cb453c
testrunner: make environment a simple class (#2769)
This patch:
- makes environment a simple class with optional methods `beforeEach`, `afterEach`, `beforeAll`, `afterAll`, `globalSetup` and `globalTeardown`
- removes capability to have multiple hooks of the same name inside suite
- removes default environment for test. (`dit` now adds a `TraceTestEnvironment` to the test)
- extracts all environments that we use in our tests in `//test/environments.js`

Downsides:
- we no longer know hook locations for the environments. This, however, should not be a big deal since stack traces (if any) will still point into it.
- this also regresses hook locations for suites for simplicity. We can get them back, but it shouldn't be pressing since we now have only one hook of each kind in every suite.
2020-06-30 16:51:03 -07:00
Andrey Lushnikov
1865c5685a
testrunner: support globalSetup and globalTeardown hooks. (#2686) 2020-06-24 22:08:46 -07:00
Andrey Lushnikov
07dbd0baae
testrunner: teach runHook to accept hook arguments (#2682) 2020-06-23 18:17:14 -07:00
Andrey Lushnikov
c61e2d6cc9
testrunner: drop nested test environments (#2681) 2020-06-23 16:45:00 -07:00
Dmitry Gozman
cf82e2c945
fix(testrunner): await terminations before reporting test results (#1855)
This way we ensure that all errors are picked up.
2020-04-17 18:42:12 -07:00
Dmitry Gozman
a3571c24c4
fix(testrunner): respect timeout=0 in hooks (#1764) 2020-04-13 14:30:51 -07:00
Dmitry Gozman
2d57fff0be
fix(tests): fix multiple browsers tests (#1718)
- Use a server only once, not once per browser.
- Avoid require cache for test files.
- Improve testrunner hooks debugging.
2020-04-08 15:54:17 -07:00
Dmitry Gozman
20ff327827
feat(testrunner): catch delegate errors (#1704)
This ensures we get a proper error when something goes wrong. Should
also help with producing the right error code in the case of internal error.

Drive-by: fix location issue which manifests on the bots.
Drive-by: remove the use of Array.prototype.flat to make it work on bots.
2020-04-07 22:56:21 -07:00
Dmitry Gozman
39e06f0286
feat(testrunner): improve reporting of unhandled errors/rejections (#1697) 2020-04-07 16:02:33 -07:00
Dmitry Gozman
f2b13c0e93
chore(testrunner): split TestRunner into parts (#1679) 2020-04-06 17:21:42 -07:00
Dmitry Gozman
823f961d8d
feat(testrunner): migrate from events to a delegate (#1647)
This allows an async handler for each event that can be awaited.
Drive-by: merge TestPass into TestRunner.
2020-04-03 15:47:25 -07:00
Dmitry Gozman
1f2803bbc6
feat(testrunner): removeEnvironment (#1650) 2020-04-03 09:48:01 -07:00
Dmitry Gozman
d38baae01c
feat(testrunner): nested environments (#1635) 2020-04-02 13:12:22 -07:00
Dmitry Gozman
f87e64544c
feat(testrunner): introduce environments (#1593) 2020-04-01 10:49:47 -07:00
Dmitry Gozman
b6166c9669
chore(testrunner): introduce Location class (#1585)
Drive-by: fix an edge when testing continued after termination.
2020-03-29 21:38:30 -07:00
Dmitry Gozman
c49b856d6d
chore(testrunner): remove setup() helper (#1584) 2020-03-29 11:37:45 -07:00
Dmitry Gozman
a41836b1f1
chore(testrunner): introduce TestRun (#1582) 2020-03-28 14:25:57 -07:00
Dmitry Gozman
b85ab891a1
chore(testrunner): make most modifiers external (#1581) 2020-03-28 08:49:00 -07:00
Dmitry Gozman
4bd46bafa9
feat(testurnner): allow multiple hooks isntances and per-test hooks (#1571) 2020-03-27 22:57:22 -07:00
Dmitry Gozman
a042466125
chore(testrunner): remove effectiveMode and effectiveExpectation (#1569) 2020-03-27 15:35:14 -07:00
Dmitry Gozman
af7a16c360
chore(testrunner): merge test spec with test, suite spec with suite (#1566) 2020-03-26 22:47:13 -07:00
Dmitry Gozman
aad82e00bd
chore(testrunner): decouple UserCallback from location and timeout (#1557)
This will make it easier to change lifetimes of Test and Suite.
2020-03-26 14:43:28 -07:00
Dmitry Gozman
81bd8de00b
feat(testrunner): composable and bindable attributes and modifiers (#1547)
This allows to make things like `fit`, `skip()` and soon even `dit` be implemented externally.
2020-03-25 22:42:09 -07:00
Dmitry Gozman
b61198458c
feat(testrunner): modifiers and attributes (#1538)
This generalizes test modifiers to support custom ones.
2020-03-25 14:40:57 -07:00
Dmitry Gozman
a2ee7a1f2a
fix(testrunner): do not spam output after termination (#1511) 2020-03-24 14:40:59 -07:00
Dmitry Gozman
63906454a7
fix(testrunner): attribute unhandle promise reject to a single worker (#1491)
When there is a single worker, we are almost sure the error originated in that worker. Attributing it helps with context by showing last run tests.
2020-03-23 14:45:21 -07:00
Andrey Lushnikov
a9ab9b0dc0
fix(testrunner): sourcemapify stack traces for test errors (#1409)
Fixes #1403
2020-03-17 09:04:44 -07:00
Dmitry Gozman
3960b179a1
fix(testrunner): fit.fail should run the test (#1407) 2020-03-16 19:12:52 -07:00
Dmitry Gozman
e7eeefe4c7
chore(testrunner): separate expectations from run mode (#1395)
Run/Focus/Skip is orthogonal to expect to Pass/Fail/Flake.
This change separates the two, in a preparation to run Fail/Flaky tests.
2020-03-15 23:10:49 -07:00
Dmitry Gozman
0d7cb29329
test: continue running tests after crash, report crashes separately (#1362) 2020-03-12 17:32:53 -07:00
Andrey Lushnikov
1cd00bd068
feat(testrunner): allow filtering by name and show all focused tests (#1354) 2020-03-11 18:30:43 -07:00
Dmitry Gozman
0ce8efab7b
test: rework testrunner workers (#1296)
This change introduces a TestWorker that can be in a certain state,
meaning it has run some beforeAll hooks of a certain test suite stack.

TestWorker can be created at any time, which allows for a number of features:
- don't run hooks for suites with no runnable tests;
- smarter test distribution (and possibility for variuos strategies);
- recovering from hook failures and test failure by creating a new worker;
- possible isolation between workers by running them in separate environments.
2020-03-10 11:30:02 -07:00
Andrey Lushnikov
d542ef849f
fix(testrunner): handle uncaught errors (#1317) 2020-03-10 11:16:54 -07:00
Andrey Lushnikov
a0e12e0a14
feat(testrunner): support .slow() for slow tests (#1274)
Slow tests have 3x timeout.
2020-03-06 15:52:24 -08:00
Pavel
ba06fb2f0d test: mark some tests as skipped 2020-03-02 14:57:54 -08:00
Pavel Feldman
5bd6e4970b
test: it.skip skips and it.fail expects to fail now (#1178) 2020-03-02 13:47:08 -08:00
Pavel Feldman
cbf65a9c56
test: chain test modifiers (#1175) 2020-03-02 11:18:42 -08:00
Dmitry Gozman
ac2f04f10f
api(selectors): pass selector name when registering, allow file path (#1162) 2020-02-28 15:34:07 -08:00
Andrey Lushnikov
0ded511d0b
feat(testrunner): better matchers (#1077)
This patch re-implements matching and reporting for test runner.
Among other improvements:
- test failures now show a short snippet from test
- test failures now explicitly say what received and what was expected
- `expect.toBe()` now does text diff when gets strings as input
- `expect.toEqual` now does object diff
2020-02-20 22:55:39 -08:00
Andrey Lushnikov
8487ef2821
feat(testrunner): add DEBUG to testrunner (#1014)
The plan is to collect logs for the whole test run and upload it later on
using https://github.com/actions/upload-artifact

Produced log size:
- 163MB (8.5MB zipped) for Chromium: `DEBUG=* npm run unit 2>log`
- 135MB (4.8MB zipped) for WebKit: `DEBUG=*,-pw:wrapped* npm run wunit 2>log`
- 29MB (4.0MB zipped) for Firefox: `DEBUG=* npm run funit 2>log`
2020-02-14 15:21:08 -08:00
Andrey Lushnikov
7128628d70 feat(testrunner): ability to repeat test suites (#681)
Now you can `fdescribe.repeat(10)` to repeat test suites.
2020-01-27 12:48:11 -08:00
Andrey Lushnikov
710554bff5
feat(testrunner): add it.repeat to repeat test multiple times (#568) 2020-01-22 15:31:52 -08:00
Andrey Lushnikov
98bf9ac1d5
fix(testrunner): return correct exit code (#474) 2020-01-13 15:30:16 -08:00
Andrey Lushnikov
02c504ca59
feat(testrunner): "sourcemapify" crash error stacktraces (#441)
Whenever test runner crashes due to UnhandledPromiseRejection, the
related error should be run through "sourcemap decoding".
2020-01-09 16:37:19 +00:00
Andrey Lushnikov
a18777673e
devops(testrunner): support source maps (#340)
This patch adds a basic source map support to test runner.

SourceMap support is powered by Chromium DevTools source map
implementation (thus copyright). Unlike popular `source-map` npm
module, it's sync and pretty straight-forward.

The `SourceMap.js` file has a few modifications wrt upstream
Chromium version:
- reverse mappings API is removed. There's no need to ever compute them
- the `upperBoundary` function from DevTools' platform is inlined
2020-01-08 16:16:54 +00:00
Andrey Lushnikov
a7f3490e48 feat(testrunner): implement "dit" command (#312) 2019-12-19 16:47:35 -07:00
Andrey Lushnikov
2acf36debc
fix(testrunner): xdescribe.skip should be a no-op 2019-12-19 00:46:03 -08:00
Andrey Lushnikov
8cc682c7b8 feat(testrunner): allow focusing all tests in a file
With this change, exporting a "fdescribe" function from a file
will focus all tests inside the file.
2019-12-18 17:13:39 -08:00