Revert "fix(test runner): ignore undefined values in fixtures definitions (#15119)"
Revert commit d7b63fa0b45ab12e40d0018b0e6f796ecd2df4b7.
Add a test for the broken behavior.
Fixes#8206.
Since #8206 is a long-awaited (~ 1 year old), popular (~ 45 reactions, frequently requested in community channels, etc.), this PR aims to unblock folks.
Notably, we do not innovate on the `webServer` API, despite knowing we're not in love with it. We'll save the innovation for either Plugins or a new `LaunchConfigs` option. (We haven't yet arrived at a Plugin API we like, and instead of launching a new option guessing what the "better" launchConfig API would be, let's wait and see how folks use this new Array-variant of `webServer` which—despite its name—can be used for non-Web Server launches!
Useful to set some options without coming up with an artificial name.
```js
test.describe(() => {
test.use({ colorScheme: 'dark' });
test('my test', () => {});
})
```
These mean "I don't want to specify this fixture/option"
instead of "I want the value of undefined", aligned with how TypeScript works.
We already do similar things in the config.
If the tests are in an explicitly configured testDir (either at the global config level or per project) .gitignore filters are not applied.
Fixes#14381
This patch:
- starts using directory of `package.json` to resolve default
output directory path
- starts using either `package.json` directory or configuration
directory to resolve all relative paths
References #12970
Resolves#11318.
* Adds `TestConfig.attachments` public API. (We opted to not implement an analog to the async `TestInfo.attach(…)` API.)
* Adds `TestConfig.attachments` to common reporters.
* Dogfoods some git and CI-info inference to generate useful atttachments
* Updates HTML Reporter to include a side bar to present a pre-defined set of attachments (a.k.a git/commit context sidebar)
Here's what it looks like:
<img width="1738" alt="Screen Shot 2022-03-21 at 3 23 28 PM" src="https://user-images.githubusercontent.com/11915034/159373291-8b937d30-fba3-472a-853a-766018f6b3e2.png">
See `tests/playwright-test/reporter-html.spec.ts` for an example of usage (for dogfood-ing only). In the future, if this becomes user-facing, there the Global Setup bit would likely become unnecessary (as would interaction with attachments array); there would likely just be a nice top-level config and/or CLI flag to enable collecting of info.
Currently, running `npx playwright test` in a subfolder of your project
will result in a `test-results` folder created in your `cwd`.
This is unexpected; instead, we should always resolve all paths
against `rootDir` - directory that contains config.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
