diff --git a/docs/src/ci.md b/docs/src/ci.md index b20999f93c..bca0803c19 100644 --- a/docs/src/ci.md +++ b/docs/src/ci.md @@ -32,6 +32,7 @@ configurations for common CI providers. mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --with-deps" ``` ```bash csharp + dotnet build pwsh bin/Debug/netX/playwright.ps1 install --with-deps ``` @@ -55,122 +56,163 @@ The [Command line tools](./browsers#install-system-dependencies) can be used to ### GitHub Actions +#### On push/pull_request + ```yml js -steps: - - uses: actions/checkout@v3 - - uses: actions/setup-node@v3 - with: - node-version: '18' - - name: Install dependencies - run: npm ci - - name: Install Playwright - run: npx playwright install --with-deps - - name: Run your tests - run: npx playwright test - - name: Upload test results - if: always() - uses: actions/upload-artifact@v3 - with: - name: playwright-report - path: playwright-report -``` - -```yml python -steps: - - uses: actions/checkout@v3 - - name: Set up Python - uses: actions/setup-python@v4 - with: - python-version: '3.11' - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install -r local-requirements.txt - pip install -e . - - name: Ensure browsers are installed - run: python -m playwright install --with-deps - - name: Run your tests - run: pytest -``` - -```yml java -steps: - - uses: actions/checkout@v3 - - uses: actions/setup-java@v3 - with: - distribution: 'temurin' - java-version: '17' - - name: Build & Install - run: mvn -B install -D skipTests --no-transfer-progress - - name: Install Playwright - run: mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --with-deps" - - name: Run tests - run: mvn test -``` - -```yml csharp -steps: - - uses: actions/checkout@v3 - - name: Setup dotnet - uses: actions/setup-dotnet@v3 - with: - dotnet-version: 6.0.x - - run: dotnet build - - name: Ensure browsers are installed - run: pwsh bin\Debug\net6.0\playwright.ps1 install --with-deps - - name: Run your tests - run: dotnet test -``` - -We run [our tests](https://github.com/microsoft/playwright/blob/main/.github/workflows/tests_secondary.yml) on GitHub Actions, across a matrix of 3 platforms (Windows, Linux, macOS) and 3 browsers (Chromium, Firefox, WebKit). - -### GitHub Actions on deployment - -This will start the tests after a [GitHub Deployment](https://developer.github.com/v3/repos/deployments/) went into the `success` state. -Services like Vercel use this pattern so you can run your end-to-end tests on their deployed environment. - -```yml name: Playwright Tests on: - deployment_status: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] jobs: test: timeout-minutes: 60 runs-on: ubuntu-latest - if: github.event.deployment_status.state == 'success' steps: - uses: actions/checkout@v3 - uses: actions/setup-node@v3 with: - node-version: '18.x' + node-version: 18 - name: Install dependencies run: npm ci - - name: Install Playwright + - name: Install Playwright Browsers run: npx playwright install --with-deps - name: Run Playwright tests run: npx playwright test - env: - # This might depend on your test-runner/language binding - PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }} + - uses: actions/upload-artifact@v3 + if: always() + with: + name: playwright-report + path: playwright-report/ + retention-days: 30 ``` -### Docker +```yml python +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] +jobs: + test: + timeout-minutes: 60 + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.11' + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r local-requirements.txt + pip install -e . + - name: Ensure browsers are installed + run: python -m playwright install --with-deps + - name: Run your tests + run: pytest +``` -We have a [pre-built Docker image](./docker.md) which can either be used directly, or as a reference to update your existing Docker definitions. +```yml java +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] +jobs: + test: + timeout-minutes: 60 + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: actions/setup-java@v3 + with: + distribution: 'temurin' + java-version: '17' + - name: Build & Install + run: mvn -B install -D skipTests --no-transfer-progress + - name: Install Playwright + run: mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --with-deps" + - name: Run tests + run: mvn test +``` -Suggested configuration -1. Using `--ipc=host` is also recommended when using Chromium—without it Chromium can run out of memory - and crash. Learn more about this option in [Docker docs](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc). -1. Seeing other weird errors when launching Chromium? Try running your container - with `docker run --cap-add=SYS_ADMIN` when developing locally. -1. Using `--init` Docker flag or [dumb-init](https://github.com/Yelp/dumb-init) is recommended to avoid special - treatment for processes with PID=1. This is a common reason for zombie processes. +```yml csharp +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] +jobs: + test: + timeout-minutes: 60 + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: Setup dotnet + uses: actions/setup-dotnet@v3 + with: + dotnet-version: 6.0.x + - run: dotnet build + - name: Ensure browsers are installed + run: pwsh bin/Debug/netX/playwright.ps1 install --with-deps + - name: Run your tests + run: dotnet test +``` -### GitHub Actions (via containers) +#### On push/pull_request (sharded) +* langs: js -GitHub Actions support [running jobs in a container](https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container) by using the [`jobs..container`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer) option. +GitHub Actions supports [sharding tests between multiple jobs](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) using the [`jobs..strategy.matrix`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix) option. The `matrix` option will run a separate job for every possible combination of the provided options. In the example below, we have 2 `project` values, 10 `shardIndex` values and 1 `shardTotal` value, resulting in a total of 20 jobs to be run. So it will split up the tests between 20 jobs, each running a different browser and a different subset of tests, see [here](./test-parallel.md#shard-tests-between-multiple-machines) for more details. ```yml js +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] +jobs: + playwright: + name: 'Playwright Tests - ${{ matrix.project }} - Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}' + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + project: [chromium, webkit] + shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + shardTotal: [10] + steps: + - uses: actions/checkout@v3 + - uses: actions/setup-node@v3 + with: + node-version: 18 + - name: Install dependencies + run: npm ci + - name: Install browsers + run: npx playwright install --with-deps + - name: Run your tests + run: npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} +``` + +> Note: The `${{ }}` is the [expression](https://docs.github.com/en/actions/learn-github-actions/expressions) syntax that allows accessing the current [context](https://docs.github.com/en/actions/learn-github-actions/contexts). In this example, we are using the [`matrix`](https://docs.github.com/en/actions/learn-github-actions/contexts#matrix-context) context to set the job variants. + +#### Via Containers + +GitHub Actions support [running jobs in a container](https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container) by using the [`jobs..container`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer) option. This is useful to not pollute the host environment with dependencies and to have a consistent environment for e.g. screenshots/visual regression testing across different operating systems. + +```yml js +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] jobs: playwright: name: 'Playwright Tests' @@ -181,7 +223,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-node@v3 with: - node-version: '18' + node-version: 18 - name: Install dependencies run: npm ci - name: Run your tests @@ -189,6 +231,12 @@ jobs: ``` ```yml python +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] jobs: playwright: name: 'Playwright Tests' @@ -211,6 +259,12 @@ jobs: ``` ```yml java +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] jobs: playwright: name: 'Playwright Tests' @@ -230,6 +284,12 @@ jobs: ``` ```yml csharp +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] jobs: playwright: name: 'Playwright Tests' @@ -247,12 +307,18 @@ jobs: run: dotnet test ``` -#### Sharding +#### Via Containers (sharded) * langs: js -GitHub Actions supports [sharding tests between multiple jobs](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) using the [`jobs..strategy.matrix`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix) option. The `matrix` option will run a separate job for every possible combination of the provided options. In the example below, we have 2 `project` values, 10 `shardIndex` values and 1 `shardTotal` value, resulting in a total of 20 jobs to be run. +GitHub Actions supports [sharding tests between multiple jobs](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) using the [`jobs..strategy.matrix`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix) option. The `matrix` option will run a separate job for every possible combination of the provided options. In the example below, we have 2 `project` values, 10 `shardIndex` values and 1 `shardTotal` value, resulting in a total of 20 jobs to be run. So it will split up the tests between 20 jobs, each running a different browser and a different subset of tests, see [here](./test-parallel.md#shard-tests-between-multiple-machines) for more details. ```yml js +name: Playwright Tests +on: + push: + branches: [ main, master ] + pull_request: + branches: [ main, master ] jobs: playwright: name: 'Playwright Tests - ${{ matrix.project }} - Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}' @@ -269,14 +335,54 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-node@v3 with: - node-version: '18' + node-version: 18 - name: Install dependencies run: npm ci - name: Run your tests run: npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} ``` -> Note: The `${{ }}` is the [expression](https://docs.github.com/en/actions/learn-github-actions/expressions) syntax that allows accessing the current [context](https://docs.github.com/en/actions/learn-github-actions/contexts). In this example, we are using the [`matrix`](https://docs.github.com/en/actions/learn-github-actions/contexts#matrix-context) context to set the job variants. +#### On deployment + +This will start the tests after a [GitHub Deployment](https://developer.github.com/v3/repos/deployments/) went into the `success` state. +Services like Vercel use this pattern so you can run your end-to-end tests on their deployed environment. + +```yml +name: Playwright Tests +on: + deployment_status: +jobs: + test: + timeout-minutes: 60 + runs-on: ubuntu-latest + if: github.event.deployment_status.state == 'success' + steps: + - uses: actions/checkout@v3 + - uses: actions/setup-node@v3 + with: + node-version: 18 + - name: Install dependencies + run: npm ci + - name: Install Playwright + run: npx playwright install --with-deps + - name: Run Playwright tests + run: npx playwright test + env: + # This might depend on your test-runner/language binding + PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }} +``` + +### Docker + +We have a [pre-built Docker image](./docker.md) which can either be used directly, or as a reference to update your existing Docker definitions. + +Suggested configuration +1. Using `--ipc=host` is also recommended when using Chromium. Without it Chromium can run out of memory + and crash. Learn more about this option in [Docker docs](https://docs.docker.com/engine/reference/run/#ipc-settings---ipc). +1. Seeing other weird errors when launching Chromium? Try running your container + with `docker run --cap-add=SYS_ADMIN` when developing locally. +1. Using `--init` Docker flag or [dumb-init](https://github.com/Yelp/dumb-init) is recommended to avoid special + treatment for processes with PID=1. This is a common reason for zombie processes. ### Azure Pipelines @@ -292,7 +398,7 @@ For running the Playwright tests use this pipeline task: jobs: - deployment: Run_E2E_Tests pool: - vmImage: ubuntu-20.04 + vmImage: ubuntu-22.04 container: mcr.microsoft.com/playwright:v1.34.0-jammy environment: testing strategy: @@ -318,7 +424,7 @@ If you also want to integrate the test results with Azure DevOps, use the task ` jobs: - deployment: Run_E2E_Tests pool: - vmImage: ubuntu-20.04 + vmImage: ubuntu-22.04 container: mcr.microsoft.com/playwright:v1.34.0-jammy environment: testing strategy: @@ -357,7 +463,7 @@ in `playwright.config.ts`. ### CircleCI -Running Playwright on CircleCI is very similar to running on GitHub Actions. In order to specify the pre-built Playwright [Docker image](./docker.md) , simply modify the agent definition with `docker:` in your config like so: +Running Playwright on CircleCI is very similar to running on GitHub Actions. In order to specify the pre-built Playwright [Docker image](./docker.md), simply modify the agent definition with `docker:` in your config like so: ```yml executors: @@ -368,16 +474,6 @@ Running Playwright on CircleCI is very similar to running on GitHub Actions. In Note: When using the docker agent definition, you are specifying the resource class of where playwright runs to the 'medium' tier [here](https://circleci.com/docs/configuration-reference?#docker-execution-environment). The default behavior of Playwright is to set the number of workers to the detected core count (2 in the case of the medium tier). Overriding the number of workers to greater than this number will cause unnecessary timeouts and failures. -Similarly, If you’re using Playwright through Jest, then you may encounter an error spawning child processes: - - ``` - [00:00.0] jest args: --e2e --spec --max-workers=36 - Error: spawn ENOMEM - at ChildProcess.spawn (internal/child_process.js:394:11) - ``` - - This is likely caused by Jest autodetecting the number of processes on the entire machine (`36`) rather than the number allowed to your container (`2`). To fix this, set `jest --maxWorkers=2` in your test command. - #### Sharding in CircleCI Sharding in CircleCI is indexed with 0 which means that you will need to override the default parallelism ENV VARS. The following example demonstrates how to run Playwright with a CircleCI Parallelism of 4 by adding 1 to the `CIRCLE_NODE_INDEX` to pass into the `--shard` cli arg. @@ -472,36 +568,6 @@ tests: ## Caching browsers -By default, Playwright downloads browser binaries when the Playwright NPM package -is installed. The NPM packages have a `postinstall` hook that downloads the browser -binaries. This behavior can be [customized with environment variables](./browsers.md#managing-browser-binaries). - -Caching browsers on CI is **strictly optional**: The `postinstall` hooks should -execute and download the browser binaries on every run. - -#### Exception: `node_modules` are cached (Node-specific) - -Most CI providers cache the [npm-cache](https://docs.npmjs.com/cli-commands/cache.html) -directory (located at `$HOME/.npm`). If your CI pipelines caches the `node_modules` -directory and you run `npm install` (instead of `npm ci`), the default configuration -**will not work**. This is because the `npm install` step will find the Playwright NPM -package on disk and not execute the `postinstall` step. - -> Travis CI automatically caches `node_modules` if your repo does not have a - `package-lock.json` file. - -This behavior can be fixed with one of the following approaches: -1. Move to caching `$HOME/.npm` or the npm-cache directory. (This is the default - behavior in most CI providers.) -1. Set `PLAYWRIGHT_BROWSERS_PATH=0` as the environment variable before running - `npm install`. This will download the browser binaries in the `node_modules` - directory and cache them with the package code. See [managing browser binaries](./browsers.md#managing-browser-binaries). -1. Use `npm ci` (instead of `npm install`) which forces a clean install: by - removing the existing `node_modules` directory. See [npm docs](https://docs.npmjs.com/cli/ci.html). -1. Cache the browser binaries, with the steps below. - -#### Directories to cache - With the default behavior, Playwright downloads the browser binaries in the following directories: @@ -523,6 +589,14 @@ DEBUG=pw:browser* npx playwright test DEBUG=pw:browser* pytest ``` +```bash java +DEBUG=pw:browser* mvn test +``` + +```bash csharp +DEBUG=pw:browser* dotnet test +``` + ## Running headed By default, Playwright launches browsers in headless mode. This can be changed by passing a flag when the browser is launched. @@ -585,3 +659,9 @@ xvfb-run node index.js ```bash python xvfb-run python test.py ``` +```bash java +xvfb-run mvn test +``` +```bash csharp +xvfb-run dotnet test +```