mirror of
https://github.com/microsoft/playwright.git
synced 2025-06-26 21:40:17 +00:00
327 lines
14 KiB
Markdown
327 lines
14 KiB
Markdown
---
|
|
id: ci-intro
|
|
title: "Setting up CI"
|
|
---
|
|
|
|
## Introduction
|
|
* langs: js
|
|
|
|
Playwright tests can be run on any CI provider. This guide covers one way of running tests on GitHub using GitHub actions. If you would like to learn more, or how to configure other CI providers, check out our detailed [doc on Continuous Integration](./ci.md).
|
|
|
|
#### You will learn
|
|
* langs: js
|
|
|
|
- [How to set up GitHub Actions](/ci-intro.md#setting-up-github-actions)
|
|
- [How to view test logs](/ci-intro.md#viewing-test-logs)
|
|
- [How to view the HTML report](/ci-intro.md#viewing-the-html-report)
|
|
- [How to view the trace](/ci-intro.md#viewing-the-trace)
|
|
- [How to publish report on the web](/ci-intro.md#publishing-report-on-the-web)
|
|
|
|
|
|
## Introduction
|
|
* langs: python, java, csharp
|
|
|
|
Playwright tests can be run on any CI provider. In this section we will cover running tests on GitHub using GitHub actions. If you would like to see how to configure other CI providers check out our detailed doc on Continuous Integration.
|
|
|
|
#### You will learn
|
|
* langs: python, java, csharp
|
|
|
|
- [How to set up GitHub Actions](/ci-intro.md#setting-up-github-actions)
|
|
- [How to view test logs](/ci-intro.md#viewing-test-logs)
|
|
- [How to view the trace](/ci-intro.md#viewing-the-trace)
|
|
|
|
|
|
## Setting up GitHub Actions
|
|
* langs: js
|
|
|
|
When [installing Playwright](./intro.md) using the [VS Code extension](./getting-started-vscode.md) or with `npm init playwright@latest` you are given the option to add a [GitHub Actions](https://docs.github.com/en/actions) workflow. This creates a `playwright.yml` file inside a `.github/workflows` folder containing everything you need so that your tests run on each push and pull request into the main/master branch. Here's how that file looks:
|
|
|
|
```yml js title=".github/workflows/playwright.yml"
|
|
name: Playwright Tests
|
|
on:
|
|
push:
|
|
branches: [ main, master ]
|
|
pull_request:
|
|
branches: [ main, master ]
|
|
jobs:
|
|
test:
|
|
timeout-minutes: 60
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: lts/*
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
- name: Install Playwright Browsers
|
|
run: npx playwright install --with-deps
|
|
- name: Run Playwright tests
|
|
run: npx playwright test
|
|
- uses: actions/upload-artifact@v4
|
|
if: ${{ !cancelled() }}
|
|
with:
|
|
name: playwright-report
|
|
path: playwright-report/
|
|
retention-days: 30
|
|
```
|
|
|
|
The workflow performs these steps:
|
|
|
|
1. Clone your repository
|
|
2. Install Node.js
|
|
3. Install NPM Dependencies
|
|
4. Install Playwright Browsers
|
|
5. Run Playwright tests
|
|
6. Upload HTML report to the GitHub UI
|
|
|
|
To learn more about this, see ["Understanding GitHub Actions"](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions).
|
|
|
|
## Setting up GitHub Actions
|
|
* langs: python, java, csharp
|
|
|
|
To add a [GitHub Actions](https://docs.github.com/en/actions) file first create `.github/workflows` folder and inside it add a `playwright.yml` file containing the example code below so that your tests will run on each push and pull request for the main/master branch.
|
|
|
|
```yml python title=".github/workflows/playwright.yml"
|
|
name: Playwright Tests
|
|
on:
|
|
push:
|
|
branches: [ main, master ]
|
|
pull_request:
|
|
branches: [ main, master ]
|
|
jobs:
|
|
test:
|
|
timeout-minutes: 60
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v4
|
|
with:
|
|
python-version: '3.11'
|
|
- name: Install dependencies
|
|
run: |
|
|
python -m pip install --upgrade pip
|
|
pip install -r requirements.txt
|
|
- name: Ensure browsers are installed
|
|
run: python -m playwright install --with-deps
|
|
- name: Run your tests
|
|
run: pytest --tracing=retain-on-failure
|
|
- uses: actions/upload-artifact@v4
|
|
if: ${{ !cancelled() }}
|
|
with:
|
|
name: playwright-traces
|
|
path: test-results/
|
|
```
|
|
|
|
```yml java title=".github/workflows/playwright.yml"
|
|
name: Playwright Tests
|
|
on:
|
|
push:
|
|
branches: [ main, master ]
|
|
pull_request:
|
|
branches: [ main, master ]
|
|
jobs:
|
|
test:
|
|
timeout-minutes: 60
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-java@v3
|
|
with:
|
|
distribution: 'temurin'
|
|
java-version: '17'
|
|
- name: Build & Install
|
|
run: mvn -B install -D skipTests --no-transfer-progress
|
|
- name: Ensure browsers are installed
|
|
run: mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --with-deps"
|
|
- name: Run tests
|
|
run: mvn test
|
|
```
|
|
|
|
```yml csharp title=".github/workflows/playwright.yml"
|
|
name: Playwright Tests
|
|
on:
|
|
push:
|
|
branches: [ main, master ]
|
|
pull_request:
|
|
branches: [ main, master ]
|
|
jobs:
|
|
test:
|
|
timeout-minutes: 60
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Setup dotnet
|
|
uses: actions/setup-dotnet@v4
|
|
with:
|
|
dotnet-version: 8.0.x
|
|
- name: Build & Install
|
|
run: dotnet build
|
|
- name: Ensure browsers are installed
|
|
run: pwsh bin/Debug/net8.0/playwright.ps1 install --with-deps
|
|
- name: Run your tests
|
|
run: dotnet test
|
|
```
|
|
|
|
To learn more about this, see ["Understanding GitHub Actions"](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions).
|
|
|
|
Looking at the list of steps in `jobs.test.steps`, you can see that the workflow performs these steps:
|
|
|
|
1. Clone your repository
|
|
2. Install language dependencies
|
|
3. Install project dependencies and build
|
|
4. Install Playwright Browsers
|
|
5. Run tests
|
|
|
|
## Create a Repo and Push to GitHub
|
|
|
|
Once you have your [GitHub actions workflow](#setting-up-github-actions) setup then all you need to do is [Create a repo on GitHub](https://docs.github.com/en/get-started/quickstart/create-a-repo) or push your code to an existing repository. Follow the instructions on GitHub and don't forget to [initialize a git repository](https://github.com/git-guides/git-init) using the `git init` command so you can [add](https://github.com/git-guides/git-add), [commit](https://github.com/git-guides/git-commit) and [push](https://github.com/git-guides/git-push) your code.
|
|
|
|
######
|
|
* langs: js, java, python
|
|
|
|
<img width="861" alt="Create a Repo and Push to GitHub" src="https://user-images.githubusercontent.com/13063165/183423254-d2735278-a2ab-4d63-bb99-48d8e5e447bc.png"/>
|
|
|
|
|
|
######
|
|
* langs: csharp
|
|
|
|
|
|
|
|
## Opening the Workflows
|
|
|
|
Click on the **Actions** tab to see the workflows. Here you will see if your tests have passed or failed.
|
|
|
|
######
|
|
* langs: js, python, java
|
|
|
|
|
|
|
|
######
|
|
* langs: csharp
|
|
|
|
|
|
|
|
On Pull Requests you can also click on the **Details** link in the [PR status check](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks).
|
|
|
|
<img width="645" alt="pr status checked" src="https://user-images.githubusercontent.com/13063165/183722462-17a985db-0e10-4205-b16c-8aaac36117b9.png" />
|
|
|
|
|
|
## Viewing Test Logs
|
|
|
|
Clicking on the workflow run will show you the all the actions that GitHub performed and clicking on **Run Playwright tests** will show the error messages, what was expected and what was received as well as the call log.
|
|
|
|
######
|
|
* langs: js, python, java
|
|
|
|
|
|
|
|
######
|
|
* langs: csharp
|
|
|
|
|
|
|
|
|
|
## HTML Report
|
|
* langs: js
|
|
|
|
The HTML Report shows you a full report of your tests. You can filter the report by browsers, passed tests, failed tests, skipped tests and flaky tests.
|
|
|
|
### Downloading the HTML Report
|
|
* langs: js
|
|
|
|
In the Artifacts section click on the **playwright-report** to download your report in the format of a zip file.
|
|
|
|
<img width="972" alt="Downloading the HTML Report" src="https://user-images.githubusercontent.com/13063165/183437023-524f1803-84e4-4862-9ce3-1d55af0e023e.png" />
|
|
|
|
### Viewing the HTML Report
|
|
* langs: js
|
|
|
|
Locally opening the report will not work as expected as you need a web server in order for everything to work correctly. First, extract the zip, preferably in a folder that already has Playwright installed. Using the command line change into the directory where the report is and use `npx playwright show-report` followed by the name of the extracted folder. This will serve up the report and enable you to view it in your browser.
|
|
|
|
|
|
```bash
|
|
npx playwright show-report name-of-my-extracted-playwright-report
|
|
```
|
|
|
|
|
|
|
|
To learn more about reports check out our detailed guide on [HTML Reporter](/test-reporters.md#html-reporter)
|
|
|
|
## Viewing the Trace
|
|
* langs: js
|
|
|
|
Once you have served the report using `npx playwright show-report`, click on the trace icon next to the test's file name as seen in the image above. You can then view the trace of your tests and inspect each action to try to find out why the tests are failing.
|
|
|
|
|
|
|
|
## Viewing the Trace
|
|
* langs: python, java
|
|
|
|
[trace.playwright.dev](https://trace.playwright.dev) is a statically hosted variant of the Trace Viewer. You can upload trace files using drag and drop.
|
|
|
|
|
|
|
|
## Viewing the Trace
|
|
* langs: csharp
|
|
|
|
You can upload Traces which get created on your CI like GitHub Actions as artifacts. This requires [starting and stopping the trace](./trace-viewer-intro#recording-a-trace). We recommend only recording traces for failing tests. Once your traces have been uploaded to CI, they can then be downloaded and opened using [trace.playwright.dev](https://trace.playwright.dev), which is a statically hosted variant of the Trace Viewer. You can upload trace files using drag and drop.
|
|
|
|
######
|
|
* langs: csharp
|
|
|
|
|
|
|
|
## Publishing report on the web
|
|
* langs: js
|
|
|
|
Downloading the HTML report as a zip file is not very convenient. However, we can utilize Azure Storage's static websites hosting capabilities to easily and efficiently serve HTML reports on the Internet, requiring minimal configuration.
|
|
|
|
1. Create an [Azure Storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create).
|
|
1. Enable [Static website hosting](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-how-to#enable-static-website-hosting) for the storage account.
|
|
1. Create a Service Principal in Azure and grant it access to Azure Blob storage. Upon successful execution, the command will display the credentials which will be used in the next step.
|
|
|
|
```bash
|
|
az ad sp create-for-rbac --name "github-actions" --role "Storage Blob Data Contributor" --scopes /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME>
|
|
```
|
|
1. Use the credentials from the previous step to set up encrypted secrets in your GitHub repository. Go to your repository's settings, under [GitHub Actions secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository), and add the following secrets:
|
|
|
|
- `AZCOPY_SPA_APPLICATION_ID`
|
|
- `AZCOPY_SPA_CLIENT_SECRET`
|
|
- `AZCOPY_TENANT_ID`
|
|
|
|
For a detailed guide on how to authorize a service principal using a client secret, refer to [this Microsoft documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory#authorize-a-service-principal-by-using-a-client-secret-1).
|
|
1. Add a step that uploads the HTML report to Azure Storage.
|
|
|
|
```yaml title=".github/workflows/playwright.yml"
|
|
...
|
|
- name: Upload HTML report to Azure
|
|
shell: bash
|
|
run: |
|
|
REPORT_DIR='run-${{ github.run_id }}-${{ github.run_attempt }}'
|
|
azcopy cp --recursive "./playwright-report/*" "https://<STORAGE_ACCOUNT_NAME>.blob.core.windows.net/\$web/$REPORT_DIR"
|
|
echo "::notice title=HTML report url::https://<STORAGE_ACCOUNT_NAME>.z1.web.core.windows.net/$REPORT_DIR/index.html"
|
|
env:
|
|
AZCOPY_AUTO_LOGIN_TYPE: SPN
|
|
AZCOPY_SPA_APPLICATION_ID: '${{ secrets.AZCOPY_SPA_APPLICATION_ID }}'
|
|
AZCOPY_SPA_CLIENT_SECRET: '${{ secrets.AZCOPY_SPA_CLIENT_SECRET }}'
|
|
AZCOPY_TENANT_ID: '${{ secrets.AZCOPY_TENANT_ID }}'
|
|
```
|
|
|
|
The contents of the `$web` storage container can be accessed from a browser by using the [public URL](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-how-to?tabs=azure-portal#portal-find-url) of the website.
|
|
|
|
:::note
|
|
This step will not work for pull requests created from a forked repository because such workflow [doesn't have access to the secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#using-encrypted-secrets-in-a-workflow).
|
|
:::
|
|
|
|
|
|
## What's Next
|
|
|
|
- [Learn how to use Locators](./locators.md)
|
|
- [Learn how to perform Actions](./input.md)
|
|
- [Learn how to write Assertions](./test-assertions.md)
|
|
- [Learn more about the Trace Viewer](/trace-viewer.md)
|
|
- [Learn more ways of running tests on GitHub Actions](/ci.md#github-actions)
|
|
- [Learn more about running tests on other CI providers](/ci.md)
|