mirror of
https://github.com/strapi/strapi.git
synced 2025-12-27 23:24:03 +00:00
fix(e2e): update data import and export scripts (#22071)
This commit is contained in:
markkaylor
GitHub
parent
af7e4e2471
commit
4119cc523a
@ -9,83 +9,79 @@ tags:
|
||||
|
||||
## Overview
|
||||
|
||||
This document looks particularly at the `@strapi/data-transfer` package framed around how we use it with our test instances in the e2e test suite. It is not a comprehensive explanation of how to use `@strapi/data-transfer`. For that information you should [view the documentation](https://docs.strapi.io/developer-docs/latest/developer-resources/data-management.html) surrounding it.
|
||||
This document explains how and why we use `@strapi/data-transfer` as means to reset and seed the database for end-to-end tests. It is not a comprehensive explanation of how to use `@strapi/data-transfer`. See the [Strapi documentation](https://docs.strapi.io/developer-docs/latest/developer-resources/data-management.html) to learn more about the feature.
|
||||
|
||||
## Why use Data Transfer?
|
||||
### Why use Data Transfer?
|
||||
|
||||
Each test should be isolated and contained e.g. if you edit an entity in one test, the next test shouldn't know or care about it otherwise your tests need to run in a specific order and become flakey quite quickly.
|
||||
We could use custom API endpoints of the application, and whilst this isn't a poor solution, it would _most likely_ require some code writing to set up the data for the schema entries. However, in `4.6.0` Strapi released the `DTS` feature (DTS – Data Transfer System). This means any member of Strapi can export the data of their instance producing a `.tar` that we can then import programatically restoring the database to this point in time and ensuring a "pure" test environment.
|
||||
|
||||
So to solve this, you could use custom API endpoints of the application and whilst this isn't a poor solution it would _most likely_ require some code writing to set up the data for the schema entries. However, in `4.6.0` Strapi released the `DTS` feature (DTS – Data Transfer System). This means any member of Strapi can export the data of their instance producing a `.tar` that we can then import programatically restoring the database to this point in time and ensuring a "pure" test environment.
|
||||
### Limitations of Data Transfer
|
||||
|
||||
## Using Data Transfer
|
||||
The main limitation with data transfer is we cannot version or review changes to the data. Making changes to the data set should be done with care since it is quite easy to export data with unknown changes to the database that could impact other tests.
|
||||
|
||||
The full documentation of the feature can be seen [here](https://docs.strapi.io/developer-docs/latest/developer-resources/data-management.html). Below are a couple of scenarios you might find yourself.
|
||||
## Updating data for tests
|
||||
|
||||
### Creating a data packet
|
||||
Each test should be isolated and not depend on another test. Data changes from one test should not leak into another test. For example, if you create a new entry for a content-type in one test case, it should not be present in the next test case. This makes tests more stable and easier to debug.
|
||||
|
||||
Because we're using a Strapi template for the test instance, it makes the most sense to add/edit the dataset in said templated instance. Begin by creating the instance:
|
||||
### The data transfer engine
|
||||
|
||||
Since the Strapi CLI will use `@strapi/data-transfer` directly it will by default not import or export admin users, API tokens, or any other features that have been included in its exclusion list.
|
||||
|
||||
For this reason, do NOT use the import or export command on the strapi test instance. A DTS engine has been created specifically for our tests cases. This allows us to redefine what should be included in the import or export for our tests. The scripts can be found in `tests/e2e/scripts/dts-import.ts` and `tests/e2e/scripts/dts-export.ts`.
|
||||
|
||||
### Importing an existing data packet
|
||||
|
||||
When you need to update the data packet for a new test, you will first need a Strapi app with the data currently used in end-to-end tests.
|
||||
|
||||
When running the `yarn test:e2e` command, test app instances are created in `test-apps/e2e/test-app-{n}`. You can use one of the these apps to update the data.
|
||||
|
||||
Navigate to one of the test-apps and run `yarn install && yarn develop`
|
||||
|
||||
Leave the development server running, and then run the following command to reset and seed the database with the current e2e data packet. The script expects the name of the data packet you want to import found in `tests/e2e/data`.
|
||||
|
||||
```shell
|
||||
yarn test:e2e
|
||||
STRAPI_LICENSE=<license-with-ee-feature> npx ts-node <PATH_TO_SCRIPT>/dts-import.ts with-admin.tar
|
||||
```
|
||||
|
||||
Then, you should be able to navigate to the app – `cd ./test-apps/test-app-XX`, the current content schemas should already be defined in there so you will be able to instantly import the current data packet to bring to life the test instance (instead of it being fresh):
|
||||
This script will include admin users and all the content-types specificed in `tests/e2e/constants.ts`
|
||||
|
||||
```shell
|
||||
yarn strapi import --file ../../../e2e/data/<backup-file-name>.tar
|
||||
```
|
||||
You should be able to login with the test app instance credentials.
|
||||
|
||||
Once that's completed, you should be able to run your Strapi instance as usual:
|
||||
| Email | Password |
|
||||
| ---------------- | ----------- |
|
||||
| test@testing.com | Testing123! |
|
||||
|
||||
```shell
|
||||
yarn develop
|
||||
```
|
||||
|
||||
> **Tip!**
|
||||
> If you can't run the test-app because it is not present in the monorepo dependencies you can fix this by making the following change to the root monorepo `package.json` file and running `yarn install` at both the root of the monorepo and the test-app you are running.
|
||||
>
|
||||
> **This change should not be committed**.
|
||||
|
||||
```
|
||||
"workspaces": [
|
||||
...
|
||||
"test-apps/e2e/*",
|
||||
]
|
||||
```
|
||||
Now that you have a Strapi instance with the same data that each e2e starts with, you can modify the data in the CMS to prepare for a new data export.
|
||||
|
||||
:::note
|
||||
If you change any of the content schemas (including adding new ones) be sure to [update the `app-template`](./01-app-template.md) otherwise DTS will fail to import the data for schemas that do not exist.
|
||||
:::
|
||||
|
||||
### Exporting a data packet
|
||||
### Exporting an updated data packet
|
||||
|
||||
Once you've created your new data from the test instance, you'll need to export
|
||||
it. Since the Strapi CLI will use `@strapi/data-transfer` directly it will by default not export admin users, API tokens, or any other features that have been included in its exclusion list. For this reason, do not use the export command on the strapi test instance. A DTS engine has been created specifically for our tests cases. This allows us to redefine what should be included in the export for our tests. The script can be found in `/e2e/scripts/dts-export.js`
|
||||
Once you've created your new data from the test instance, you'll need to export it so it can be used in the end-to-end tests.
|
||||
|
||||
Be sure to include the content types you would like exported in the `ALLOWED_CONTENT_TYPES` array found in `e2e/constants.js`.
|
||||
Be sure to include the content types you would like exported in the `ALLOWED_CONTENT_TYPES` array found in `tests/e2e/constants.js`.
|
||||
|
||||
The script accepts the backup destination filename as a parameter. Run it from the directory
|
||||
of your strapi test insance to create the backup.
|
||||
The script accepts the backup destination filename as an argument. Run it from the directory of the strapi instance you created earlier based on the test-app template.
|
||||
|
||||
```shell
|
||||
node <PATH_TO_SCRIPT>/dts-export.js backup-with-admin-user
|
||||
npx ts-node <PATH_TO_SCRIPT>/dts-export.ts updated-data-packet
|
||||
```
|
||||
|
||||
If you are exporting data for an EE feature you will need to run the script with the `STRAPI_LICENSE` env
|
||||
|
||||
```shell
|
||||
STRAPI_LICENSE=<license-with-ee-feature> node <PATH_TO_SCRIPT>/dts-export.js backup-with-admin-user
|
||||
STRAPI_LICENSE=<license-with-ee-feature> npx ts-node <PATH_TO_SCRIPT>/dts-export.ts updated-data-packet
|
||||
```
|
||||
|
||||
Once this has been done, add the `.tar` backup to `/e2e/data` so the helper
|
||||
functions can import it correctly.
|
||||
The script will create a file `updated-data-packet.tar`. You can copy this file over to `tests/e2e/data` so it can be used in the appropriate tests.
|
||||
|
||||
As our suite of e2e tests grows we may hold more DTS backups in order to restore
|
||||
the Strapi application to a desired state prior to testing.
|
||||
### Importing the data packet in test scenarios
|
||||
|
||||
### Importing in test scenarios
|
||||
There's an abstraction for importing the data programmatically during tests named `resetDatabaseAndImportDataFromPath` found in `tests/e2e/utils/dts-import.ts`. Typically, you'll want to run this **before** each test:
|
||||
|
||||
There's an abstraction for importing the data programmatically during tests named `resetDatabaseAndImportDataFromPath` found in `e2e/utils/dts-import.js`. Typically, you'll want to run this **before** each test:
|
||||
|
||||
```ts {2,5-8}
|
||||
```ts
|
||||
import { test } from '@playwright/test';
|
||||
import { resetDatabaseAndImportDataFromPath } from './utils/dts-import';
|
||||
|
||||
@ -100,5 +96,3 @@ test.describe('Strapi Application', () => {
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
The path is relative to the root of the strapi project where you called `yarn test:e2e`.
|
||||
|
||||
Binary file not shown.
@ -1,4 +1,4 @@
|
||||
const { exportData } = require('../utils/dts-export.ts');
|
||||
import { exportData } from '../utils/dts-export';
|
||||
|
||||
// TODO: make an actual yargs command and pass common options to exportData so it's easier to build the test data
|
||||
exportData();
|
||||
|
||||
17
tests/e2e/scripts/dts-import.ts
Normal file
17
tests/e2e/scripts/dts-import.ts
Normal file
@ -0,0 +1,17 @@
|
||||
import { resetDatabaseAndImportDataFromPath } from '../utils/dts-import';
|
||||
|
||||
const importData = async () => {
|
||||
const args = process.argv.slice(2);
|
||||
const filePath = args[0];
|
||||
|
||||
if (!filePath) {
|
||||
console.error('Please provide the name of the file you want to import from tests/e2e/data');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
await resetDatabaseAndImportDataFromPath(filePath);
|
||||
console.log('Data transfer succeeded');
|
||||
process.exit(0);
|
||||
};
|
||||
|
||||
importData();
|
||||
@ -1,4 +1,8 @@
|
||||
'use strict';
|
||||
import type { Core } from '@strapi/strapi';
|
||||
|
||||
import dts from '@strapi/data-transfer';
|
||||
import { createStrapi } from '@strapi/strapi';
|
||||
import { ALLOWED_CONTENT_TYPES } from '../constants';
|
||||
|
||||
const {
|
||||
file: {
|
||||
@ -8,18 +12,10 @@ const {
|
||||
providers: { createLocalStrapiSourceProvider },
|
||||
},
|
||||
engine: { createTransferEngine },
|
||||
} = require('@strapi/data-transfer');
|
||||
const { createStrapi } = require('@strapi/strapi');
|
||||
const { ALLOWED_CONTENT_TYPES } = require('../constants');
|
||||
} = dts;
|
||||
|
||||
/**
|
||||
* Export the data from a strapi project.
|
||||
* This script should be run as `node <path-to>/dts-export.js [exportFilePath]` from the
|
||||
* root directory of a strapi project e.g. `/examples/kitchensink`. Remember to import
|
||||
* the `with-admin` tar file into the project first because the tests rely on the data.
|
||||
*/
|
||||
const exportData = async () => {
|
||||
let args = process.argv.slice(2);
|
||||
export const exportData = async (): Promise<void> => {
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
if (args.length !== 1) {
|
||||
console.error('Please provide the export file name as a parameter.');
|
||||
@ -69,21 +65,21 @@ const exportData = async () => {
|
||||
process.exit(0);
|
||||
};
|
||||
|
||||
const createSourceProvider = (strapi) =>
|
||||
const createSourceProvider = (strapi: Core.Strapi) =>
|
||||
createLocalStrapiSourceProvider({
|
||||
async getStrapi() {
|
||||
return strapi;
|
||||
},
|
||||
});
|
||||
|
||||
const createDestinationProvider = (filePath) =>
|
||||
const createDestinationProvider = (filePath: string) =>
|
||||
createLocalFileDestinationProvider({
|
||||
file: { path: filePath },
|
||||
encryption: { enabled: false },
|
||||
compression: { enabled: false },
|
||||
});
|
||||
|
||||
const createStrapiInstance = async (logLevel = 'error') => {
|
||||
const createStrapiInstance = async (logLevel = 'error'): Promise<Core.Strapi> => {
|
||||
const app = createStrapi();
|
||||
|
||||
app.log.level = logLevel;
|
||||
@ -91,7 +87,3 @@ const createStrapiInstance = async (logLevel = 'error') => {
|
||||
|
||||
return loadedApp;
|
||||
};
|
||||
|
||||
module.exports = {
|
||||
exportData,
|
||||
};
|
||||
|
||||
@ -25,7 +25,7 @@ export const resetDatabaseAndImportDataFromPath = async (
|
||||
modifiedContentTypesFn: (cts: string[]) => string[] = (cts) => cts,
|
||||
configuration: RestoreConfiguration = { coreStore: true }
|
||||
) => {
|
||||
const filePath = join('./tests/e2e/data/', file);
|
||||
const filePath = resolve(__dirname, '../data/', file);
|
||||
const source = createSourceProvider(filePath);
|
||||
const includedTypes = modifiedContentTypesFn(ALLOWED_CONTENT_TYPES);
|
||||
const destination = createDestinationProvider(includedTypes, configuration);
|
||||
@ -37,7 +37,7 @@ export const resetDatabaseAndImportDataFromPath = async (
|
||||
links: [
|
||||
{
|
||||
// only transfer relations to+from requested content types
|
||||
filter(link) {
|
||||
filter(link: any) {
|
||||
return (
|
||||
includedTypes.includes(link.left.type) &&
|
||||
(includedTypes.includes(link.right.type) || link.right.type === undefined)
|
||||
@ -48,7 +48,7 @@ export const resetDatabaseAndImportDataFromPath = async (
|
||||
entities: [
|
||||
{
|
||||
// only include entities of requested content types
|
||||
filter(entity) {
|
||||
filter(entity: any) {
|
||||
return includedTypes.includes(entity.type);
|
||||
},
|
||||
},
|
||||
@ -60,12 +60,9 @@ export const resetDatabaseAndImportDataFromPath = async (
|
||||
|
||||
try {
|
||||
// reset the transfer token to allow the transfer if it's been wiped (that is, not included in previous import data)
|
||||
const res = await fetch(
|
||||
`http://127.0.0.1:${process.env.PORT ?? 1337}/api/config/resettransfertoken`,
|
||||
{
|
||||
method: 'POST',
|
||||
}
|
||||
);
|
||||
await fetch(`http://127.0.0.1:${process.env.PORT ?? 1337}/api/config/resettransfertoken`, {
|
||||
method: 'POST',
|
||||
});
|
||||
} catch (err) {
|
||||
console.error('Token reset failed.' + JSON.stringify(err, null, 2));
|
||||
process.exit(1);
|
||||
@ -79,14 +76,17 @@ export const resetDatabaseAndImportDataFromPath = async (
|
||||
}
|
||||
};
|
||||
|
||||
const createSourceProvider = (filePath) =>
|
||||
const createSourceProvider = (filePath: string) =>
|
||||
createLocalFileSourceProvider({
|
||||
file: { path: resolve(filePath) },
|
||||
file: { path: filePath },
|
||||
encryption: { enabled: false },
|
||||
compression: { enabled: false },
|
||||
});
|
||||
|
||||
const createDestinationProvider = (includedTypes = [], configuration: RestoreConfiguration) => {
|
||||
const createDestinationProvider = (
|
||||
includedTypes: any[] = [],
|
||||
configuration: RestoreConfiguration
|
||||
) => {
|
||||
return createRemoteStrapiDestinationProvider({
|
||||
url: new URL(`http://127.0.0.1:${process.env.PORT ?? 1337}/admin`),
|
||||
auth: { type: 'token', token: CUSTOM_TRANSFER_TOKEN_ACCESS_KEY },
|
||||
|
||||
@ -96,7 +96,7 @@ yargs
|
||||
async handler(argv) {
|
||||
try {
|
||||
if (await pathExists(path.join(testRoot, '.env'))) {
|
||||
// Run tests with the env variables specified in the e2e/app-template/.env
|
||||
// Run tests with the env variables specified in the e2e/.env file
|
||||
dotenv.config({ path: path.join(testRoot, '.env') });
|
||||
}
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user