From 09813c23dd5f8815ed0bf56f0afab67e5eb433cb Mon Sep 17 00:00:00 2001 From: Imri Paran Date: Wed, 13 Mar 2024 11:30:11 +0100 Subject: [PATCH] removed docs 1.4.x (#15550) --- .../deployment/backup-restore-metadata.md | 274 ++++-------------- .../deployment/backup-restore-metadata.md | 85 ------ 2 files changed, 52 insertions(+), 307 deletions(-) delete mode 100644 openmetadata-docs/content/v1.4.x/deployment/backup-restore-metadata.md diff --git a/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/backup-restore-metadata.md b/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/backup-restore-metadata.md index 67d83ceb4b4..1355e4947a8 100644 --- a/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/backup-restore-metadata.md +++ b/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/backup-restore-metadata.md @@ -7,249 +7,79 @@ slug: /deployment/backup-restore-metadata ## Introduction -The goal of OpenMetadata is to enable company-wide collaboration around metadata. The more we use it, the more value -this brings to the table, which means that keeping the metadata safe can become a critical activity for our Disaster -Recovery practices. +Since version 1.4.0, OpenMetadata's encourages using the builtin-tools for creating logical backups of the metadata: -While there are cloud services that feature automatic snapshots and replication, the metadata CLI -now allows all users to perform backups regardless of the underlying infrastructure. +- `mysqldump` for MySQL +- `pg_dump` for Postgres ## Requirements -The backup CLI needs to be used with `openmetadata-ingestion` version 0.13.1 or higher. +- mysqldump 8.3 or higher (ingestion container is shipped with mysqldump 8.3) +- pg_dump 13.3 or higher -However, we recommend to make sure that you are always in the latest `openmetadata-ingestion` -version to have all the improvements shipped in the CLI. +# Example -## Installation +Start a local instance of OpenMetadata using the `docker-compose` file provided in the repository. Then, we can use the following commands to backup the metadata: -The CLI comes bundled in the base `openmetadata-ingestion` Python package. You can install it with: +## MySQL -```commandline -pip install openmetadata-ingestion +### 1. Start a local docker deployment + +```shell +docker/run_local_docker.sh ``` -One of the `backup` features is to upload the generated backup to cloud storage (currently supporting S3 and Azure Blob). To use this, -you can instead install the package with the backup plugin: +Ingest some data... -```commandline -pip install "openmetadata-ingestion[backup,mysql]" +### 2. Backup and Restore + +```shell +BACKUP_FILE="backup_$(date +%Y%m%d%H%M).sql" +DOCKER_COMPOSE_FILE="docker/development/docker-compose.yml" +# backup +docker compose -f $DOCKER_COMPOSE_FILE exec ingestion mysqldump --no-tablespaces -u openmetadata_user -popenmetadata_password -h mysql -P 3306 openmetadata_db > $BACKUP_FILE +# create the restore database +docker compose -f $DOCKER_COMPOSE_FILE exec mysql mysql -u root -ppassword -e "create database restore;" +docker compose -f $DOCKER_COMPOSE_FILE exec mysql mysql -u root -ppassword -e "grant all privileges on restore.* to 'openmetadata_user'@'%';" +docker compose -f $DOCKER_COMPOSE_FILE exec mysql mysql -u root -ppassword -e "flush privileges;" +# restore from the backup +docker compose -f $DOCKER_COMPOSE_FILE exec -T ingestion mysql -u openmetadata_user -popenmetadata_password -h mysql -P 3306 restore < $BACKUP_FILE ``` -## Requirements & Considerations +### 3. Restart the docker deployment with the restored database -This is a custom utility. As almost all tables contain `GENERATED` columns, directly using `mysqldump` is not an -option out of the box, as it would require some further cleaning steps to get the data right. - -Instead, we have created a utility that will just dump the necessary data. - -The requirement for running the process is that the target database should have the Flyway migrations executed. - -The backup utility will provide an SQL file which will do two things: - -1. TRUNCATE the OpenMetadata tables -2. INSERT the data that has been saved - -You can then run the script's statements to restore the data. - -{% note %} - -Make sure that the migrations have been run correctly (find out how [here](/deployment/bare-metal#4.-prepare-the-openmetadata-database-and-indexes)). - -Also, make sure that the target database does not already have any OpenMetadata data, or if it does, that you are OK -replacing it with whatever comes from the SQL script. - -{% /note %} - -## Backup CLI - -After the installation, we can take a look at the different options to run the CLI: - -```commandline -> metadata backup -h -usage: metadata backup [-h] -H HOST -u USER -p PASSWORD -d DATABASE [--port PORT] [--output OUTPUT] - [--upload-destination-type {AWS,AZURE}] [--upload UPLOAD UPLOAD UPLOAD] [-o OPTIONS] [-a ARGUMENTS] - [-s SCHEMA] - -optional arguments: - -h, --help show this help message and exit - -H HOST, --host HOST Host that runs the database - -u USER, --user USER User to run the backup - -p PASSWORD, --password PASSWORD - Credentials for the user - -d DATABASE, --database DATABASE - Database to backup - --port PORT Database service port - --output OUTPUT Local path to store the backup - --upload-destination-type {AWS,AZURE} - AWS or AZURE - --upload UPLOAD UPLOAD UPLOAD - S3 endpoint, bucket & key to upload the backup file - -o OPTIONS, --options OPTIONS - -a ARGUMENTS, --arguments ARGUMENTS - -s SCHEMA, --schema SCHEMA +```shell +export OM_DATABASE=restore +docker compose -f $DOCKER_COMPOSE_FILE up -d ``` -If using MySQL: +## PostgreSQL -```bash -python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306 +### 1. Start a local docker deployment + +```shell +docker/run_local_docker.sh -d postgres ``` -If using Postgres, add the `-s` parameter specifying the `schema`: +Ingest some data... -```bash -python -m metadata backup -u openmetadata_user -p openmetadata_password -H postgresql -d openmetadata_db --port 5432 -s public +### 2. Backup and Restore + +```shell +BACKUP_FILE="backup_$(date +%Y%m%d%H%M).sql" +DOCKER_COMPOSE_FILE="docker/development/docker-compose-postgres.yml" +# backup +docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password ingestion pg_dump -U openmetadata_user -h postgresql -d openmetadata_db > $BACKUP_FILE +# create the restore database +docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password postgresql psql -U postgres -c "create database restore;" +docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password postgresql psql -U postgres -c "ALTER DATABASE restore OWNER TO openmetadata_user;" +# restore from the backup +docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password -T ingestion psql -U openmetadata_user -h postgresql -d restore < $BACKUP_FILE ``` -### Database Connection +### 3. Restart the docker deployment with the restored database -There is a set of four required parameters, the minimum required for us to access the database service and run the -backup: `host`, `user`, `password` and `database` to point to. Note that the user should have at least read access to the -database. By default, we'll try to connect through the port `3306`, but this can be overridden with the `--port` option. - -### Output - -The CLI will create a dump file that looks like `openmetadata_YYYYmmddHHMM_backup.sql`. This will help us identify the -date each backup was generated. We can also specify an output path, which we'll create if it does not exist, via -`--output`. - -### Uploading to S3 - -To run this, make sure to have `AWS_ACCESS_KEY_ID` and -`AWS_SECRET_ACCESS_KEY` as environment variables with permissions to the bucket that you'd like to point to. Afterwards, -we can just use `--upload ` to have the CLI upload the file. In this case, you'll get both the -local dump file and the one in the cloud. - -### Uploading to Azure Blob - - -To run this, make sure to have Azure CLI configured with permissions to the Blob that you'd like to point to. Afterwards, -we can just use `--upload ` to have the CLI upload the file. In this case, you'll get both the -local dump file and the one in the cloud. - -### Connection Options and Arguments - -You can pass any required connection options or arguments to the MySQL connection via `-o , -o [...]` -or `-a , -a [...]`. - -### Trying it out - -We can do a test locally preparing some containers: - -1. `sh docker/run_local_docker.sh` to start the `docker compose` service. -2. `docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"` to start minio, an object - storage S3 compatible. -3. Connect to [http://localhost:9001](http://localhost:9001) to reach the minio console and create a bucket - called `my-bucket` -4. Finally, we just need to prepare the environment variables as: - ``` - export AWS_ACCESS_KEY_ID=minioadmin - export AWS_SECRET_ACCESS_KEY=minioadmin - ``` - -An example of S3 CLI call will look as: - -```commandline -metadata backup -u openmetadata_user -p openmetadata_password \ - -H localhost -d openmetadata_db --output=dir1/dir2 \ - --upload-destination-type AWS \ - --upload http://localhost:9000 my-bucket backup/ +```shell +export OM_DATABASE=restore +docker compose -f $DOCKER_COMPOSE_FILE up -d ``` - -And we'll get the following output: - -```commandline -Creating OpenMetadata backup for localhost:3306/openmetadata_db... -Backup stored locally under dir1/dir2/openmetadata_202201250823_backup.sql -Uploading dir1/dir2/openmetadata_202201250823_backup.sql to http://localhost:9000/my-bucket/backup/openmetadata_202201250823_backup.sql... -``` - -If we now head to the minio console and check the `my-backup` bucket, we'll see our SQL dump in there. - -{% image src="/images/v1.4/deployment/backup/minio-example.png" alt="minio" /%} - -An example of Azure Blob CLI call will look as: - -```commandline -metadata backup -u openmetadata_user -p openmetadata_password \ - -H localhost -d openmetadata_db --output=dir1/dir2 \ - --upload-destination-type AZURE \ - --upload https://container.blob.core.windows.net/ container-name Folder-name/ -``` -And we'll get the following output: - -```commandline -Creating OpenMetadata backup for localhost:3306/openmetadata_db... -Backup stored locally under openmetadata_202212161559_backup.sql -Uploading openmetadata_202212161559_backup.sql to https://container.blob.core.windows.net//container-name... -``` - -# Restore Metadata - -Make sure that when restoring metadata, your OpenMetadata server is **NOT RUNNING**. Otherwise, there could be -clashes on cached values and specific elements that the server creates at start-time and that will be present -on the restore SQL file as well. - -## Introduction - -SQL file which is generated using Backup metadata CLI -can restore using Restore metadata CLI. - -## Requirements - -The restore CLI needs to be used with `openmetadata-ingestion` version 0.12.1 or higher. - -## Restore CLI - -After the installation, we can take a look at the different options to run the CLI: - -```commandline -> metadata restore -h -usage: metadata restore [-h] -H HOST -u USER -p PASSWORD -d DATABASE [--port PORT] --input INPUT [-o OPTIONS] - [-a ARGUMENTS] [-s SCHEMA] - -optional arguments: - -h, --help show this help message and exit - -H HOST, --host HOST Host that runs the database - -u USER, --user USER User to run the restore backup - -p PASSWORD, --password PASSWORD - Credentials for the user - -d DATABASE, --database DATABASE - Database to restore - --port PORT Database service port - --input INPUT Local backup file path for restore - -o OPTIONS, --options OPTIONS - -a ARGUMENTS, --arguments ARGUMENTS - -s SCHEMA, --schema SCHEMA -``` - -### Output - -The CLI will give messages like this `Backup restored from openmetadata_202209301715_backup.sql` when backup restored completed. - -### Trying it out - -An example CLI call will look as: - -```commandline -metadata restore -u openmetadata_user -p openmetadata_password -H localhost -d openmetadata_db --input openmetadata_202209301715_backup.sql -``` - -And we'll get the following output: - -```commandline -Restoring OpenMetadata backup for localhost:3306/openmetadata_db... -Backup restored from openmetadata_202209301715_backup.sql -``` - -## Troubleshooting - -It is not recommended to restore a backup of an older version to a newer version. Doing this may result into number of issues since the data is not in the expected shape, as migration has not been performed on this data yet. Read [this document](https://docs.open-metadata.org/deployment/upgrade/how-does-it-work) to understand how does the backup and restore works in OpenMetadata. - -In case you are facing some issues because you restored the backup of older version into a newer version. Follow the below steps to resolve the issues: - -1. Delete the current OpenMetadata instance and clean all the data, i.e. delete the `openmetadata_db` from your database source. In case you are using docker or kubernetes mode of deployment then clean all your volumes attached to database image. -2. Spin up a new instance of OpenMetadata of the older version. For example, if your backup file was generated by version 1.1.0 then spin up a fresh instance of OpenMetadata version 1.1.0. -3. Restore the data using the `metadata restore` command as described in the above doc. -4. Once the restoration process is completed follow the steps defined in [this document](https://docs.open-metadata.org/v1.4.x/deployment/upgrade) to upgrade your instance to the newer version depending on your deployment mode. \ No newline at end of file diff --git a/openmetadata-docs/content/v1.4.x/deployment/backup-restore-metadata.md b/openmetadata-docs/content/v1.4.x/deployment/backup-restore-metadata.md deleted file mode 100644 index 1355e4947a8..00000000000 --- a/openmetadata-docs/content/v1.4.x/deployment/backup-restore-metadata.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -title: Backup Metadata -slug: /deployment/backup-restore-metadata ---- - -# Backup & Restore Metadata - -## Introduction - -Since version 1.4.0, OpenMetadata's encourages using the builtin-tools for creating logical backups of the metadata: - -- `mysqldump` for MySQL -- `pg_dump` for Postgres - -## Requirements - -- mysqldump 8.3 or higher (ingestion container is shipped with mysqldump 8.3) -- pg_dump 13.3 or higher - -# Example - -Start a local instance of OpenMetadata using the `docker-compose` file provided in the repository. Then, we can use the following commands to backup the metadata: - -## MySQL - -### 1. Start a local docker deployment - -```shell -docker/run_local_docker.sh -``` - -Ingest some data... - -### 2. Backup and Restore - -```shell -BACKUP_FILE="backup_$(date +%Y%m%d%H%M).sql" -DOCKER_COMPOSE_FILE="docker/development/docker-compose.yml" -# backup -docker compose -f $DOCKER_COMPOSE_FILE exec ingestion mysqldump --no-tablespaces -u openmetadata_user -popenmetadata_password -h mysql -P 3306 openmetadata_db > $BACKUP_FILE -# create the restore database -docker compose -f $DOCKER_COMPOSE_FILE exec mysql mysql -u root -ppassword -e "create database restore;" -docker compose -f $DOCKER_COMPOSE_FILE exec mysql mysql -u root -ppassword -e "grant all privileges on restore.* to 'openmetadata_user'@'%';" -docker compose -f $DOCKER_COMPOSE_FILE exec mysql mysql -u root -ppassword -e "flush privileges;" -# restore from the backup -docker compose -f $DOCKER_COMPOSE_FILE exec -T ingestion mysql -u openmetadata_user -popenmetadata_password -h mysql -P 3306 restore < $BACKUP_FILE -``` - -### 3. Restart the docker deployment with the restored database - -```shell -export OM_DATABASE=restore -docker compose -f $DOCKER_COMPOSE_FILE up -d -``` - -## PostgreSQL - -### 1. Start a local docker deployment - -```shell -docker/run_local_docker.sh -d postgres -``` - -Ingest some data... - -### 2. Backup and Restore - -```shell -BACKUP_FILE="backup_$(date +%Y%m%d%H%M).sql" -DOCKER_COMPOSE_FILE="docker/development/docker-compose-postgres.yml" -# backup -docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password ingestion pg_dump -U openmetadata_user -h postgresql -d openmetadata_db > $BACKUP_FILE -# create the restore database -docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password postgresql psql -U postgres -c "create database restore;" -docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password postgresql psql -U postgres -c "ALTER DATABASE restore OWNER TO openmetadata_user;" -# restore from the backup -docker compose -f $DOCKER_COMPOSE_FILE exec -e PGPASSWORD=openmetadata_password -T ingestion psql -U openmetadata_user -h postgresql -d restore < $BACKUP_FILE -``` - -### 3. Restart the docker deployment with the restored database - -```shell -export OM_DATABASE=restore -docker compose -f $DOCKER_COMPOSE_FILE up -d -```