From 0896be2513b8f8072ef5e40663e5a18a78fed97b Mon Sep 17 00:00:00 2001 From: Akash Jain <15995028+akash-jain-10@users.noreply.github.com> Date: Fri, 8 Mar 2024 12:54:01 +0530 Subject: [PATCH] fix(docs): Unify Prerequisites of Kubernetes Deployment Docs (#15480) * fix(docs): Unify Prerequisites for kubernetes deployments * fix(docs): specify version as `higher` instead of `greater` --- .../v1.3.x/deployment/bare-metal/index.md | 12 ++-- .../content/v1.3.x/deployment/docker/index.md | 6 +- .../v1.3.x/deployment/kubernetes/aks.md | 56 +++++++++++++++++++ .../v1.3.x/deployment/kubernetes/eks.md | 6 +- .../v1.3.x/deployment/kubernetes/gke.md | 6 +- .../v1.3.x/deployment/kubernetes/on-prem.md | 49 ++++++++++++++++ 6 files changed, 120 insertions(+), 15 deletions(-) diff --git a/openmetadata-docs/content/v1.3.x/deployment/bare-metal/index.md b/openmetadata-docs/content/v1.3.x/deployment/bare-metal/index.md index eb879a4aaa0..e812dd46a5d 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/bare-metal/index.md +++ b/openmetadata-docs/content/v1.3.x/deployment/bare-metal/index.md @@ -9,7 +9,7 @@ Requirements This guide assumes you have access to a command-line environment or or Mac OS X or PowerShell on Microsoft Windows. This guide also assumes that your command-line environment has access to the tar utility. Please review additional requirements listed in the subsections below. -## Java (version 17.0.0 or greater) +## Java (version 17.0.0 or higher) OpenMetadata is built using Java, DropWizard, and Jetty. @@ -22,7 +22,7 @@ java --version To install Java or upgrade to Java 17 or greater, see the instructions for your operating system at [How do I install Java?](https://java.com/en/download/help/download_options.html#mac). -## MySQL (version 8.0.0 or greater) +## MySQL (version 8.0.0 or higher) To install MySQL see the instructions for your operating system (OS) at [Installing and Upgrading MySQL](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/installing.html) or visit one of the following OS-specific guides. @@ -39,7 +39,7 @@ You can refer a sample script [here](https://github.com/open-metadata/OpenMetada {%/note%} -## Postgres (version between 12.0 or greater) +## Postgres (version 12.0 or higher) To install Postgres see the instructions for your operating system (OS) at [Postgres Download](https://www.postgresql.org/download/) {%note%} @@ -60,7 +60,7 @@ Please follow the instructions here to [install ElasticSearch](https://www.elast If you are using AWS OpenSearch Service, OpenMetadata Supports AWS OpenSearch Service engine version up to 2.7. For more information on AWS OpenSearch Service, please visit the official docs [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html). -## Airflow (version 2.0.0 or greater) or other workflow schedulers +## Airflow (version 2.0.0 or higher) or other workflow schedulers OpenMetadata performs metadata ingestion using the Ingestion Framework. Learn more about how to deploy and manage the ingestion workflows [here](/deployment/ingestion). @@ -168,9 +168,9 @@ If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](ht We support -- Amazon RDS (MySQL) engine version 8 or greater +- Amazon RDS (MySQL) engine version 8 or higher - Amazon OpenSearch (ElasticSearch) engine version up to 8.10.2 or Amazon OpenSearch engine version up to 2.7 -- Amazon RDS (PostgreSQL) engine version between 12 and 14.6 +- Amazon RDS (PostgreSQL) engine version between 12 or higher For Production Systems, we recommend Amazon RDS to be in Multiple Availability Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availability Zones with minimum 3 Master Nodes. diff --git a/openmetadata-docs/content/v1.3.x/deployment/docker/index.md b/openmetadata-docs/content/v1.3.x/deployment/docker/index.md index 55e9f13cf83..b94756e777f 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/docker/index.md +++ b/openmetadata-docs/content/v1.3.x/deployment/docker/index.md @@ -20,7 +20,7 @@ For Production Deployment using Docker, we recommend bringing your own Databases {% partial file="/v1.3/deployment/configure-external-orchestrator-for-ingestion-service.md" /%} -### Docker (version 20.10.0 or greater) +### Docker (version 20.10.0 or higher) [Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers. @@ -242,9 +242,9 @@ If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](ht We support -- Amazon RDS (MySQL) engine version 8 or greater +- Amazon RDS (MySQL) engine version 8 or higher - Amazon OpenSearch (ElasticSearch) engine version up to 8.10.2 or Amazon OpenSearch engine version up to 2.7 -- Amazon RDS (PostgreSQL) engine version 12 or greater +- Amazon RDS (PostgreSQL) engine version 12 or higher Note:- When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, diff --git a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/aks.md b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/aks.md index 407ad235c09..74e640d5225 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/aks.md +++ b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/aks.md @@ -5,6 +5,62 @@ slug: /deployment/kubernetes/aks # Openmetadata Deployment on Azure Kubernetes Service Cluster Openmetadata can be deployed on Azure Kubernetes Service. It however requires certain cloud specific configurations with regards to setting up storage accounts for Airflow which is one of its dependencies. +## Prerequisites + +### Azure Services for Database and Search Engine as Elastic Cloud + +It is recommended to use [Azure SQL](https://azure.microsoft.com/en-in/products/azure-sql/database) and [Elastic Cloud on Azure](https://www.elastic.co/partners/microsoft-azure) for Production Deployments. + +We support + +- Azure SQL (MySQL) engine version 8 or higher +- Azure SQL (PostgreSQL) engine version 12 or higher +- Elastic Cloud (ElasticSearch version 8.10.2) + +Once you have the Azure SQL and Elastic Cloud on Azure configured, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. + +```yaml +# openmetadata-values.prod.yaml +... +openmetadata: + config: + elasticsearch: + host: + searchType: elasticsearch + port: 443 + scheme: https + connectionTimeoutSecs: 5 + socketTimeoutSecs: 60 + keepAliveTimeoutSecs: 600 + batchSize: 10 + auth: + enabled: true + username: + password: + secretRef: elasticsearch-secrets + secretKey: openmetadata-elasticsearch-password + database: + host: + port: 3306 + driverClass: com.mysql.cj.jdbc.Driver + dbScheme: mysql + dbUseSSL: true + databaseName: + auth: + username: + password: + secretRef: mysql-secrets + secretKey: openmetadata-mysql-password + ... +``` + +We recommend - +- Azure SQL to be Multi Zone Available and Production Workload Environment +- Elastic Cloud Environment with multiple zones and minimum 2 nodes + +Make sure to create database and elastic cloud credentials as Kubernetes Secrets mentioned [here](/quick-start/local-kubernetes-deployment#2.-create-kubernetes-secrets-required-for-helm-charts). + +Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](/deployment/kubernetes/faqs#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts). ### Step 1 - Create a AKS cluster If you are deploying on a new cluster set the `EnableAzureDiskFileCSIDriver=true` to enable container storage interface storage drivers. diff --git a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/eks.md b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/eks.md index 532a3665982..eed9b0353aa 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/eks.md +++ b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/eks.md @@ -23,9 +23,9 @@ It is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html We support -- Amazon RDS (MySQL) engine version 8 or greater -- Amazon RDS (PostgreSQL) engine version between 12 or greater -- Amazon OpenSearch engine version 2.7 +- Amazon RDS (MySQL) engine version 8 or higher +- Amazon RDS (PostgreSQL) engine version 12 or higher +- Amazon OpenSearch engine version 2.X (upto 2.7) {%note noteType="Tip"%} When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, as you can see in the ElasticSearch configuration example below. diff --git a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/gke.md b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/gke.md index c0859678f4b..08fa279a21f 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/gke.md +++ b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/gke.md @@ -28,9 +28,9 @@ All the code snippets in this section assume the `default` namespace for kuberne It is recommended to use GCP [Cloud SQL](https://cloud.google.com/sql/) services for Database and [Elastic Cloud GCP](https://www.elastic.co/partners/google-cloud) for Search Engine for Production. We support - -- Cloud SQL (MySQL) engine version 8 or greater -- Cloud SQL (postgreSQL) engine version 12 or greater -- ElasticSearch Engine version 8.10 +- Cloud SQL (MySQL) engine version 8 or higher +- Cloud SQL (postgreSQL) engine version 12 or higher +- ElasticSearch Engine version 8.X (upto 8.10.X) We recommend - - CloudSQL to be Multi Zone Available diff --git a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/on-prem.md b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/on-prem.md index 402ac84aa05..2939280c83f 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/on-prem.md +++ b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/on-prem.md @@ -16,6 +16,55 @@ This guide presumes you have an on premises Kubernetes cluster setup, and you ar ## Prerequisites +### External Database and Search Engine as ElasticSearch / OpenSearch + +We support + +- MySQL engine version 8 or higher +- PostgreSQL engine version 12 or higher +- ElasticSearch version 8.X (upto 8.10.2) or OpenSearch Version 2.X (upto 2.7) + +Once you have the External Database and Search Engine configured, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. + +```yaml +# openmetadata-values.prod.yaml +... +openmetadata: + config: + elasticsearch: + host: + searchType: elasticsearch # or `opensearch` if Search Engine is OpenSearch + port: 443 + scheme: https + connectionTimeoutSecs: 5 + socketTimeoutSecs: 60 + keepAliveTimeoutSecs: 600 + batchSize: 10 + auth: + enabled: true + username: + password: + secretRef: elasticsearch-secrets + secretKey: openmetadata-elasticsearch-password + database: + host: + port: 3306 + driverClass: com.mysql.cj.jdbc.Driver + dbScheme: mysql + dbUseSSL: true + databaseName: + auth: + username: + password: + secretRef: mysql-secrets + secretKey: openmetadata-mysql-password + ... +``` + +Make sure to create database and search engine credentials as Kubernetes Secrets mentioned [here](/quick-start/local-kubernetes-deployment#2.-create-kubernetes-secrets-required-for-helm-charts). + +Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](/deployment/kubernetes/faqs#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts). + ### Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes).