mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2026-01-06 04:26:57 +00:00
docs: 1.2.1 Breaking Changes and miscellaneous fixes (#13955)
* Remove Deprecated docs of docker volume * fix(docs): Update Kubernetes Docs from 1.1.X * docs: Add 1.2.1 breaking changes * docs: Update Helm Values * 1.2.1 docs --------- Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
This commit is contained in:
Akash Jain
GitHub
parent
2ae0333551
commit
0c31dc4641
@ -109,6 +109,43 @@ After the migration is finished, you can revert this changes.
|
||||
|
||||
# Breaking Changes
|
||||
|
||||
## 1.2.1
|
||||
|
||||
### Application Logo and Login Configuration Migrated to UI
|
||||
|
||||
The following configuration block has been removed from `openmetadata.yaml`:
|
||||
|
||||
```yaml
|
||||
applicationConfig:
|
||||
logoConfig:
|
||||
customLogoUrlPath: ${OM_CUSTOM_LOGO_URL_PATH:-""}
|
||||
customMonogramUrlPath: ${OM_CUSTOM_MONOGRAM_URL_PATH:-""}
|
||||
loginConfig:
|
||||
maxLoginFailAttempts: ${OM_MAX_FAILED_LOGIN_ATTEMPTS:-3}
|
||||
accessBlockTime: ${OM_LOGIN_ACCESS_BLOCK_TIME:-600}
|
||||
jwtTokenExpiryTime: ${OM_JWT_EXPIRY_TIME:-3600}
|
||||
```
|
||||
|
||||
This change removes the traditional way of providing **Custom URL** logos configurations as part of OpenMetadata Configurations
|
||||
file and migrate this to be driven and configured right from UI from `Settings` > `OpenMetadata` > `Custom Logo`.
|
||||
|
||||
The same applies to the **Login Configuration**, which can now be configured under `Settings` > `OpenMetadata` > `Login Configuration`.
|
||||
|
||||
Note that these environment variables will now have no effect. If you are deploying on Bare Metal, make sure to use the latest `openmetadata.yaml` file.
|
||||
|
||||
|
||||
### OpenMetadata Helm Chart Dependencies Migrate from ElasticSearch to OpenSearch Charts
|
||||
|
||||
As part of `1.2.1`, we migrated the base dependencies for OpenMetadata Helm Chart to use OpenSearch version `2.7` instead of ElasticSearch `8.X`. This is a reactive change done as community driven [ElasticSearch Helm Chart](https://github.com/elastic/helm-charts) project has been deprecated in the favor of Elastic Stack Operator which cannot be added as an helm chart dependency.
|
||||
|
||||
For new users, this is an unnoticeable change who will be installing the OpenMetadata dependencies using quickstart guides.
|
||||
|
||||
For existing users, who have their proof-of-concept environments using the OpenMetadata Dependencies and are looking to upgrade to newer helm release -
|
||||
- The default OpenMetadata helm values for `openmetadata.config.elasticsearch.*` has been updated to connect to OpenSearch from OpenMetadata Dependencies Helm Chart. Please refer to the [helm values](/deployment/kubernetes/helm-values) and update your custom installation accordingly.
|
||||
- Post upgrade, you will need to follow the steps here to [rebuild and reindex your search indexing](/deployment/upgrade#reindex).
|
||||
|
||||
## 1.2.0
|
||||
|
||||
### Database connection SSL Configuration
|
||||
|
||||
With 1.2.X, the environment variable `DB_USE_SSL` is deprecated in favour of `DB_PARAMS`.
|
||||
|
||||
@ -1,137 +0,0 @@
|
||||
---
|
||||
title: Docker Volumes
|
||||
slug: /deployment/docker/volumes
|
||||
---
|
||||
## Docker Volumes
|
||||
Advance section
|
||||
|
||||
|
||||
Volumes provide the ability to connect specific filesystem paths of the container back to the host machine. If a directory or a file in the container is mounted, changes in that directory or file can also be seen on the host machine.we are going to use a mapping of a directory present on the host macine with the container path.
|
||||
|
||||
{%note%}
|
||||
|
||||
To make changes to the `docker-compose.yaml` file you will need to download it from the release page [here](https://github.com/open-metadata/OpenMetadata/releases/latest). The latest version is at the top of the page
|
||||
{%/note%}
|
||||
|
||||
## Volumes for MYSQL container:
|
||||
Following are the changes we have to do while mounting the directory for mysql in OpenMetadata.
|
||||
- Update or add the volume in the docker-compose.yml file
|
||||
Open the file `docker-compose.yml` downloaded from the Release page [Link](https://github.com/open-metadata/OpenMetadata/releases/download/0.13.0-release/docker-compose.yml) .
|
||||
|
||||
```commandline
|
||||
version: "3.9"
|
||||
services:
|
||||
mysql:
|
||||
...
|
||||
volumes:
|
||||
- ./docker-volume/db:/var/lib/mysql
|
||||
...
|
||||
```
|
||||
## Volumes for PostgreSQL container:
|
||||
Following are the changes we have to do while mounting the directory for postgresql in OpenMetadata.
|
||||
- Update or add the volume in the docker-compose.yml file.
|
||||
Open the file `docker-compose.yml` downloaded from the Release page [Link](https://github.com/open-metadata/OpenMetadata/releases/download/0.13.0-release/docker-compose.yml) .
|
||||
|
||||
```commandline
|
||||
version: "3.9"
|
||||
services:
|
||||
postgresql:
|
||||
...
|
||||
volumes:
|
||||
- ./docker-volume/db:/var/lib/postgresql/data
|
||||
...
|
||||
```
|
||||
## Volumes for Elasticsearch container:
|
||||
Following are the changes we have to do while mounting the directory for Elasticsearch in OpenMetadata.
|
||||
- Update or add the volume in the docker-compose.yml file.
|
||||
Open the file `docker-compose.yml` downloaded from the Release page [Link](https://github.com/open-metadata/OpenMetadata/releases/download/0.13.0-release/docker-compose.yml) .
|
||||
|
||||
```commandline
|
||||
version: "3.9"
|
||||
services:
|
||||
elasticsearch:
|
||||
...
|
||||
volumes:
|
||||
- ./docker-volume/es-data:/usr/share/elasticsearch/data
|
||||
...
|
||||
```
|
||||
## Volumes for ingestion container
|
||||
Following are the changes we have to do while mounting the directory for ingestion in OpenMetadata. Here we will maintaing different directory for dag_generated_configs, dags and secrets.
|
||||
- Remove the below section from the docker-compose.yml file.
|
||||
Open the file `docker-compose.yml` downloaded from the Release page [Link](https://github.com/open-metadata/OpenMetadata/releases/download/0.13.0-release/docker-compose.yml) .
|
||||
|
||||
```commandline
|
||||
volumes:
|
||||
ingestion-volume-dag-airflow:
|
||||
ingestion-volume-dags:
|
||||
ingestion-volume-tmp:
|
||||
```
|
||||
- Update or add the volume in the docker-compose.yml file.
|
||||
Open the file `docker-compose.yml` downloaded from the Release page [Link](https://github.com/open-metadata/OpenMetadata/releases/download/0.13.0-release/docker-compose.yml) .
|
||||
|
||||
```commandline
|
||||
version: "3.9"
|
||||
services:
|
||||
ingestion:
|
||||
...
|
||||
volumes:
|
||||
- ./docker-volume/dag_config:/opt/airflow/dag_generated_configs
|
||||
- ./docker-volume/dags:/opt/airflow/dags
|
||||
- ./docker-volume/secrets:/tmp
|
||||
...
|
||||
```
|
||||
|
||||
Once these changes are done in the docker-compose.yml file It should look simlarly in the below format
|
||||
|
||||
```commandline
|
||||
version: "3.9"
|
||||
services:
|
||||
mysql:
|
||||
container_name: openmetadata_mysql
|
||||
image: openmetadata/db:1.2.0
|
||||
restart: always
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: password
|
||||
expose:
|
||||
- 3306
|
||||
volumes:
|
||||
- ./docker-volume/db:/var/lib/mysql
|
||||
networks:
|
||||
app_net:
|
||||
ipv4_address: 172.16.240.10
|
||||
ingestion:
|
||||
container_name: openmetadata_ingestion
|
||||
image: openmetadata/ingestion:1.2.0
|
||||
depends_on:
|
||||
- mysql
|
||||
expose:
|
||||
- 8080
|
||||
ports:
|
||||
- 8080:8080
|
||||
networks:
|
||||
- app_net
|
||||
extra_hosts:
|
||||
- "localhost:172.16.240.10"
|
||||
- "localhost:172.16.240.11"
|
||||
- "localhost:172.16.240.13"
|
||||
volumes:
|
||||
- ./docker-volume/dag_config:/opt/airflow/dag_generated_configs
|
||||
- ./docker-volume/dags:/opt/airflow/dags
|
||||
- ./docker-volume/secrets:/tmp
|
||||
```
|
||||
|
||||
Once these changes are done, restart the container via:
|
||||
|
||||
```commandline
|
||||
docker compose down && docker compose up -d
|
||||
```
|
||||
|
||||
Since our Ingestion Docker Image is based on Official Airflow Docker image as base, it is recommended to provide right permissions to host directories for airflow user running inside the container as per the instructions mentioned [here](https://airflow.apache.org/docs/apache-airflow/2.3.3/start/docker.html?highlight=docker#setting-the-right-airflow-user).
|
||||
|
||||
{%note%}
|
||||
If you are starting the OpenMetadata Docker containers using below command:
|
||||
```command line
|
||||
metadata docker --start
|
||||
```
|
||||
Then the docker volume directory will be created under the folder where the docker-compose file will be present.
|
||||
{%/note%}
|
||||
@ -8,7 +8,7 @@ slug: /deployment/kubernetes/eks
|
||||
OpenMetadata supports the Installation and Running of Application on Elastic Kubernetes Services (EKS) through Helm Charts.
|
||||
However, there are some additional configurations which needs to be done as prerequisites for the same.
|
||||
|
||||
{%note%}
|
||||
{%note noteType="Warning"%}
|
||||
|
||||
All the code snippets in this section assume the `default` namespace for kubernetes.
|
||||
This guide presumes you have AWS EKS Cluster already available.
|
||||
@ -17,6 +17,65 @@ This guide presumes you have AWS EKS Cluster already available.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### AWS Services for Database as RDS and Search Engine as ElasticSearch
|
||||
|
||||
It is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway) for Production Deployments.
|
||||
|
||||
We support
|
||||
|
||||
- Amazon RDS (MySQL) engine version 8 or greater
|
||||
- Amazon RDS (PostgreSQL) engine version between 12 or greater
|
||||
- Amazon OpenSearch engine version 2.7
|
||||
|
||||
{%note noteType="Tip"%}
|
||||
When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, as you can see in the ElasticSearch configuration example below.
|
||||
{%/note%}
|
||||
|
||||
We recommend
|
||||
- Amazon RDS to be in Multiple Availability Zones.
|
||||
- Amazon OpenSearch (or ElasticSearch) Service with Multiple Availability Zones with minimum 2 Nodes.
|
||||
|
||||
Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch.
|
||||
|
||||
```yaml
|
||||
# openmetadata-values.prod.yaml
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
elasticsearch:
|
||||
host: <AMAZON_OPENSEARCH_SERVICE_ENDPOINT_WITHOUT_HTTPS>
|
||||
searchType: opensearch
|
||||
port: 443
|
||||
scheme: https
|
||||
connectionTimeoutSecs: 5
|
||||
socketTimeoutSecs: 60
|
||||
keepAliveTimeoutSecs: 600
|
||||
batchSize: 10
|
||||
auth:
|
||||
enabled: true
|
||||
username: <AMAZON_OPENSEARCH_USERNAME>
|
||||
password:
|
||||
secretRef: elasticsearch-secrets
|
||||
secretKey: openmetadata-elasticsearch-password
|
||||
database:
|
||||
host: <AMAZON_RDS_ENDPOINT>
|
||||
port: 3306
|
||||
driverClass: com.mysql.cj.jdbc.Driver
|
||||
dbScheme: mysql
|
||||
dbUseSSL: true
|
||||
databaseName: <RDS_DATABASE_NAME>
|
||||
auth:
|
||||
username: <RDS_DATABASE_USERNAME>
|
||||
password:
|
||||
secretRef: mysql-secrets
|
||||
secretKey: openmetadata-mysql-password
|
||||
...
|
||||
```
|
||||
|
||||
Make sure to create RDS and OpenSearch credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/deployment/kubernetes#quickstart).
|
||||
|
||||
Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](/deployment/kubernetes/faqs#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts).
|
||||
|
||||
### Create Elastic File System in AWS
|
||||
|
||||
You can follow official AWS Guides [here](https://docs.aws.amazon.com/efs/latest/ug/gs-step-two-create-efs-resources.html) to provision EFS File System in the same VPC which is associated with your EKS Cluster.
|
||||
|
||||
214
openmetadata-docs/content/v1.2.x/deployment/kubernetes/faqs.md
Normal file
214
openmetadata-docs/content/v1.2.x/deployment/kubernetes/faqs.md
Normal file
@ -0,0 +1,214 @@
|
||||
---
|
||||
title: FAQs
|
||||
slug: /deployment/kubernetes/faqs
|
||||
---
|
||||
|
||||
# Java Memory Heap Issue
|
||||
|
||||
If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue -
|
||||
|
||||
```
|
||||
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender"
|
||||
Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space
|
||||
```
|
||||
|
||||
This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable
|
||||
|
||||
```yaml
|
||||
extraEnvs:
|
||||
- name: OPENMETADATA_HEAP_OPTS
|
||||
value: "-Xmx2G -Xms2G"
|
||||
```
|
||||
|
||||
The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool.
|
||||
|
||||
Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values <values.yml> --namespace <namespaceName>`. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes.
|
||||
|
||||
# PostgreSQL Issue permission denied to create extension "pgcrypto"
|
||||
|
||||
{% partial file="/v1.2/deployment/postgresql-issue-permission-denied-extension-pgcrypto.md" /%}
|
||||
|
||||
# How to extend and use custom docker images with OpenMetadata Helm Charts ?
|
||||
|
||||
## Extending OpenMetadata Server Docker Image
|
||||
|
||||
### 1. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/server`
|
||||
|
||||
OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata).
|
||||
A typical scenario will be to install organization certificates for connecting with inhouse systems.
|
||||
|
||||
For Example -
|
||||
|
||||
```
|
||||
FROM docker.getcollate.io/openmetadata/server:x.y.z
|
||||
WORKDIR /home/
|
||||
COPY <my-organization-certs> .
|
||||
RUN update-ca-certificates
|
||||
```
|
||||
where `docker.getcollate.io/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/server:1.2.0`.
|
||||
This image needs to be built and published to the container registry of your choice.
|
||||
|
||||
### 2. Update your openmetadata helm values yaml
|
||||
|
||||
The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents -
|
||||
|
||||
```yaml
|
||||
...
|
||||
image:
|
||||
repository: <your repository>
|
||||
# Overrides the image tag whose default is the chart appVersion.
|
||||
tag: <your tag>
|
||||
...
|
||||
```
|
||||
|
||||
### 3. Install / Upgrade your helm release
|
||||
|
||||
Upgrade/Install your openmetadata helm charts with the below single command:
|
||||
|
||||
```bash
|
||||
helm upgrade --install openmetadata open-metadata/openmetadata--values values.yaml
|
||||
```
|
||||
|
||||
## Extending OpenMetadata Ingestion Docker Image
|
||||
|
||||
One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors.
|
||||
You can find a complete working example of this [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-connector). After
|
||||
you have your code ready, the steps would be the following:
|
||||
|
||||
### 1. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/ingestion`:
|
||||
|
||||
For example -
|
||||
|
||||
```
|
||||
FROM docker.getcollate.io/openmetadata/ingestion:x.y.z
|
||||
|
||||
USER airflow
|
||||
# Let's use the home directory of airflow user
|
||||
WORKDIR /home/airflow
|
||||
|
||||
# Install our custom connector
|
||||
COPY <your_package> <your_package>
|
||||
COPY setup.py .
|
||||
RUN pip install --no-deps .
|
||||
```
|
||||
|
||||
where `docker.getcollate.io/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/ingestion:1.2.0`.
|
||||
This image needs to be built and published to the container registry of your choice.
|
||||
|
||||
### 2. Update the airflow in openmetadata dependencies values YAML
|
||||
|
||||
The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use
|
||||
our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the
|
||||
following contents:
|
||||
|
||||
```yaml
|
||||
airflow:
|
||||
airflow:
|
||||
image:
|
||||
repository: <your repository> # by default, openmetadata/ingestion
|
||||
tag: <your tag> # by default, the version you are deploying, e.g., 1.1.0
|
||||
pullPolicy: "IfNotPresent"
|
||||
```
|
||||
|
||||
### 3. Install / Upgrade helm release
|
||||
|
||||
Upgrade/Install your openmetadata-dependencies helm charts with the below single command:
|
||||
|
||||
```bash
|
||||
helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values.deps.yaml
|
||||
```
|
||||
|
||||
# How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ?
|
||||
|
||||
If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI -
|
||||
|
||||
```commandline
|
||||
helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --set mysql.enabled=false --set elasticsearch.enabled=false
|
||||
```
|
||||
|
||||
Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch .
|
||||
|
||||
```yaml
|
||||
mysql:
|
||||
enabled: false
|
||||
...
|
||||
elasticsearch:
|
||||
enabled: false
|
||||
...
|
||||
...
|
||||
```
|
||||
|
||||
# How to configure external database like PostgreSQL with OpenMetadata Helm Charts ?
|
||||
|
||||
OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same.
|
||||
|
||||
## Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL
|
||||
|
||||
We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations.
|
||||
|
||||
You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md).
|
||||
|
||||
With OpenMetadata Dependencies Helm Charts, your helm values would look something like below -
|
||||
|
||||
```yaml
|
||||
...
|
||||
airflow:
|
||||
externalDatabase:
|
||||
type: postgresql
|
||||
host: <postgresql_endpoint>
|
||||
port: 5432
|
||||
database: <airflow_database_name>
|
||||
user: <airflow_database_login_user>
|
||||
passwordSecret: airflow-postgresql-secrets
|
||||
passwordSecretKey: airflow-postgresql-password
|
||||
...
|
||||
```
|
||||
|
||||
For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=<password>`.
|
||||
|
||||
## Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL
|
||||
|
||||
Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL.
|
||||
|
||||
With OpenMetadata Helm Charts, your helm values would look something like below -
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
...
|
||||
database:
|
||||
host: <postgresql_endpoint>
|
||||
port: 5432
|
||||
driverClass: org.postgresql.Driver
|
||||
dbScheme: postgresql
|
||||
dbUseSSL: true
|
||||
databaseName: <openmetadata_database_name>
|
||||
auth:
|
||||
username: <database_login_user>
|
||||
password:
|
||||
secretRef: openmetadata-postgresql-secrets
|
||||
secretKey: openmetadata-postgresql-password
|
||||
```
|
||||
For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=<password>`.
|
||||
|
||||
Once you make the above changes to your helm values, run the below command to install/upgrade helm charts -
|
||||
|
||||
```commandline
|
||||
helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values <<path-to-values-file>> --namespace <kubernetes_namespace>
|
||||
helm upgrade --install openmetadata open-metadata/openmetadata --values <<path-to-values-file>> --namespace <kubernetes_namespace>
|
||||
```
|
||||
|
||||
# Getting an error when install OpenMetadata Dependencies Helm Charts on EKS with EFS
|
||||
|
||||
If you are facing the below issue -
|
||||
|
||||
```
|
||||
MountVolume.SetUp failed for volume "openmetadata-dependencies-dags-pv" : rpc error: code = Internal desc = Could not mount "fs-012345abcdef:/airflow-dags" at "/var/lib/kubelet/pods/xyzabc-123-0062-44c3-b0e9-fa193c19f41c/volumes/kubernetes.io~csi/openmetadata-dependencies-dags-pv/mount": mount failed: exit status 1 Mounting command: mount Mounting arguments: -t efs -o tls fs-012345abcdef:/airflow-dags /var/lib/kubelet/pods/xyzabc-123-0062-44c3-b0e9-fa193c19f41c/volumes/kubernetes.io~csi/openmetadata-dependencies-dags-pv/mount Output: Failed to locate an available port in the range [20049, 20449], try specifying a different port range in /etc/amazon/efs/efs-utils.conf
|
||||
```
|
||||
|
||||
This error is typically related to EKS Cluster not able to reach to EFS File systems. You can check the security groups associated between the connectivity EFS and EKS. [Here is an article](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/efs-create-filesystem.md) which further describes the steps required to create Security Group Rules for EKS to use EFS over `port 2049`.
|
||||
|
||||
It can also happen if the mount targets are already available for EKS Nodes but the Nodes do not pick that up. In such cases, you can do an [AWS AutoScaling Group instance refresh](https://docs.aws.amazon.com/autoscaling/ec2/userguide/start-instance-refresh.html) in order for EKS nodes to get the available mount targets.
|
||||
@ -1,23 +0,0 @@
|
||||
---
|
||||
title: GKE Setup Troubleshooting
|
||||
slug: /deployment/kubernetes/gke-troubleshooting
|
||||
---
|
||||
|
||||
# GKE Setup Troubleshooting
|
||||
|
||||
If you came across `invalid access type while creating the pvc`, and the permission pod is stuck in "pending" state.
|
||||
|
||||
The above error might have occurred due to the pvc volumes not setup or pvc volumes are not mounted properly.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/deployment/troubleshoot/dag-log.png"
|
||||
alt="dag-log" /%}
|
||||
{% image
|
||||
src="/images/v1.2/deployment/troubleshoot/permission-pod-events.png"
|
||||
alt="permission-pod-events"
|
||||
caption="Permission pod events" /%}
|
||||
|
||||
Please validate:
|
||||
- all the prerequisites mentioned in this [section](/deployment/kubernetes/gke)
|
||||
- the configuration of `dags_pv_pvc.yml` file
|
||||
- `storageClassName` field in YAML file
|
||||
@ -15,7 +15,7 @@ The reason being that ElasticSearch Pods require Elevated permissions to run ini
|
||||
|
||||
{%/note%}
|
||||
|
||||
{%note%}
|
||||
{%note noteType="Warning"%}
|
||||
|
||||
All the code snippets in this section assume the `default` namespace for kubernetes.
|
||||
|
||||
@ -23,6 +23,90 @@ All the code snippets in this section assume the `default` namespace for kuberne
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Cloud Database with CloudSQL and ElasticCloud for GCP as Search Engine
|
||||
|
||||
It is recommended to use GCP [Cloud SQL](https://cloud.google.com/sql/) services for Database and [Elastic Cloud GCP](https://www.elastic.co/partners/google-cloud) for Search Engine for Production.
|
||||
|
||||
We support -
|
||||
- Cloud SQL (MySQL) engine version 8 or greater
|
||||
- Cloud SQL (postgreSQL) engine version 12 or greater
|
||||
- ElasticSearch Engine version 8.10
|
||||
|
||||
We recommend -
|
||||
- CloudSQL to be Multi Zone Available
|
||||
- Elastic Cloud Environment with multiple zones and minimum 2 nodes
|
||||
|
||||
Once you have the Database and Search Engine configured and available, update the helm values below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch.
|
||||
|
||||
```yaml
|
||||
# openmetadata-values.prod.yaml
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
elasticsearch:
|
||||
host: <ELASTIC_CLOUD_SERVICE_ENDPOINT_WITHOUT_HTTPS>
|
||||
searchType: elasticsearch
|
||||
port: 443
|
||||
scheme: https
|
||||
connectionTimeoutSecs: 5
|
||||
socketTimeoutSecs: 60
|
||||
keepAliveTimeoutSecs: 600
|
||||
batchSize: 10
|
||||
auth:
|
||||
enabled: true
|
||||
username: <ELASTIC_CLOUD_USERNAME>
|
||||
password:
|
||||
secretRef: elasticsearch-secrets
|
||||
secretKey: openmetadata-elasticsearch-password
|
||||
database:
|
||||
host: <GCP_CLOUD_SQL_ENDPOINT_IP>
|
||||
port: 3306
|
||||
driverClass: com.mysql.cj.jdbc.Driver
|
||||
dbScheme: mysql
|
||||
dbUseSSL: true
|
||||
databaseName: <GCP_CLOUD_SQL_DATABASE_NAME>
|
||||
auth:
|
||||
username: <GCP_CLOUD_SQL_DATABASE_USERNAME>
|
||||
password:
|
||||
secretRef: mysql-secrets
|
||||
secretKey: openmetadata-mysql-password
|
||||
...
|
||||
```
|
||||
|
||||
{%note noteType="Tip"%}
|
||||
|
||||
For Database as PostgreSQL, the use the below config for database values -
|
||||
|
||||
```yaml
|
||||
# openmetadata-values.prod.yaml
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
...
|
||||
database:
|
||||
host: <GCP_CLOUD_SQL_ENDPOINT_IP>
|
||||
port: 5432
|
||||
driverClass: org.postgresql.Driver
|
||||
dbScheme: postgresql
|
||||
dbUseSSL: true
|
||||
databaseName: <GCP_CLOUD_SQL_DATABASE_NAME>
|
||||
auth:
|
||||
username: <GCP_CLOUD_SQL_DATABASE_USERNAME>
|
||||
password:
|
||||
secretRef: sql-secrets
|
||||
secretKey: openmetadata-sql-password
|
||||
```
|
||||
|
||||
{%/note%}
|
||||
|
||||
{%note noteType="Tip"%}
|
||||
|
||||
Make sure to create CloudSQL and ElasticSearch credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/deployment/kubernetes#quickstart).
|
||||
|
||||
Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](/deployment/kubernetes/faqs#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts).
|
||||
|
||||
{%/note%}
|
||||
|
||||
### Persistent Volumes with ReadWriteMany Access Modes
|
||||
|
||||
OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes).
|
||||
@ -288,7 +372,7 @@ airflow:
|
||||
```
|
||||
{%note%}
|
||||
|
||||
For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3).
|
||||
For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0).
|
||||
|
||||
When deploying openmeteadata dependencies helm chart, use the below command -
|
||||
|
||||
@ -308,7 +392,26 @@ helm install openmetadata-dependencies open-metadata/openmetadata-dependencies -
|
||||
Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart -
|
||||
|
||||
```commandline
|
||||
helm install openmetadata open-metadata/openmetadata
|
||||
helm install openmetadata open-metadata/openmetadata --values <path-to-values-file>
|
||||
```
|
||||
Again, this uses the values defined [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml).
|
||||
Use the `--values` flag to point to your own YAML configuration if needed.
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
## Pods are stuck in Pending State due to Persistent Volume Creation Failure
|
||||
|
||||
If you came across `invalid access type while creating the pvc`, and the permission pod is stuck in "pending" state.
|
||||
|
||||
The above error might have occurred due to the pvc volumes not setup or pvc volumes are not mounted properly.
|
||||
|
||||
{% image
|
||||
src="/images/v1.2/deployment/troubleshoot/dag-log.png"
|
||||
alt="dag-log" /%}
|
||||
{% image
|
||||
src="/images/v1.2/deployment/troubleshoot/permission-pod-events.png"
|
||||
alt="permission-pod-events"
|
||||
caption="Permission pod events" /%}
|
||||
|
||||
Please validate:
|
||||
- all the prerequisites mentioned in this [section](#prerequisites)
|
||||
- the configuration of `dags_pv_pvc.yml` file
|
||||
- `storageClassName` field in YAML file
|
||||
|
||||
@ -7,324 +7,54 @@ slug: /deployment/kubernetes
|
||||
|
||||
OpenMetadata supports the Installation and Running of Application on kubernetes through Helm Charts.
|
||||
|
||||
## Introduction
|
||||
|
||||
[openmetadata-helm-charts](https://github.com/open-metadata/openmetadata-helm-charts) houses Kubernetes Helm charts
|
||||
for deploying OpenMetadata and its dependencies (Elasticsearch, MySQL and Airflow) on a Kubernetes cluster.
|
||||
|
||||
## Kubernetes Deployment Architecture
|
||||
{% image src="/images/v1.2/deployment/kubernetes/k8_architecture.png" alt="Kubernetes Deployment Architecture" /%}
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A [Kubernetes](https://kubernetes.io/) cluster on any cloud
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/) to manage Kubernetes resources
|
||||
- [Helm](https://helm.sh/) to deploy resources based on Helm charts from the OpenMetadata repository
|
||||
|
||||
{%note%}
|
||||
|
||||
OpenMetadata ONLY supports Helm 3.
|
||||
|
||||
This guide assumes your helm chart release names are `openmetadata` and `openmetadata-dependencies` and the kubernetes namespace
|
||||
used is `default`.
|
||||
|
||||
{%/note%}
|
||||
|
||||
{%note%}
|
||||
|
||||
If you are using minikube to start a local kubernetes instance on Mac OS with M1 chipset, use the following command to start the cluster required for OpenMetadata Helm Charts to install locally (with docker desktop running as container runtime engine).
|
||||
|
||||
`minikube start --cpus=4 --memory=8192 --cni=bridge --driver=docker`
|
||||
|
||||
{%/note%}
|
||||
|
||||
## Quickstart
|
||||
|
||||
Assuming Kubernetes setup is done and your Kubernetes context points to a correct Kubernetes cluster, first we create
|
||||
kubernetes secrets that contains MySQL and Airflow passwords as secrets.
|
||||
|
||||
```commandline
|
||||
kubectl create secret generic mysql-secrets --from-literal=openmetadata-mysql-password=openmetadata_password
|
||||
kubectl create secret generic airflow-secrets --from-literal=openmetadata-airflow-password=admin
|
||||
```
|
||||
|
||||
The above commands sets the passwords as an example. Change to any password of your choice.
|
||||
|
||||
Next, install the OpenMetadata dependencies by adding the OpenMetadata Helm repository with the following command.
|
||||
|
||||
```commandline
|
||||
helm repo add open-metadata https://helm.open-metadata.org/
|
||||
```
|
||||
|
||||
Run the command `helm repo list` to ensure the OpenMetadata repository was added.
|
||||
|
||||
```commandline
|
||||
NAME URL
|
||||
open-metadata https://helm.open-metadata.org/
|
||||
```
|
||||
|
||||
Assuming `kubectl` context points to the correct kubernetes cluster, first create the kubernetes secrets that contain airflow
|
||||
mysql password as secrets.
|
||||
|
||||
```commandline
|
||||
kubectl create secret generic airflow-mysql-secrets --from-literal=airflow-mysql-password=airflow_pass
|
||||
```
|
||||
|
||||
Deploy the dependencies by running the following command:
|
||||
|
||||
```commandline
|
||||
helm install openmetadata-dependencies open-metadata/openmetadata-dependencies
|
||||
```
|
||||
|
||||
{%note%}
|
||||
|
||||
The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml).
|
||||
You can modify any configuration and deploy by passing your own `values.yaml`
|
||||
|
||||
```commandline
|
||||
helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values <path-to-values-file>
|
||||
```
|
||||
|
||||
{%/note%}
|
||||
|
||||
Run `kubectl get pods` to check whether all the pods for the dependencies are running. You should get a result similar to below.
|
||||
|
||||
```commandline
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
elasticsearch-0 1/1 Running 0 4m26s
|
||||
mysql-0 1/1 Running 0 4m26s
|
||||
openmetadata-dependencies-db-migrations-5984f795bc-t46wh 1/1 Running 0 4m26s
|
||||
openmetadata-dependencies-scheduler-5b574858b6-75clt 1/1 Running 0 4m26s
|
||||
openmetadata-dependencies-sync-users-654b7d58b5-2z5sf 1/1 Running 0 4m26s
|
||||
openmetadata-dependencies-triggerer-8d498cc85-wjn69 1/1 Running 0 4m26s
|
||||
openmetadata-dependencies-web-64bc79d7c6-7n6v2 1/1 Running 0 4m26s
|
||||
```
|
||||
|
||||
Please note that the pods names above as openmetadata-dependencies-* are part of airflow deployments.
|
||||
|
||||
Helm Chart for OpenMetadata Dependencies uses the following helm charts:
|
||||
- [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/8.8.23) (helm chart version 8.8.23)
|
||||
- [ElasticSearch](https://artifacthub.io/packages/helm/elastic/elasticsearch/7.10.2) (helm chart version 8.10.2)
|
||||
- [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.6.1) (helm chart version 8.6.1)
|
||||
|
||||
If you want to customise helm values for the dependencies as per your cluster, you can follow the above links and update
|
||||
your custom helm `values.yaml`.
|
||||
|
||||
Airflow uses DAGs and Logs for persistence volumes with the Kubernetes Executor.
|
||||
|
||||
Make sure your kubernetes cluster storage provisioner has persistent volumes capability of `ReadWriteMany` Access Mode. Modify the Helm Values for airflow as per
|
||||
your requirement here.
|
||||
|
||||
This is not required if you are deploying on kubernetes cluster created by `minkube` or Docker Desktop. Check the storage
|
||||
provider compatibility [here](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes).
|
||||
|
||||
Next, deploy OpenMetadata by running the following command:
|
||||
|
||||
```commandline
|
||||
helm install openmetadata open-metadata/openmetadata
|
||||
```
|
||||
|
||||
Again, this uses the values defined [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml).
|
||||
Use the `--values` flag to point to your own YAML configuration if needed.
|
||||
|
||||
{%note%}
|
||||
|
||||
The values in `values.yaml` are preset to match with dependencies deployed using `openmetadata-dependencies` with release name "openmetadata-dependencies".
|
||||
If you deployed helm chart using different release name, make sure to update `values.yaml` accordingly before installing.
|
||||
|
||||
{%/note%}
|
||||
|
||||
Run `kubectl get pods --selector=app.kubernetes.io/name=openmetadata` to check the status of pods running. You should get a result similar to the output below:
|
||||
|
||||
```commandline
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
openmetadata-5c55f6759c-52dvq 1/1 Running 0 90s
|
||||
```
|
||||
|
||||
## Port Forwarding
|
||||
|
||||
To expose the OpenMetadata UI on a local Kubernetes instance, run this command:
|
||||
|
||||
```commandline
|
||||
kubectl port-forward <openmetadata-front end pod name> 8585:8585
|
||||
```
|
||||
|
||||
## Enable Security
|
||||
|
||||
Please follow our [Enable Security Guide](/deployment/kubernetes/security) to configure security for your OpenMetadata
|
||||
installation.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### View helm chart deployment status
|
||||
|
||||
Run the below command to view status of openmetadata helm chart deployed:
|
||||
|
||||
```commandline
|
||||
helm status openmetadata
|
||||
```
|
||||
|
||||
For more information, visit helm command line reference [here](https://helm.sh/docs/helm/helm_status/).
|
||||
|
||||
### View openmetadata kubernetes pod logs
|
||||
|
||||
Run the below command to list openmetadata kubernetes pods deployed in a namespace:
|
||||
|
||||
```commandline
|
||||
kubectl get pods --namespace <NAMESPACE_NAME> -l='app.kubernetes.io/managed-by=Helm,app.kubernetes.io/instance=<RELEASE_NAME>'
|
||||
```
|
||||
|
||||
For example, list pods deployed by helm release name `ometa` in the namespace `ometa-dev`:
|
||||
|
||||
```commandline
|
||||
kubectl get pods --namespace ometa-dev -l='app.kubernetes.io/managed-by=Helm,app.kubernetes.io/instance=ometa'
|
||||
```
|
||||
|
||||
Next, view the logs of pod by running the below command,
|
||||
|
||||
```commandline
|
||||
kubectl logs <POD_NAME> --namespace <NAMESPACE_NAME>
|
||||
```
|
||||
|
||||
For more information, visit the kubectl logs command line reference documentation [here](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/).
|
||||
|
||||
### Java Memory Heap Issue
|
||||
|
||||
If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue -
|
||||
|
||||
```
|
||||
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncAppender-Worker-async-file-appender"
|
||||
Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "AsyncAppender-Worker-async-file-appender" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "dw-46" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "AsyncAppender-Worker-async-console-appender" java.lang.OutOfMemoryError: Java heap space
|
||||
```
|
||||
|
||||
This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable
|
||||
|
||||
```yaml
|
||||
extraEnvs:
|
||||
- name: OPENMETADATA_HEAP_OPTS
|
||||
value: "-Xmx2G -Xms2G"
|
||||
```
|
||||
|
||||
The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool.
|
||||
|
||||
Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values <values.yml> --namespace <namespaceName>`. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes.
|
||||
|
||||
### Uninstall OpenMetadata Helm Charts
|
||||
|
||||
Use the below command to uninstall OpenMetadata Helm Charts completely.
|
||||
|
||||
```commandline
|
||||
helm uninstall openmetadata
|
||||
helm uninstall openmetadata-dependencies
|
||||
```
|
||||
|
||||
MySql and ElasticSearch OpenMetadata Dependencies as deployed as StatefulSets and have persistent volumes (pv) and
|
||||
persistent volume claims (`pvc`). These will need to be manually cleaned after helm uninstall. You can use `kubectl delete`
|
||||
CLI command for the same.
|
||||
|
||||
## Run OpenMetadata with AWS Services
|
||||
|
||||
If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway).
|
||||
|
||||
We support
|
||||
|
||||
- Amazon RDS (MySQL) engine version 8 or greater
|
||||
- Amazon OpenSearch (ElasticSearch) engine version up to 8.10.2 or Amazon OpenSearch engine version up to 2.7
|
||||
- Amazon RDS (PostgreSQL) engine version between 12 and 14.6
|
||||
|
||||
For Production Systems, we recommend Amazon RDS to be in Multiple Availability Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availability Zones with minimum 3 Master Nodes.
|
||||
|
||||
Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch.
|
||||
|
||||
```yaml
|
||||
# openmetadata-values.prod.yaml
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
elasticsearch:
|
||||
host: <AMAZON_OPENSEARCH_SERVICE_ENDPOINT_WITHOUT_HTTPS>
|
||||
port: 443
|
||||
scheme: https
|
||||
connectionTimeoutSecs: 5
|
||||
socketTimeoutSecs: 60
|
||||
keepAliveTimeoutSecs: 600
|
||||
batchSize: 10
|
||||
auth:
|
||||
enabled: false
|
||||
username: <AMAZON_OPENSEARCH_USERNAME>
|
||||
password:
|
||||
secretRef: elasticsearch-secrets
|
||||
secretKey: openmetadata-elasticsearch-password
|
||||
database:
|
||||
host: <AMAZON_RDS_ENDPOINT>
|
||||
port: 3306
|
||||
driverClass: com.mysql.cj.jdbc.Driver
|
||||
dbScheme: mysql
|
||||
dbParams: "allowPublicKeyRetrieval=true&useSSL=true&serverTimezone=UTC"
|
||||
databaseName: <RDS_DATABASE_NAME>
|
||||
auth:
|
||||
username: <RDS_DATABASE_USERNAME>
|
||||
password:
|
||||
secretRef: mysql-secrets
|
||||
secretKey: openmetadata-mysql-password
|
||||
...
|
||||
```
|
||||
|
||||
Make sure to create RDS and OpenSearch credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/deployment/kubernetes#quickstart).
|
||||
|
||||
|
||||
## Using a Custom Ingestion Image
|
||||
|
||||
One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors.
|
||||
You can find a complete working example of this [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-connector). After
|
||||
you have your code ready, the steps would be the following:
|
||||
|
||||
### 1. Create a `Dockerfile` based on `openmetadata/ingestion`:
|
||||
|
||||
For example:
|
||||
|
||||
```
|
||||
FROM openmetadata/ingestion:x.y.z
|
||||
|
||||
# Let's use the same workdir as the ingestion image
|
||||
WORKDIR ingestion
|
||||
USER airflow
|
||||
|
||||
# Install our custom connector
|
||||
COPY <your_package> <your_package>
|
||||
COPY setup.py .
|
||||
RUN pip install --no-deps .
|
||||
```
|
||||
|
||||
where `openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `openmetadata/ingestion:1.2.0`.
|
||||
This image needs to be built and published to the container registry of your choice.
|
||||
|
||||
### 2. Update your dependency values YAML
|
||||
|
||||
The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use
|
||||
our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the
|
||||
following contents:
|
||||
|
||||
```yaml
|
||||
airflow:
|
||||
airflow:
|
||||
image:
|
||||
repository: <your repository> # by default, openmetadata/ingestion
|
||||
tag: <your tag> # by default, the version you are deploying, e.g., 1.2.0
|
||||
pullPolicy: "IfNotPresent"
|
||||
```
|
||||
|
||||
### 3. Install or Upgrade openmetadata-dependencies
|
||||
|
||||
Upgrade/Install your openmetadata-dependencies helm charts with the below single command:
|
||||
|
||||
```bash
|
||||
helm upgrade --install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values.deps.yaml
|
||||
```
|
||||
|
||||
### 4. Install OpenMetadata
|
||||
|
||||
Finally, follow the usual installation for the `openmetadata` helm charts.
|
||||
Below is the expected Kubernetes Deployment Architecture for OpenMetadata Application in **Production**.
|
||||
|
||||
{% image src="/images/v1.2/deployment/kubernetes/kubernetes-architecture-prod.png" alt="Kubernetes Deployment Architecture" /%}
|
||||
|
||||
In the above architecture diagram, OpenMetadata Application is deployed using Helm Charts. The various kubernetes manifests that supports the installation. With the above architecture, OpenMetadata Application Connects with external dependencies which is Database, ElasticSearch and Orchestration tools like airflow.
|
||||
|
||||
The OpenMetadata Helm Charts Exposes the Application from Kubernetes Service at Port `8585` and `8586`. The Health Checks and Metrics endpoints are available on port `8586`.
|
||||
|
||||
Network Policies and Ingresses are optional manifests and disabled by default. These can be installed / enabled using the [Helm Values](/deployment/kubernetes/helm-values).
|
||||
|
||||
## Links
|
||||
|
||||
{% inlineCalloutContainer %}
|
||||
{% inlineCallout
|
||||
color="violet-70"
|
||||
icon="cloud"
|
||||
bold="Helm Values"
|
||||
href="/deployment/kubernetes/helm-values" %}
|
||||
For customizing OpenMetadata Helm Deployments
|
||||
{% /inlineCallout %}
|
||||
{% inlineCallout
|
||||
color="violet-70"
|
||||
icon="cloud"
|
||||
bold="Deploy in AWS EKS"
|
||||
href="/deployment/kubernetes/eks" %}
|
||||
Deploy OpenMetadata in AWS Kubernetes
|
||||
{% /inlineCallout %}
|
||||
{% inlineCallout
|
||||
color="violet-70"
|
||||
icon="cloud"
|
||||
bold="Deploy in GCP GKE"
|
||||
href="/deployment/kubernetes/gke" %}
|
||||
Deploy OpenMetadata in GCP Kubernetes
|
||||
{% /inlineCallout %}
|
||||
{% inlineCallout
|
||||
color="violet-70"
|
||||
icon="cloud"
|
||||
bold="Deploy in OnPremises Kubernetes"
|
||||
href="/deployment/kubernetes/on-prem" %}
|
||||
Deploy OpenMetadata in On Premises Kubernetes
|
||||
{% /inlineCallout %}
|
||||
{% inlineCallout
|
||||
color="violet-70"
|
||||
icon="cloud"
|
||||
bold="FAQs"
|
||||
href="/deployment/kubernetes/faqs" %}
|
||||
FAQs for common production scenarios
|
||||
{% /inlineCallout %}
|
||||
{% /inlineCalloutContainer %}
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Kubernetes On Premises Deployment
|
||||
slug: /deployment/kubernetes/onprem
|
||||
slug: /deployment/kubernetes/on-prem
|
||||
---
|
||||
|
||||
# On Premises Kubernetes Deployment
|
||||
@ -8,9 +8,9 @@ slug: /deployment/kubernetes/onprem
|
||||
OpenMetadata supports the Installation and Running of application on OnPremises Kubernetes through Helm Charts.
|
||||
However, there are some additional configurations which needs to be done as prerequisites for the same.
|
||||
|
||||
{%note%}
|
||||
{%note noteType="Warning"%}
|
||||
|
||||
This guide presumes you have an on premises Kubernetes cluster setup and you are installing OpenMetadata in `default` namespace.
|
||||
This guide presumes you have an on premises Kubernetes cluster setup, and you are installing OpenMetadata in `default` namespace.
|
||||
|
||||
{%/note%}
|
||||
|
||||
@ -24,7 +24,7 @@ The workaround is to create nfs-share and use that as the persistent claim to de
|
||||
|
||||
{%note%}
|
||||
|
||||
This guide assumes you have NFS Server already setup with Hostname or IP Address which is reachable from your on premises Kubernetes cluster and you have configured a path to be used for OpenMetadata Airflow Helm Dependency.
|
||||
This guide assumes you have NFS Server already setup with Hostname or IP Address which is reachable from your on premises Kubernetes cluster, and you have configured a path to be used for OpenMetadata Airflow Helm Dependency.
|
||||
|
||||
{%/note%}
|
||||
|
||||
@ -109,7 +109,7 @@ kubectl create -f logs_pvc.yml
|
||||
|
||||
## Change owner and permission manually on disks
|
||||
|
||||
Since airflow pods run as non root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once.
|
||||
Since airflow pods run as non-root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once.
|
||||
|
||||
```yaml
|
||||
# permissions_pod.yml
|
||||
@ -146,7 +146,7 @@ spec:
|
||||
|
||||
{%note%}
|
||||
|
||||
Airflow runs the pods with linux user name as airflow and linux user id as 50000.
|
||||
Airflow runs the pods with linux username as airflow and linux user id as 50000.
|
||||
|
||||
{%/note%}
|
||||
|
||||
@ -188,7 +188,7 @@ airflow:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3).
|
||||
For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0).
|
||||
|
||||
When deploying openmetadata dependencies helm chart, use the below command -
|
||||
|
||||
|
||||
@ -9,7 +9,7 @@ Follow the steps for setting up the SSO, and then check the specific `Kubernetes
|
||||
|
||||
{%note%}
|
||||
|
||||
By default Basic Authentication will be enabled as authentication mechanism.
|
||||
By default, Basic Authentication will be enabled as authentication mechanism.
|
||||
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
@ -58,7 +58,7 @@ By default Basic Authentication will be enabled as authentication mechanism.
|
||||
href="/deployment/security/google/kubernetes"%}
|
||||
Configure Google SSO to access the UI and APIs
|
||||
{%/inlineCallout%}
|
||||
{%/inlineCallout
|
||||
{%inlineCallout
|
||||
color="violet-70"
|
||||
bold="Okta SSO"
|
||||
icon="add_moderator"
|
||||
|
||||
@ -81,8 +81,6 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
| openmetadata.config.airflow.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER_HOST_API_URL |
|
||||
| openmetadata.config.airflow.sslCertificatePath | string | `/no/path` | PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH |
|
||||
| openmetadata.config.airflow.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
|
||||
| openmetadata.config.basicLogin.maxLoginFailAttempts | int | 3 | OM_MAX_FAILED_LOGIN_ATTEMPTS |
|
||||
| openmetadata.config.basicLogin.accessBlockTime | int | 600 | OM_LOGIN_ACCESS_BLOCK_TIME |
|
||||
| openmetadata.config.clusterName | string | `openmetadata` | OPENMETADATA_CLUSTER_NAME |
|
||||
| openmetadata.config.database.auth.password.secretRef | string | `mysql-secrets` | DB_USER_PASSWORD |
|
||||
| openmetadata.config.database.auth.password.secretKey | string | `openmetadata-mysql-password` | DB_USER_PASSWORD |
|
||||
@ -97,10 +95,10 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
| openmetadata.config.elasticsearch.auth.username | string | `elasticsearch` | ELASTICSEARCH_USER |
|
||||
| openmetadata.config.elasticsearch.auth.password.secretRef | string | `elasticsearch-secrets` | ELASTICSEARCH_PASSWORD |
|
||||
| openmetadata.config.elasticsearch.auth.password.secretKey | string | `openmetadata-elasticsearch-password` | ELASTICSEARCH_PASSWORD |
|
||||
| openmetadata.config.elasticsearch.host | string | `elasticsearch` | ELASTICSEARCH_HOST |
|
||||
| openmetadata.config.elasticsearch.host | string | `opensearch` | ELASTICSEARCH_HOST |
|
||||
| openmetadata.config.elasticsearch.keepAliveTimeoutSecs | int | `600` | ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS |
|
||||
| openmetadata.config.elasticsearch.port | int | 9200 | ELASTICSEARCH_PORT |
|
||||
| openmetadata.config.elasticsearch.searchType | string | `elasticsearch` | SEARCH_TYPE |
|
||||
| openmetadata.config.elasticsearch.searchType | string | `opensearch` | SEARCH_TYPE |
|
||||
| openmetadata.config.elasticsearch.scheme | string | `http` | ELASTICSEARCH_SCHEME |
|
||||
| openmetadata.config.elasticsearch.searchIndexMappingLanguage | string | `EN`| ELASTICSEARCH_INDEX_MAPPING_LANG |
|
||||
| openmetadata.config.elasticsearch.trustStore.enabled | bool | `false` | |
|
||||
@ -121,7 +119,6 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
| openmetadata.config.jwtTokenConfiguration.keyId | string | `Gb389a-9f76-gdjs-a92j-0242bk94356` | JWT_KEY_ID |
|
||||
| openmetadata.config.logLevel | string | `INFO` | LOG_LEVEL |
|
||||
| openmetadata.config.maskPasswordsApi | bool | `false` | MASK_PASSWORDS_API |
|
||||
| openmetadata.config.migrationConfigs.extensionPath | string | `Empty string` | MIGRATION_EXTENSION_PATH |
|
||||
| openmetadata.config.openmetadata.adminPort | int | 8586 | SERVER_ADMIN_PORT |
|
||||
| openmetadata.config.openmetadata.host | string | `openmetadata` | OPENMETADATA_SERVER_URL |
|
||||
| openmetadata.config.openmetadata.port | int | 8585 | SERVER_PORT |
|
||||
@ -193,7 +190,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
| fullnameOverride | string | `"openmetadata"` |
|
||||
| image.pullPolicy | string | `"Always"` |
|
||||
| image.repository | string | `"docker.getcollate.io/openmetadata/server"` |
|
||||
| image.tag | string | `1.2.0` |
|
||||
| image.tag | string | `1.2.1` |
|
||||
| imagePullSecrets | list | `[]` |
|
||||
| ingress.annotations | object | `{}` |
|
||||
| ingress.className | string | `""` |
|
||||
|
||||
@ -38,8 +38,6 @@ site_menu:
|
||||
|
||||
- category: Deployment / Docker Deployment
|
||||
url: /deployment/docker
|
||||
- category: Deployment / Docker Deployment / Volumes
|
||||
url: /deployment/docker/volumes
|
||||
- category: Deployment / Docker Deployment / Enable Security
|
||||
url: /deployment/docker/security
|
||||
|
||||
@ -57,8 +55,8 @@ site_menu:
|
||||
url: /deployment/kubernetes/security
|
||||
- category: Deployment / Kubernetes Deployment / Helm Values
|
||||
url: /deployment/kubernetes/helm-values
|
||||
- category: Deployment / Kubernetes Deployment / GKE Troubleshooting
|
||||
url: /deployment/kubernetes/gke-troubleshooting
|
||||
- category: Deployment / Kubernetes Deployment / FAQs
|
||||
url: /deployment/kubernetes/faqs
|
||||
|
||||
- category: Deployment / Ingestion
|
||||
url: /deployment/ingestion
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
Loading…
x
Reference in New Issue
Block a user