From dd52612b26f989882dc9f6d2c6efa91128cf183b Mon Sep 17 00:00:00 2001 From: Akash Jain <15995028+akash-jain-10@users.noreply.github.com> Date: Wed, 17 Aug 2022 20:40:39 +0530 Subject: [PATCH] feat: helm-aws-eks-docs (#6745) * Added EKS Docs * minor fixes to docs * fix: typo * Add note on assumption --- .../content/deployment/kubernetes/eks.md | 214 ++++++++++++++++++ .../content/deployment/kubernetes/gke.md | 2 + .../content/deployment/kubernetes/on-prem.md | 10 +- openmetadata-docs/content/menu.md | 9 +- 4 files changed, 227 insertions(+), 8 deletions(-) create mode 100644 openmetadata-docs/content/deployment/kubernetes/eks.md diff --git a/openmetadata-docs/content/deployment/kubernetes/eks.md b/openmetadata-docs/content/deployment/kubernetes/eks.md new file mode 100644 index 00000000000..5cfada82b46 --- /dev/null +++ b/openmetadata-docs/content/deployment/kubernetes/eks.md @@ -0,0 +1,214 @@ +--- +title: AWS EKS Deployment +slug: /deployment/kubernetes/eks +--- + +# EKS on Amazon Web Services Deployment + +OpenMetadata supports the Installation and Running of Application on Elastic Kubernetes Services (EKS) through Helm Charts. +However, there are some additional configurations which needs to be done as prerequisites for the same. + + + +All the code snippets in this section assume the `default` namespace for kubernetes. +This guide presumes you have AWS EKS Cluster already available. + + + +## Prerequisites + +### Create Elastic File System in AWS + +You can follow official AWS Guides [here](https://docs.aws.amazon.com/efs/latest/ug/gs-step-two-create-efs-resources.html) to provision EFS File System in the same VPC which is associated with your EKS Cluster. + +### Persistent Volumes with ReadWriteMany Access Modes + +OpenMetadata helm chart depends on Airflow and Airflow expects a presistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). + +In AWS, this is achieved by Elastic File System (EFS) service. AWS Elastic Block Store (EBS) does not provide ReadWriteMany Volume access mode as EBS will only be attached to one Kubernetes Node at any given point of time. + +In order to provision persistent volumes from AWS EFS, you will need to setup and install [aws-efs-csi-driver](https://github.com/kubernetes-sigs/aws-efs-csi-driver). + +The below guide provides Persistent Volumes provisioning as static volumes (meaning you will be responsible to create, maintain and destroy Persistent Volumes). + +## Provision EFS backed PVs, PVCs for Airflow DAGs and Airflow Logs + + + +```yaml +# dags_pv_pvc.yml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: openmetadata-dependencies-dags-pv + labels: + app: airflow-dags +spec: + storageClassName: "" + accessModes: + - ReadWriteMany + persistentVolumeReclaimPolicy: Retain + csi: + driver: efs.csi.aws.com + volumeHandle: [FileSystemId] # Replace with EFS File System Id + +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + labels: + app: airflow-dags + name: openmetadata-dependencies-dags-pvc + namespace: default +spec: + accessModes: + - ReadWriteMany + storageClassName: "" + resources: + requests: + storage: 5Gi +``` + +Create Persistent Volumes and Persistent Volume claims with the below command. + +```commandline +kubectl create -f dags_pv_pvc.yml +``` + + + + + +```yaml +# logs_pv_pvc.yml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: openmetadata-dependencies-logs-pv + labels: + app: airflow-logs +spec: + storageClassName: "" + accessModes: + - ReadWriteMany + persistentVolumeReclaimPolicy: Retain + csi: + driver: efs.csi.aws.com + volumeHandle: [FileSystemId] # Replace with EFS File System Id + +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: openmetadata-dependencies-logs-pvc + namespace: default + labels: + app: airflow-dags +spec: + accessModes: + - ReadWriteMany + storageClassName: "" + resources: + requests: + storage: 10Gi +``` + +Create Persistent Volumes and Persistent Volume claims with the below command. + +```commandline +kubectl create -f logs_pv_pvc.yml +``` + + + +## Change owner and permission manually on disks + +Since airflow pods run as non root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. + +You can find more reference on AWS EFS permissions in docs [here](https://docs.aws.amazon.com/efs/latest/ug/using-fs.html). + +```yaml +# permissions_pod.yml +apiVersion: v1 +kind: Pod +metadata: + creationTimestamp: null + labels: + run: my-permission-pod + name: my-permission-pod +spec: + containers: + - image: busybox + name: my-permission-pod + volumeMounts: + - name: airflow-dags + mountPath: /airflow-dags + - name: airflow-logs + mountPath: /airflow-logs + command: + - "chown -R 50000 /airflow-dags /airflow-logs" + # if needed + - "chmod -R a+rwx /airflow-dags" + volumes: + - name: airflow-logs + persistentVolumeClaim: + claimName: openmetadata-dependencies-logs + - name: airflow-dags + persistentVolumeClaim: + claimName: openmetadata-dependencies-dags + dnsPolicy: ClusterFirst + restartPolicy: Always +``` + + + +Airflow runs the pods with linux user name as airflow and linux user id as 50000. + + + +Run the below command to create the pod and fix the permissions + +```commandline +kubectl create -f permissions_pod.yml +``` + +## Create OpenMetadata dependencies Values + +Override openmetadata dependencies airflow helm values to bind the efs persistent volumes for DAGs and logs. + +```yaml +# values-dependencies.yml +airflow: + airflow: + extraVolumeMounts: + - mountPath: /airflow-logs + name: efs-airflow-logs + - mountPath: /airflow-dags/dags + name: efs-airflow-dags + extraVolumes: + - name: efs-airflow-logs + persistentVolumeClaim: + claimName: openmetadata-dependencies-logs + - name: efs-airflow-dags + persistentVolumeClaim: + claimName: openmetadata-dependencies-dags + config: + AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/airflow-dags/dags" + dags: + path: /airflow-dags/dags + persistence: + enabled: false + logs: + path: /airflow-logs + persistence: + enabled: false +``` + +For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3). + +Follow [OpenMetadata Kubernetes Deployment](/deployment/kubernetes) to install and deploy helm charts with EFS volumes. +When deploying openmetadata dependencies helm chart, use the below command - + +```commandline +helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values-dependencies.yaml +``` diff --git a/openmetadata-docs/content/deployment/kubernetes/gke.md b/openmetadata-docs/content/deployment/kubernetes/gke.md index 59d313b2023..68462b27583 100644 --- a/openmetadata-docs/content/deployment/kubernetes/gke.md +++ b/openmetadata-docs/content/deployment/kubernetes/gke.md @@ -233,7 +233,9 @@ spec: ``` + Airflow runs the pods with linux user name as airflow and linux user id as 50000. + Run the below command to create the pod and fix the permissions diff --git a/openmetadata-docs/content/deployment/kubernetes/on-prem.md b/openmetadata-docs/content/deployment/kubernetes/on-prem.md index 5ba6a99c14d..390dfa005d4 100644 --- a/openmetadata-docs/content/deployment/kubernetes/on-prem.md +++ b/openmetadata-docs/content/deployment/kubernetes/on-prem.md @@ -51,9 +51,9 @@ Replace the `NFS_HOSTNAME_OR_IP` with your NFS Server value and run the commands This will create a new StorageClass with `nfs-subdir-external-provisioner`. You can view the same using the kubectl command `kubectl get storageclass -n nfs-provisioner`.

-### Provision NFS backed PV and PVC for Airflow DAGs and Airflow Logs +## Provision NFS backed PVC for Airflow DAGs and Airflow Logs - + ```yaml # dags_pvc.yml @@ -82,7 +82,7 @@ kubectl create -f dags_pvc.yml - + ```yaml # logs_pvc.yml @@ -149,7 +149,9 @@ spec: ``` + Airflow runs the pods with linux user name as airflow and linux user id as 50000. + Run the below command to create the pod and fix the permissions @@ -193,7 +195,7 @@ airflow: For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3). Follow [OpenMetadata Kubernetes Deployment](/deployment/kubernetes) to install and deploy helm charts with nfs volumes. -When deploying openmeteadata dependencies helm chart, use the below command - +When deploying openmetadata dependencies helm chart, use the below command - ```commandline helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values values-dependencies.yaml diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index 77b58050a2b..5bff37d3b0a 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -40,11 +40,12 @@ site_menu: - category: Deployment / Kubernetes Deployment url: /deployment/kubernetes - - category: Deployment / Kubernetes Deployment / Kubernetes Helm Values - url: /deployment/kubernetes/helm-values - - category: Deployment / Kubernetes Deployment / GKE Deployment + - category: Deployment / Kubernetes Deployment / EKS + url: /deployment/kubernetes/eks + - category: Deployment / Kubernetes Deployment / GKE url: /deployment/kubernetes/gke - - category: Deployment / Kubernetes Deployment / Kubernetes On Premises Deployment + + - category: Deployment / Kubernetes Deployment / On Premises url: /deployment/kubernetes/onprem - category: Deployment / Kubernetes Deployment / Enable Security url: /deployment/kubernetes/security