diff --git a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/values.md b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/values.md index b892388a66a..19dc8268857 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/kubernetes/values.md +++ b/openmetadata-docs/content/v1.3.x/deployment/kubernetes/values.md @@ -14,6 +14,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | Key | Type | Default | Environment Variable from openmetadata.yaml | |-----|------|---------| ---------------------- | | openmetadata.config.authentication.enabled | bool | `true` | | +| openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION_CLIENT_TYPE | | openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER | | openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS | | openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY | @@ -50,6 +51,24 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES | +| openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC_CALLBACK | +| openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC_CLIENT_AUTH_METHOD | +| openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC_CLIENT_ID | +| openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC_CLIENT_ID | +| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC_CLIENT_SECRET | +| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC_CLIENT_SECRET | +| openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC_CUSTOM_PARAMS | +| openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC_DISABLE_PKCE | +| openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC_DISCOVERY_URI | +| openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | | +| openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC_MAX_CLOCK_SKEW | +| openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC_TYPE | +| openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC_PREFERRED_JWS | +| openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC_RESPONSE_TYPE | +| openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC_SCOPE | +| openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC_SERVER_URL | +| openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC_TENANT | +| openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC_USE_NONCE | | openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE | | openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID | | openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL | @@ -88,6 +107,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.airflow.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW_PASSWORD | | openmetadata.config.airflow.auth.username | string | `admin` | AIRFLOW_USERNAME | | openmetadata.config.airflow.enabled | bool | `true` | | +| openmetadata.config.airflow.host | string | `http://openmetadata-dependencies-web:8080` | PIPELINE_SERVICE_CLIENT_ENDPOINT | | openmetadata.config.airflow.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER_HOST_API_URL | | openmetadata.config.airflow.sslCertificatePath | string | `/no/path` | PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH | | openmetadata.config.airflow.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL | @@ -151,12 +171,20 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL | | openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP | | openmetadata.config.secretsManager.enabled | bool | `true` | | -| openmetadata.config.secretsManager.provider | string | `db` | SECRET_MANAGER | +| openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET_MANAGER | | openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX | | openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS | | openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID | +| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM_SM_CLIENT_ID | +| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_ID | +| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM_SM_CLIENT_SECRET | +| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_SECRET | +| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM_SM_TENANT_ID | +| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM_SM_TENANT_ID | +| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM_SM_VAULT_NAME | +| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM_SM_VAULT_NAME | | openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY | @@ -170,8 +198,8 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL | | openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY | | openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME | -| openmetadata.config.upgradeMigrationConfigs.force | bool | `false` | | -| openmetadata.config.upgradeMigrationConfigs.migrationLimitParam | int | `1200` | MIGRATION_LIMIT_PARAM | +| openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | | +| openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | | | openmetadata.config.web.enabled | bool | `true` | | | openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED | | openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED | @@ -210,7 +238,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | fullnameOverride | string | `"openmetadata"` | | image.pullPolicy | string | `"Always"` | | image.repository | string | `"docker.getcollate.io/openmetadata/server"` | -| image.tag | string | `1.3.1` | +| image.tag | string | `1.3.3` | | imagePullSecrets | list | `[]` | | ingress.annotations | object | `{}` | | ingress.className | string | `""` | diff --git a/openmetadata-docs/content/v1.3.x/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md b/openmetadata-docs/content/v1.3.x/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md index 23f9e208d24..1a1ae3d78af 100644 --- a/openmetadata-docs/content/v1.3.x/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md +++ b/openmetadata-docs/content/v1.3.x/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md @@ -64,9 +64,19 @@ secretsManagerConfiguration: parameters: enabled: true vaultName: ${OM_SM_VAULT_NAME:-""} +pipelineServiceClientConfiguration: + secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow} +``` + +For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - + +```yaml +--- +... extraEnvs: - - name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER - value: airflow +- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER + value: airflow +... ``` #### Client Secret Credential @@ -83,9 +93,19 @@ secretsManagerConfiguration: clientSecret: ${OM_SM_CLIENT_SECRET:-""} tenantId: ${OM_SM_TENANT_ID:-""} vaultName: ${OM_SM_VAULT_NAME:-""} +pipelineServiceClientConfiguration: + secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow} +``` + +For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - + +```yaml +--- +... extraEnvs: - - name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER - value: airflow +- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER + value: airflow +... ``` The changes to be done in `airflow.yaml` file of the Airflow are: diff --git a/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/kubernetes/values.md b/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/kubernetes/values.md index 79af2bd6e28..19dc8268857 100644 --- a/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/kubernetes/values.md +++ b/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/kubernetes/values.md @@ -14,6 +14,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | Key | Type | Default | Environment Variable from openmetadata.yaml | |-----|------|---------| ---------------------- | | openmetadata.config.authentication.enabled | bool | `true` | | +| openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION_CLIENT_TYPE | | openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER | | openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS | | openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY | @@ -50,6 +51,24 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES | +| openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC_CALLBACK | +| openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC_CLIENT_AUTH_METHOD | +| openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC_CLIENT_ID | +| openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC_CLIENT_ID | +| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC_CLIENT_SECRET | +| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC_CLIENT_SECRET | +| openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC_CUSTOM_PARAMS | +| openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC_DISABLE_PKCE | +| openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC_DISCOVERY_URI | +| openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | | +| openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC_MAX_CLOCK_SKEW | +| openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC_TYPE | +| openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC_PREFERRED_JWS | +| openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC_RESPONSE_TYPE | +| openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC_SCOPE | +| openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC_SERVER_URL | +| openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC_TENANT | +| openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC_USE_NONCE | | openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE | | openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID | | openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL | @@ -152,12 +171,20 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL | | openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP | | openmetadata.config.secretsManager.enabled | bool | `true` | | -| openmetadata.config.secretsManager.provider | string | `db` | SECRET_MANAGER | +| openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET_MANAGER | | openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX | | openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS | | openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID | +| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM_SM_CLIENT_ID | +| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_ID | +| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM_SM_CLIENT_SECRET | +| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_SECRET | +| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM_SM_TENANT_ID | +| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM_SM_TENANT_ID | +| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM_SM_VAULT_NAME | +| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM_SM_VAULT_NAME | | openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY | @@ -171,8 +198,8 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL | | openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY | | openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME | -| openmetadata.config.upgradeMigrationConfigs.force | bool | `false` | | -| openmetadata.config.upgradeMigrationConfigs.migrationLimitParam | int | `1200` | MIGRATION_LIMIT_PARAM | +| openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | | +| openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | | | openmetadata.config.web.enabled | bool | `true` | | | openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED | | openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED | @@ -211,7 +238,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | fullnameOverride | string | `"openmetadata"` | | image.pullPolicy | string | `"Always"` | | image.repository | string | `"docker.getcollate.io/openmetadata/server"` | -| image.tag | string | `1.3.1` | +| image.tag | string | `1.3.3` | | imagePullSecrets | list | `[]` | | ingress.annotations | object | `{}` | | ingress.className | string | `""` | diff --git a/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md b/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md index 05aae16b420..1a1ae3d78af 100644 --- a/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md +++ b/openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md @@ -10,42 +10,106 @@ for the non-managed follow only the steps related to the Airflow server and CLI. ## Setup -### 1. Permissions needed +### 1. Create Principal + +#### Service Principal 1. Go to `Microsoft Entra ID` and create an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app). 2. Inside the App Registration, go to `Certificates & Secrets` and create a `Client secret`. Note down the `Value`, it will be our `clientSecret` configuration. 3. From the App Registration overview page, note down the `Application (client) ID` and the `Directory (tenant) ID`. -4. In your Key Vault overview page, note down the `Vault URI`. -5. Go to `Access Control (IAM)` and click on `Add Role Assignment`. -6. Give the permission `Key Vault Secrets Officer` to your App Registration. -### 2. Update configuration +#### Managed Identity (recommnded) -We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials for our -App Registration. +1. In your Azure subscription create [Manged Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) +2. Use this created identity - for AKS users this means you need to use [Pod Identity](https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity) or [Workload Identity (recommnded)](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet). + +{% note %} + +Note that the using Managed Identity require using [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential). + +{% /note %} + +### 2. Add RBAC roles + +It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - `Key Vault Secrets Officer` - to be able to create/read/update secrets in the Vault. + +While the Airflow part only needs to read the secrets hence the role `Key Vault Secrets Officer`. + +#### Open Metadata server + +1. In your Key Vault overview page, note down the `Vault URI`. +2. Go to `Access Control (IAM)` and click on `Add Role Assignment`. +3. Give the permission `Key Vault Secrets Officer` to your Principal. + +#### Airflow + +1. In your Key Vault overview page, note down the `Vault URI`. +2. Go to `Access Control (IAM)` and click on `Add Role Assignment`. +3. Give the permission `Key Vault Secrets Users` to your Principal. + +### 3. Update configuration + +We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: -```yaml -... -secretsManagerConfiguration: - secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'. - prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as -- - tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]` - parameters: - clientId: ${OM_SM_CLIENT_ID:-""} - clientSecret: ${OM_SM_CLIENT_SECRET:-""} - tenantId: ${OM_SM_TENANT_ID:-""} - vaultName: ${OM_SM_VAULT_NAME:-""} +#### Default Azure Credential +```yaml +--- +secretsManagerConfiguration: + secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'. + prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as -- + tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]` + parameters: + enabled: true + vaultName: ${OM_SM_VAULT_NAME:-""} pipelineServiceClientConfiguration: - # ... - # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env - # Supported: noop, airflow, env - secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"} + secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow} +``` + +For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - + +```yaml +--- +... +extraEnvs: +- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER + value: airflow ... ``` +#### Client Secret Credential + +```yaml +--- +secretsManagerConfiguration: + secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'. + prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as -- + tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]` + parameters: + enabled: true + clientId: ${OM_SM_CLIENT_ID:-""} + clientSecret: ${OM_SM_CLIENT_SECRET:-""} + tenantId: ${OM_SM_TENANT_ID:-""} + vaultName: ${OM_SM_VAULT_NAME:-""} +pipelineServiceClientConfiguration: + secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow} +``` + +For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - + +```yaml +--- +... +extraEnvs: +- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER + value: airflow +... +``` + +The changes to be done in `airflow.yaml` file of the Airflow are: + {% note %} Note that the **Key Vault Name** parameter is MANDATORY for the system to know where to store and retrieve the secrets. @@ -75,16 +139,40 @@ AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= `, parameter is provided, we will use Azure's [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential). +{% note %} + +Also if you are using [Microsoft Entra Workload ID](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) with [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) then you need also to use projected service account instead one created by Airflow and OpenMetadata: + +airflow.yaml: + +```yaml +--- +serviceAccount: + create: false + name: "name-of-your-service-account" +``` + +openmetadata.yaml: + +```yaml +--- +serviceAccount: + create: false + name: "name-of-your-service-account" +``` + +{% /note %} + ### 3. Restart both servers -After updating the configuration files, we are ready to restart both services. When the OM server starts, it will -automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it +After updating the configuration files, we are ready to restart both services. When the OM server starts, it will +automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it from our DB. -If everything goes as planned, all the data would be displayed using the parameters names which starts with +If everything goes as planned, all the data would be displayed using the parameters names which starts with `openmetadata-...` in your Key Vault console. -**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have +**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `--` @@ -93,7 +181,7 @@ to the created secret. ## CLI -After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the +After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration: ```yaml @@ -106,7 +194,7 @@ workflowConfig: ``` Make sure to follow the steps [here](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential) to allow -the Python client to authenticate to Azure. +the Python client to authenticate to Azure. {% note %} @@ -118,7 +206,6 @@ Key Vault service to point to. You can specify as well the environment variables of your App Registration if you're running the ingestion outside of Azure: [docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python). - ## Airflow If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure