chore(docs): Update Helm Values; Azure Key Vault Docs for 1.4.X (#16014)

* chore(docs): Update Helm Values; Azure Key Vault Docs for 1.4.X

* fix: use bare-metal config values and update example

* docs: Add note on PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER usage with Helm Values
This commit is contained in:
Akash Jain 2024-04-24 15:53:58 +05:30 committed by GitHub
parent cd73328f87
commit f90f1301d9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 203 additions and 41 deletions

View File

@ -14,6 +14,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| Key | Type | Default | Environment Variable from openmetadata.yaml | | Key | Type | Default | Environment Variable from openmetadata.yaml |
|-----|------|---------| ---------------------- | |-----|------|---------| ---------------------- |
| openmetadata.config.authentication.enabled | bool | `true` | | | openmetadata.config.authentication.enabled | bool | `true` | |
| openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION_CLIENT_TYPE |
| openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER | | openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER |
| openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS | | openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS |
| openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY | | openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY |
@ -50,6 +51,24 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES |
| openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC_CALLBACK |
| openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC_CLIENT_AUTH_METHOD |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC_CUSTOM_PARAMS |
| openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC_DISABLE_PKCE |
| openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC_DISCOVERY_URI |
| openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | |
| openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC_MAX_CLOCK_SKEW |
| openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC_TYPE |
| openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC_PREFERRED_JWS |
| openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC_RESPONSE_TYPE |
| openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC_SCOPE |
| openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC_SERVER_URL |
| openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC_TENANT |
| openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC_USE_NONCE |
| openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE | | openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE |
| openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID | | openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID |
| openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL | | openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL |
@ -88,6 +107,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.airflow.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW_PASSWORD | | openmetadata.config.airflow.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW_PASSWORD |
| openmetadata.config.airflow.auth.username | string | `admin` | AIRFLOW_USERNAME | | openmetadata.config.airflow.auth.username | string | `admin` | AIRFLOW_USERNAME |
| openmetadata.config.airflow.enabled | bool | `true` | | | openmetadata.config.airflow.enabled | bool | `true` | |
| openmetadata.config.airflow.host | string | `http://openmetadata-dependencies-web:8080` | PIPELINE_SERVICE_CLIENT_ENDPOINT |
| openmetadata.config.airflow.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER_HOST_API_URL | | openmetadata.config.airflow.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER_HOST_API_URL |
| openmetadata.config.airflow.sslCertificatePath | string | `/no/path` | PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH | | openmetadata.config.airflow.sslCertificatePath | string | `/no/path` | PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH |
| openmetadata.config.airflow.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL | | openmetadata.config.airflow.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
@ -151,12 +171,20 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL | | openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
| openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP | | openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP |
| openmetadata.config.secretsManager.enabled | bool | `true` | | | openmetadata.config.secretsManager.enabled | bool | `true` | |
| openmetadata.config.secretsManager.provider | string | `db` | SECRET_MANAGER | | openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET_MANAGER |
| openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX | | openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX |
| openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS | | openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS |
| openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | | | openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION | | openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY |
@ -170,8 +198,8 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL | | openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL |
| openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY | | openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY |
| openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME | | openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME |
| openmetadata.config.upgradeMigrationConfigs.force | bool | `false` | | | openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | |
| openmetadata.config.upgradeMigrationConfigs.migrationLimitParam | int | `1200` | MIGRATION_LIMIT_PARAM | | openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | |
| openmetadata.config.web.enabled | bool | `true` | | | openmetadata.config.web.enabled | bool | `true` | |
| openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED | | openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED |
| openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED | | openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED |
@ -210,7 +238,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| fullnameOverride | string | `"openmetadata"` | | fullnameOverride | string | `"openmetadata"` |
| image.pullPolicy | string | `"Always"` | | image.pullPolicy | string | `"Always"` |
| image.repository | string | `"docker.getcollate.io/openmetadata/server"` | | image.repository | string | `"docker.getcollate.io/openmetadata/server"` |
| image.tag | string | `1.3.1` | | image.tag | string | `1.3.3` |
| imagePullSecrets | list | `[]` | | imagePullSecrets | list | `[]` |
| ingress.annotations | object | `{}` | | ingress.annotations | object | `{}` |
| ingress.className | string | `""` | | ingress.className | string | `""` |

View File

@ -64,9 +64,19 @@ secretsManagerConfiguration:
parameters: parameters:
enabled: true enabled: true
vaultName: ${OM_SM_VAULT_NAME:-""} vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs: extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER - name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow value: airflow
...
``` ```
#### Client Secret Credential #### Client Secret Credential
@ -83,9 +93,19 @@ secretsManagerConfiguration:
clientSecret: ${OM_SM_CLIENT_SECRET:-""} clientSecret: ${OM_SM_CLIENT_SECRET:-""}
tenantId: ${OM_SM_TENANT_ID:-""} tenantId: ${OM_SM_TENANT_ID:-""}
vaultName: ${OM_SM_VAULT_NAME:-""} vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs: extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER - name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow value: airflow
...
``` ```
The changes to be done in `airflow.yaml` file of the Airflow are: The changes to be done in `airflow.yaml` file of the Airflow are:

View File

@ -14,6 +14,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| Key | Type | Default | Environment Variable from openmetadata.yaml | | Key | Type | Default | Environment Variable from openmetadata.yaml |
|-----|------|---------| ---------------------- | |-----|------|---------| ---------------------- |
| openmetadata.config.authentication.enabled | bool | `true` | | | openmetadata.config.authentication.enabled | bool | `true` | |
| openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION_CLIENT_TYPE |
| openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER | | openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER |
| openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS | | openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS |
| openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY | | openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY |
@ -50,6 +51,24 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES |
| openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC_CALLBACK |
| openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC_CLIENT_AUTH_METHOD |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC_CUSTOM_PARAMS |
| openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC_DISABLE_PKCE |
| openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC_DISCOVERY_URI |
| openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | |
| openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC_MAX_CLOCK_SKEW |
| openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC_TYPE |
| openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC_PREFERRED_JWS |
| openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC_RESPONSE_TYPE |
| openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC_SCOPE |
| openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC_SERVER_URL |
| openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC_TENANT |
| openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC_USE_NONCE |
| openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE | | openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE |
| openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID | | openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID |
| openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL | | openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL |
@ -152,12 +171,20 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL | | openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
| openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP | | openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP |
| openmetadata.config.secretsManager.enabled | bool | `true` | | | openmetadata.config.secretsManager.enabled | bool | `true` | |
| openmetadata.config.secretsManager.provider | string | `db` | SECRET_MANAGER | | openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET_MANAGER |
| openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX | | openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX |
| openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS | | openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS |
| openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | | | openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION | | openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY |
@ -171,8 +198,8 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL | | openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL |
| openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY | | openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY |
| openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME | | openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME |
| openmetadata.config.upgradeMigrationConfigs.force | bool | `false` | | | openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | |
| openmetadata.config.upgradeMigrationConfigs.migrationLimitParam | int | `1200` | MIGRATION_LIMIT_PARAM | | openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | |
| openmetadata.config.web.enabled | bool | `true` | | | openmetadata.config.web.enabled | bool | `true` | |
| openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED | | openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED |
| openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED | | openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED |
@ -211,7 +238,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| fullnameOverride | string | `"openmetadata"` | | fullnameOverride | string | `"openmetadata"` |
| image.pullPolicy | string | `"Always"` | | image.pullPolicy | string | `"Always"` |
| image.repository | string | `"docker.getcollate.io/openmetadata/server"` | | image.repository | string | `"docker.getcollate.io/openmetadata/server"` |
| image.tag | string | `1.3.1` | | image.tag | string | `1.3.3` |
| imagePullSecrets | list | `[]` | | imagePullSecrets | list | `[]` |
| ingress.annotations | object | `{}` | | ingress.annotations | object | `{}` |
| ingress.className | string | `""` | | ingress.className | string | `""` |

View File

@ -10,42 +10,106 @@ for the non-managed follow only the steps related to the Airflow server and CLI.
## Setup ## Setup
### 1. Permissions needed ### 1. Create Principal
#### Service Principal
1. Go to `Microsoft Entra ID` and create an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app). 1. Go to `Microsoft Entra ID` and create an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app).
2. Inside the App Registration, go to `Certificates & Secrets` and create a `Client secret`. Note down the `Value`, it will be our `clientSecret` configuration. 2. Inside the App Registration, go to `Certificates & Secrets` and create a `Client secret`. Note down the `Value`, it will be our `clientSecret` configuration.
3. From the App Registration overview page, note down the `Application (client) ID` and the `Directory (tenant) ID`. 3. From the App Registration overview page, note down the `Application (client) ID` and the `Directory (tenant) ID`.
4. In your Key Vault overview page, note down the `Vault URI`.
5. Go to `Access Control (IAM)` and click on `Add Role Assignment`.
6. Give the permission `Key Vault Secrets Officer` to your App Registration.
### 2. Update configuration #### Managed Identity (recommnded)
We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials for our 1. In your Azure subscription create [Manged Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)
App Registration. 2. Use this created identity - for AKS users this means you need to use [Pod Identity](https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity) or [Workload Identity (recommnded)](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet).
{% note %}
Note that the using Managed Identity require using [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential).
{% /note %}
### 2. Add RBAC roles
It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - `Key Vault Secrets Officer` - to be able to create/read/update secrets in the Vault.
While the Airflow part only needs to read the secrets hence the role `Key Vault Secrets Officer`.
#### Open Metadata server
1. In your Key Vault overview page, note down the `Vault URI`.
2. Go to `Access Control (IAM)` and click on `Add Role Assignment`.
3. Give the permission `Key Vault Secrets Officer` to your Principal.
#### Airflow
1. In your Key Vault overview page, note down the `Vault URI`.
2. Go to `Access Control (IAM)` and click on `Add Role Assignment`.
3. Give the permission `Key Vault Secrets Users` to your Principal.
### 3. Update configuration
We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials.
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
#### Default Azure Credential
```yaml ```yaml
... ---
secretsManagerConfiguration: secretsManagerConfiguration:
secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'. secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key> prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]` tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
parameters: parameters:
enabled: true
vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
...
```
#### Client Secret Credential
```yaml
---
secretsManagerConfiguration:
secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
parameters:
enabled: true
clientId: ${OM_SM_CLIENT_ID:-""} clientId: ${OM_SM_CLIENT_ID:-""}
clientSecret: ${OM_SM_CLIENT_SECRET:-""} clientSecret: ${OM_SM_CLIENT_SECRET:-""}
tenantId: ${OM_SM_TENANT_ID:-""} tenantId: ${OM_SM_TENANT_ID:-""}
vaultName: ${OM_SM_VAULT_NAME:-""} vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration: pipelineServiceClientConfiguration:
# ... secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
# Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env ```
# Supported: noop, airflow, env
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"} For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
... ...
``` ```
The changes to be done in `airflow.yaml` file of the Airflow are:
{% note %} {% note %}
Note that the **Key Vault Name** parameter is MANDATORY for the system to know where to store and retrieve the secrets. Note that the **Key Vault Name** parameter is MANDATORY for the system to know where to store and retrieve the secrets.
@ -75,6 +139,30 @@ AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= <App Registration Se
If only the `<Key Vault Name>`, parameter is provided, we will use Azure's [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential). If only the `<Key Vault Name>`, parameter is provided, we will use Azure's [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential).
{% note %}
Also if you are using [Microsoft Entra Workload ID](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) with [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) then you need also to use projected service account instead one created by Airflow and OpenMetadata:
airflow.yaml:
```yaml
---
serviceAccount:
create: false
name: "name-of-your-service-account"
```
openmetadata.yaml:
```yaml
---
serviceAccount:
create: false
name: "name-of-your-service-account"
```
{% /note %}
### 3. Restart both servers ### 3. Restart both servers
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will After updating the configuration files, we are ready to restart both services. When the OM server starts, it will
@ -118,7 +206,6 @@ Key Vault service to point to.
You can specify as well the environment variables of your App Registration if you're running the ingestion You can specify as well the environment variables of your App Registration if you're running the ingestion
outside of Azure: [docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python). outside of Azure: [docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python).
## Airflow ## Airflow
If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure