yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-31 12:39:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(docs): Update Helm Values; Azure Key Vault Docs for 1.4.X (#16014)

Browse Source 
* chore(docs): Update Helm Values; Azure Key Vault Docs for 1.4.X

* fix: use bare-metal config values and update example

* docs: Add note on PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER usage with Helm Values
This commit is contained in:
Akash Jain 2024-04-24 15:53:58 +05:30 committed by GitHub
parent cd73328f87
commit f90f1301d9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 203 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
openmetadata-docs/content/v1.3.x/deployment/kubernetes/values.md
View File

@ -14,6 +14,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| Key | Type | Default | Environment Variable from openmetadata.yaml |
|-----|------|---------| ---------------------- |
| openmetadata.config.authentication.enabled | bool | `true` | |
| openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION_CLIENT_TYPE |
| openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER |
| openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS |
| openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY |
@ -50,6 +51,24 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES |
| openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC_CALLBACK |
| openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC_CLIENT_AUTH_METHOD |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC_CUSTOM_PARAMS |
| openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC_DISABLE_PKCE |
| openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC_DISCOVERY_URI |
| openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | |
| openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC_MAX_CLOCK_SKEW |
| openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC_TYPE |
| openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC_PREFERRED_JWS |
| openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC_RESPONSE_TYPE |
| openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC_SCOPE |
| openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC_SERVER_URL |
| openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC_TENANT |
| openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC_USE_NONCE |
| openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE |
| openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID |
| openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL |
@ -88,6 +107,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.airflow.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW_PASSWORD |
| openmetadata.config.airflow.auth.username | string | `admin` | AIRFLOW_USERNAME |
| openmetadata.config.airflow.enabled | bool | `true` | |
| openmetadata.config.airflow.host | string | `http://openmetadata-dependencies-web:8080` | PIPELINE_SERVICE_CLIENT_ENDPOINT |
| openmetadata.config.airflow.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER_HOST_API_URL |
| openmetadata.config.airflow.sslCertificatePath | string | `/no/path` | PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH |
| openmetadata.config.airflow.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
@ -151,12 +171,20 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
| openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP |
| openmetadata.config.secretsManager.enabled | bool | `true` | |
| openmetadata.config.secretsManager.provider | string | `db` | SECRET_MANAGER |
| openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET_MANAGER |
| openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX |
| openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS |
| openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY |
@ -170,8 +198,8 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL |
| openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY |
| openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME |
| openmetadata.config.upgradeMigrationConfigs.force | bool | `false` | |
| openmetadata.config.upgradeMigrationConfigs.migrationLimitParam | int | `1200` | MIGRATION_LIMIT_PARAM |
| openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | |
| openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | |
| openmetadata.config.web.enabled | bool | `true` | |
| openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED |
| openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED |
@ -210,7 +238,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| fullnameOverride | string | `"openmetadata"` |
| image.pullPolicy | string | `"Always"` |
| image.repository | string | `"docker.getcollate.io/openmetadata/server"` |
| image.tag | string | `1.3.1` |
| image.tag | string | `1.3.3` |
| imagePullSecrets | list | `[]` |
| ingress.annotations | object | `{}` |
| ingress.className | string | `""` |

28
openmetadata-docs/content/v1.3.x/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md
View File

@ -64,9 +64,19 @@ secretsManagerConfiguration:
parameters:
enabled: true
vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
...
```
#### Client Secret Credential
@ -83,9 +93,19 @@ secretsManagerConfiguration:
clientSecret: ${OM_SM_CLIENT_SECRET:-""}
tenantId: ${OM_SM_TENANT_ID:-""}
vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
...
```
The changes to be done in `airflow.yaml` file of the Airflow are:

35
openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/kubernetes/values.md
View File

@ -14,6 +14,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| Key | Type | Default | Environment Variable from openmetadata.yaml |
|-----|------|---------| ---------------------- |
| openmetadata.config.authentication.enabled | bool | `true` | |
| openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION_CLIENT_TYPE |
| openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION_PROVIDER |
| openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION_PUBLIC_KEYS |
| openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION_AUTHORITY |
@ -50,6 +51,24 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST |
| openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES |
| openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC_CALLBACK |
| openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC_CLIENT_AUTH_METHOD |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC_CLIENT_ID |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC_CLIENT_SECRET |
| openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC_CUSTOM_PARAMS |
| openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC_DISABLE_PKCE |
| openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC_DISCOVERY_URI |
| openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | |
| openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC_MAX_CLOCK_SKEW |
| openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC_TYPE |
| openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC_PREFERRED_JWS |
| openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC_RESPONSE_TYPE |
| openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC_SCOPE |
| openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC_SERVER_URL |
| openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC_TENANT |
| openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC_USE_NONCE |
| openmetadata.config.authentication.saml.debugMode | bool | false | SAML_DEBUG_MODE |
| openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML_IDP_ENTITY_ID |
| openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML_IDP_SSO_LOGIN_URL |
@ -152,12 +171,20 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE_SERVICE_CLIENT_VERIFY_SSL |
| openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE_SERVICE_CLIENT_HOST_IP |
| openmetadata.config.secretsManager.enabled | bool | `true` | |
| openmetadata.config.secretsManager.provider | string | `db` | SECRET_MANAGER |
| openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET_MANAGER |
| openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET_MANAGER_PREFIX |
| openmetadata.config.secretsManager.tags | list | `[]` | SECRET_MANAGER_TAGS |
| openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_ID |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM_SM_CLIENT_SECRET |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM_SM_TENANT_ID |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM_SM_VAULT_NAME |
| openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM_SM_REGION |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM_SM_ACCESS_KEY |
| openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM_SM_ACCESS_KEY |
@ -171,8 +198,8 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM_SUPPORT_URL |
| openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP_SERVER_STRATEGY |
| openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP_SERVER_USERNAME |
| openmetadata.config.upgradeMigrationConfigs.force | bool | `false` | |
| openmetadata.config.upgradeMigrationConfigs.migrationLimitParam | int | `1200` | MIGRATION_LIMIT_PARAM |
| openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | |
| openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | |
| openmetadata.config.web.enabled | bool | `true` | |
| openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED |
| openmetadata.config.web.csp.enabled | bool | `false` | WEB_CONF_XSS_CSP_ENABLED |
@ -211,7 +238,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
| fullnameOverride | string | `"openmetadata"` |
| image.pullPolicy | string | `"Always"` |
| image.repository | string | `"docker.getcollate.io/openmetadata/server"` |
| image.tag | string | `1.3.1` |
| image.tag | string | `1.3.3` |
| imagePullSecrets | list | `[]` |
| ingress.annotations | object | `{}` |
| ingress.className | string | `""` |

145
openmetadata-docs/content/v1.4.x-SNAPSHOT/deployment/secrets-manager/supported-implementations/azure-key-vault/index.md
View File

@ -10,42 +10,106 @@ for the non-managed follow only the steps related to the Airflow server and CLI.
## Setup
### 1. Permissions needed
### 1. Create Principal
#### Service Principal
1. Go to `Microsoft Entra ID` and create an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app).
2. Inside the App Registration, go to `Certificates & Secrets` and create a `Client secret`. Note down the `Value`, it will be our `clientSecret` configuration.
3. From the App Registration overview page, note down the `Application (client) ID` and the `Directory (tenant) ID`.
4. In your Key Vault overview page, note down the `Vault URI`.
5. Go to `Access Control (IAM)` and click on `Add Role Assignment`.
6. Give the permission `Key Vault Secrets Officer` to your App Registration.
### 2. Update configuration
#### Managed Identity (recommnded)
We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials for our
App Registration.
1. In your Azure subscription create [Manged Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)
2. Use this created identity - for AKS users this means you need to use [Pod Identity](https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity) or [Workload Identity (recommnded)](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet).
{% note %}
Note that the using Managed Identity require using [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential).
{% /note %}
### 2. Add RBAC roles
It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - `Key Vault Secrets Officer` - to be able to create/read/update secrets in the Vault.
While the Airflow part only needs to read the secrets hence the role `Key Vault Secrets Officer`.
#### Open Metadata server
1. In your Key Vault overview page, note down the `Vault URI`.
2. Go to `Access Control (IAM)` and click on `Add Role Assignment`.
3. Give the permission `Key Vault Secrets Officer` to your Principal.
#### Airflow
1. In your Key Vault overview page, note down the `Vault URI`.
2. Go to `Access Control (IAM)` and click on `Add Role Assignment`.
3. Give the permission `Key Vault Secrets Users` to your Principal.
### 3. Update configuration
We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials.
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
```yaml
...
secretsManagerConfiguration:
secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
parameters:
clientId: ${OM_SM_CLIENT_ID:-""}
clientSecret: ${OM_SM_CLIENT_SECRET:-""}
tenantId: ${OM_SM_TENANT_ID:-""}
vaultName: ${OM_SM_VAULT_NAME:-""}
#### Default Azure Credential
```yaml
---
secretsManagerConfiguration:
secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
parameters:
enabled: true
vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
# ...
# Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env
# Supported: noop, airflow, env
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"}
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
...
```
#### Client Secret Credential
```yaml
---
secretsManagerConfiguration:
secretsManager: managed-azure-kv # or env var SECRET_MANAGER. For non-managed use 'azure-kv'.
prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as <prefix>-<clusterName>-<key>
tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
parameters:
enabled: true
clientId: ${OM_SM_CLIENT_ID:-""}
clientSecret: ${OM_SM_CLIENT_SECRET:-""}
tenantId: ${OM_SM_TENANT_ID:-""}
vaultName: ${OM_SM_VAULT_NAME:-""}
pipelineServiceClientConfiguration:
secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-airflow}
```
For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below -
```yaml
---
...
extraEnvs:
- name: PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER
value: airflow
...
```
The changes to be done in `airflow.yaml` file of the Airflow are:
{% note %}
Note that the **Key Vault Name** parameter is MANDATORY for the system to know where to store and retrieve the secrets.
@ -75,16 +139,40 @@ AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_CLIENT_SECRET= <App Registration Se
If only the `<Key Vault Name>`, parameter is provided, we will use Azure's [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential).
{% note %}
Also if you are using [Microsoft Entra Workload ID](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) with [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) then you need also to use projected service account instead one created by Airflow and OpenMetadata:
airflow.yaml:
```yaml
---
serviceAccount:
create: false
name: "name-of-your-service-account"
```
openmetadata.yaml:
```yaml
---
serviceAccount:
create: false
name: "name-of-your-service-account"
```
{% /note %}
### 3. Restart both servers
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will
automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will
automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it
from our DB.
If everything goes as planned, all the data would be displayed using the parameters names which starts with
If everything goes as planned, all the data would be displayed using the parameters names which starts with
`openmetadata-...` in your Key Vault console.
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have
to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be
added before the `clusterName`, i.e., `<prefix>-<clusterName>-<key>`
@ -93,7 +181,7 @@ to the created secret.
## CLI
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the
`workflowConfig` we have to add the secret manager configuration:
```yaml
@ -106,7 +194,7 @@ workflowConfig:
```
Make sure to follow the steps [here](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential) to allow
the Python client to authenticate to Azure.
the Python client to authenticate to Azure.
{% note %}
@ -118,7 +206,6 @@ Key Vault service to point to.
You can specify as well the environment variables of your App Registration if you're running the ingestion
outside of Azure: [docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python).
## Airflow
If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure