mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-08-22 16:08:13 +00:00
fix(docs): kubernetes docs for breaking changes for 1.1.0 (#12215)
* update global.* to openmetadata.config.* * update values * fix typo * Add more docs * elastic search application version related docs update
This commit is contained in:
Akash Jain
GitHub
parent
3e879136c0
commit
72bb254342
@ -160,16 +160,17 @@ If using Kubernetes, make sure that you are passing the correct values to Helm C
|
||||
|
||||
```yaml
|
||||
# Custom OpenMetadata Values.yaml
|
||||
global:
|
||||
airflow:
|
||||
enabled: true
|
||||
# endpoint url for airflow
|
||||
host: http://openmetadata-dependencies-web.default.svc.cluster.local:8080
|
||||
auth:
|
||||
username: admin
|
||||
password:
|
||||
secretRef: airflow-secrets
|
||||
secretKey: openmetadata-airflow-password
|
||||
openmetadata:
|
||||
config:
|
||||
pipelineServiceClientConfig:
|
||||
enabled: true
|
||||
# endpoint url for airflow
|
||||
apiEndpoint: http://openmetadata-dependencies-web.default.svc.cluster.local:8080
|
||||
auth:
|
||||
username: admin
|
||||
password:
|
||||
secretRef: airflow-secrets
|
||||
secretKey: openmetadata-airflow-password
|
||||
```
|
||||
|
||||
#### Validating the installation
|
||||
|
||||
@ -239,34 +239,35 @@ Once you have the RDS and OpenSearch Services Setup, you can update the environm
|
||||
```yaml
|
||||
# openmetadata-values.prod.yaml
|
||||
...
|
||||
global:
|
||||
elasticsearch:
|
||||
host: <AMAZON_OPENSEARCH_SERVICE_ENDPOINT_WITHOUT_HTTPS>
|
||||
port: 443
|
||||
scheme: https
|
||||
connectionTimeoutSecs: 5
|
||||
socketTimeoutSecs: 60
|
||||
keepAliveTimeoutSecs: 600
|
||||
batchSize: 10
|
||||
auth:
|
||||
enabled: false
|
||||
username: <AMAZON_OPENSEARCH_USERNAME>
|
||||
password:
|
||||
secretRef: elasticsearch-secrets
|
||||
secretKey: openmetadata-elasticsearch-password
|
||||
database:
|
||||
host: <AMAZON_RDS_ENDPOINT>
|
||||
port: 3306
|
||||
driverClass: com.mysql.cj.jdbc.Driver
|
||||
dbScheme: mysql
|
||||
dbUseSSL: true
|
||||
databaseName: <RDS_DATABASE_NAME>
|
||||
auth:
|
||||
username: <RDS_DATABASE_USERNAME>
|
||||
password:
|
||||
secretRef: mysql-secrets
|
||||
secretKey: openmetadata-mysql-password
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
elasticsearch:
|
||||
host: <AMAZON_OPENSEARCH_SERVICE_ENDPOINT_WITHOUT_HTTPS>
|
||||
port: 443
|
||||
scheme: https
|
||||
connectionTimeoutSecs: 5
|
||||
socketTimeoutSecs: 60
|
||||
keepAliveTimeoutSecs: 600
|
||||
batchSize: 10
|
||||
auth:
|
||||
enabled: false
|
||||
username: <AMAZON_OPENSEARCH_USERNAME>
|
||||
password:
|
||||
secretRef: elasticsearch-secrets
|
||||
secretKey: openmetadata-elasticsearch-password
|
||||
database:
|
||||
host: <AMAZON_RDS_ENDPOINT>
|
||||
port: 3306
|
||||
driverClass: com.mysql.cj.jdbc.Driver
|
||||
dbScheme: mysql
|
||||
dbUseSSL: true
|
||||
databaseName: <RDS_DATABASE_NAME>
|
||||
auth:
|
||||
username: <RDS_DATABASE_USERNAME>
|
||||
password:
|
||||
secretRef: mysql-secrets
|
||||
secretKey: openmetadata-mysql-password
|
||||
...
|
||||
```
|
||||
|
||||
Make sure to create RDS and OpenSearch credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/deployment/kubernetes#quickstart).
|
||||
|
||||
@ -98,7 +98,9 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
| openmetadata.config.elasticsearch.auth.password.secretRef | string | `elasticsearch-secrets` | ELASTICSEARCH_PASSWORD |
|
||||
| openmetadata.config.elasticsearch.auth.password.secretKey | string | `openmetadata-elasticsearch-password` | ELASTICSEARCH_PASSWORD |
|
||||
| openmetadata.config.elasticsearch.host | string | `elasticsearch` | ELASTICSEARCH_HOST |
|
||||
| openmetadata.config.elasticsearch.keepAliveTimeoutSecs | int | `600` | ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS |
|
||||
| openmetadata.config.elasticsearch.port | int | 9200 | ELASTICSEARCH_PORT |
|
||||
| openmetadata.config.elasticsearch.searchType | string | `ElasticSearch` | SEARCH_TYPE |
|
||||
| openmetadata.config.elasticsearch.scheme | string | `http` | ELASTICSEARCH_SCHEME |
|
||||
| openmetadata.config.elasticsearch.searchIndexMappingLanguage | string | `EN`| ELASTICSEARCH_INDEX_MAPPING_LANG |
|
||||
| openmetadata.config.elasticsearch.trustStore.enabled | bool | `false` | |
|
||||
@ -128,6 +130,7 @@ This page list all the supported helm values for OpenMetadata Helm Charts.
|
||||
| openmetadata.config.pipelineServiceClientConfig.apiEndpoint | string | `http://openmetadata-dependencies-web.default.svc.cluster.local:8080` | PIPELINE_SERVICE_CLIENT_ENDPOINT |
|
||||
| openmetadata.config.pipelineServiceClientConfig.className | string | `org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient` | PIPELINE_SERVICE_CLIENT_CLASS_NAME |
|
||||
| openmetadata.config.pipelineServiceClientConfig.enabled | bool | `true` | |
|
||||
| openmetadata.config.pipelineServiceClientConfig.healthCheckInterval | int | `300` | PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL |
|
||||
| openmetadata.config.pipelineServiceClientConfig.ingestionIpInfoEnabled | bool | `false` | PIPELINE_SERVICE_IP_INFO_ENABLED |
|
||||
| openmetadata.config.pipelineServiceClientConfig.metadataApiEndpoint | string | `http://openmetadata.default.svc.cluster.local:8585/api` | SERVER_HOST_API_URL |
|
||||
| openmetadata.config.pipelineServiceClientConfig.sslCertificatePath | string | `/no/path` | PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH |
|
||||
|
||||
@ -11,21 +11,22 @@ Once the `Client Id` and `Client Secret` are generated, see the snippet below fo
|
||||
place the client id value and update the authorizer configurations in the `values.yaml`.
|
||||
|
||||
```yaml
|
||||
global:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "admin"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "aws-cognito"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "{Cognito Domain}/{User Pool ID}/.well-known/jwks.json" # Update with your Cognito Domain and User Pool ID
|
||||
authority: "{Cognito Domain}/{User Pool ID}" # Update with your Cognito Domain and User Pool ID as follows - https://cognito-idp.us-west-1.amazonaws.com/us-west-1_DL8xfTzj8
|
||||
clientId: "{Client ID}" # Update with your Client ID
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "admin"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "aws-cognito"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "{Cognito Domain}/{User Pool ID}/.well-known/jwks.json" # Update with your Cognito Domain and User Pool ID
|
||||
authority: "{Cognito Domain}/{User Pool ID}" # Update with your Cognito Domain and User Pool ID as follows - https://cognito-idp.us-west-1.amazonaws.com/us-west-1_DL8xfTzj8
|
||||
clientId: "{Client ID}" # Update with your Client ID
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
Finally, update the Airflow information with the JWT token
|
||||
|
||||
@ -80,5 +80,26 @@ global:
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "suresh"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "auth0"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "{Auth0 Domain Name}/.well-known/jwks.json"
|
||||
authority: "https://parth-panchal.us.auth0.com/"
|
||||
clientId: "{Client ID}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for
|
||||
ingesting data from Airflow.
|
||||
|
||||
@ -71,6 +71,28 @@ global:
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
- "user2"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "azure"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "https://login.microsoftonline.com/common/discovery/keys"
|
||||
authority: "https://login.microsoftonline.com/{Tenant ID}"
|
||||
clientId: "{Client ID}" # Azure Application
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
{% note %}
|
||||
|
||||
Follow [this](/deployment/security/azure#step-10-update-ingestion-bot-with-azure-sso-service-application) guide to configure the `ingestion-bot` credentials for ingesting data from Airflow.
|
||||
|
||||
@ -203,14 +203,15 @@ Update your helm values to mount Kubernetes Secrets as Volumes and update the Jw
|
||||
|
||||
```yaml
|
||||
# openmetadata.prod.values.yml
|
||||
global:
|
||||
...
|
||||
jwtTokenConfiguration:
|
||||
rsapublicKeyFilePath: "/etc/openmetadata/jwtkeys/public_key.der"
|
||||
rsaprivateKeyFilePath: "/etc/openmetadata/jwtkeys/private_key.der"
|
||||
jwtissuer: "open-metadata.org" # update this as per your environment
|
||||
keyId: "c8ec220c-be7d-4e47-97c7-098bf6a57ce1" # update this to a unique uuid4
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
...
|
||||
jwtTokenConfiguration:
|
||||
rsapublicKeyFilePath: "/etc/openmetadata/jwtkeys/public_key.der"
|
||||
rsaprivateKeyFilePath: "/etc/openmetadata/jwtkeys/private_key.der"
|
||||
jwtissuer: "open-metadata.org" # update this as per your environment
|
||||
keyId: "c8ec220c-be7d-4e47-97c7-098bf6a57ce1" # update this to a unique uuid4
|
||||
...
|
||||
extraVolumes:
|
||||
- name: openmetadata-jwt-vol
|
||||
secret:
|
||||
|
||||
@ -59,8 +59,9 @@ Update your helm `maskPasswordsApi` value:
|
||||
|
||||
```yaml
|
||||
# openmetadata.prod.values.yml
|
||||
global:
|
||||
...
|
||||
maskPasswordsApi: true
|
||||
...
|
||||
openmetadata:
|
||||
config:
|
||||
...
|
||||
maskPasswordsApi: true
|
||||
...
|
||||
```
|
||||
@ -78,10 +78,11 @@ airflowConfiguration:
|
||||
We have to update in the `values.yaml` file with:
|
||||
|
||||
```yaml
|
||||
global:
|
||||
airflow:
|
||||
verifySsl: "validate"
|
||||
sslCertificatePath: "/path/to/certificate/in/airflow"
|
||||
openmetadata:
|
||||
config:
|
||||
pipelineServiceClientConfig:
|
||||
verifySsl: "validate"
|
||||
sslCertificatePath: "/path/to/certificate/in/airflow"
|
||||
```
|
||||
|
||||
#### 2. Ignore the certification validation (not recommended for production):
|
||||
@ -108,9 +109,10 @@ airflowConfiguration:
|
||||
We have to update in the `values.yaml` file with:
|
||||
|
||||
```yaml
|
||||
global:
|
||||
airflow:
|
||||
verifySsl: "ignore"
|
||||
openmetadata:
|
||||
config:
|
||||
pipelineServiceClientConfig:
|
||||
verifySsl: "ignore"
|
||||
```
|
||||
|
||||
Once one of the configurations is set, we can restart our OM server and deploy or redeploy without any issues.
|
||||
|
||||
@ -60,5 +60,28 @@ global:
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
- "user2"
|
||||
botPrincipals:
|
||||
- "<service_application_client_id>"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "google"
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
authority: "https://accounts.google.com"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for
|
||||
ingesting data from Airflow.
|
||||
@ -65,5 +65,26 @@ global:
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "admin-user"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "custom-oidc"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "http://localhost:8081/auth/realms/data-sec/protocol/openid-connect/certs"
|
||||
authority: "http://localhost:8081/auth/realms/data-sec"
|
||||
clientId: "{Client ID}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for
|
||||
ingesting data from Airflow.
|
||||
@ -87,5 +87,27 @@ global:
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
- "user2"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "okta"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "{ISSUER_URL}/v1/keys"
|
||||
authority: "{ISSUER_URL}"
|
||||
clientId: "{CLIENT_ID - SPA APP}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for
|
||||
ingesting data from Airflow.
|
||||
@ -59,5 +59,49 @@ global:
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
# JWT Filter
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "suresh"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "custom-oidc"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "{IssuerUrl}/certs"
|
||||
authority: "{IssuerUrl}"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
### After 1.1.0
|
||||
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
# JWT Filter
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "suresh"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "custom-oidc"
|
||||
publicKeys:
|
||||
- "http://openmetadata:8585/api/v1/config/jwks"
|
||||
- "{IssuerUrl}/certs"
|
||||
authority: "{IssuerUrl}"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for
|
||||
ingesting data from Airflow.
|
||||
@ -76,6 +76,66 @@ If you are upgrading production this is the recommended version to upgrade to.
|
||||
|
||||
## Breaking Changes for 1.1 Stable Release
|
||||
|
||||
### OpenMetadata Helm Chart Values
|
||||
|
||||
With `1.1.0` we are moving away from `global.*` helm values under openmetadata helm charts to `openmetadata.config.*`. This change is introduce as helm reserves global chart values across all the helm charts. This conflicted the use of OpenMetadata helm charts along with other helm charts for organizations using common helm values yaml files.
|
||||
|
||||
For example, with `1.0.X` Application version Releases, helm values would look like below -
|
||||
```yaml
|
||||
global:
|
||||
...
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
botPrincipals:
|
||||
- "<service_application_client_id>"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "google"
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
- "http://openmetadata:8585/api/v1/system/config/jwks"
|
||||
authority: "https://accounts.google.com"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
...
|
||||
```
|
||||
|
||||
With OpenMetadata Application version `1.1.0` and above, the above config will need to be updated as
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
- "user2"
|
||||
botPrincipals:
|
||||
- "<service_application_client_id>"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "google"
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
- "http://openmetadata:8585/api/v1/system/config/jwks"
|
||||
authority: "https://accounts.google.com"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
A quick and easy way to update the config is to use [yq](https://mikefarah.gitbook.io/yq/) utility to manipulate YAML files.
|
||||
|
||||
```bash
|
||||
yq -i -e '{"openmetadata": {"config": .global}}' openmetadata.values.yml
|
||||
```
|
||||
|
||||
The above command will update `global.*` with `openmetadata.config.*` yaml config. Please note, the above command is only recommended for users with custom helm values file explicit for OpenMetadata Helm Charts.
|
||||
|
||||
For more information, visit the official helm docs for [global chart values](https://helm.sh/docs/chart_template_guide/subcharts_and_globals/#global-chart-values).
|
||||
|
||||
### Pipeline Service Client Configuration
|
||||
|
||||
If reusing an old YAML configuration file, make sure to add the following inside `pipelineServiceClientConfiguration`:
|
||||
@ -224,3 +284,5 @@ impacts BigQuery, Datalake and any other source where you are directly passing t
|
||||
- Glue now supports custom database names via `databaseName`.
|
||||
- Snowflake supports the `clientSessionKeepAlive` parameter to keep the session open for long processes.
|
||||
- Kafka and Redpanda now have the `saslMechanism` based on enum values `["PLAIN", "GSSAPI", "SCRAM-SHA-256", "SCRAM-SHA-512", "OAUTHBEARER"]`.
|
||||
- OpenMetadata Server Docker Image now installs the OpenMetadata Libraries under `/opt/openmetadata` directory
|
||||
- Bumped up ElasticSearch version for Docker and Kubernetes OpenMetadata Dependencies Helm Chart to `7.16.3`
|
||||
@ -180,25 +180,27 @@ global:
|
||||
✅ After 1.0.0 Helm Chart Release, the `global.pipelineServiceClient` section will replace the above `airflow` section -
|
||||
|
||||
```yaml
|
||||
global:
|
||||
openmetadata:
|
||||
config:
|
||||
...
|
||||
pipelineServiceClientConfig:
|
||||
enabled: true
|
||||
className: "org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"
|
||||
# endpoint url for airflow
|
||||
apiEndpoint: http://openmetadata-dependencies-web.default.svc.cluster.local:8080
|
||||
# this will be the api endpoint url of OpenMetadata Server
|
||||
metadataApiEndpoint: http://openmetadata.default.svc.cluster.local:8585/api
|
||||
# possible values are "no-ssl", "ignore", "validate"
|
||||
verifySsl: "no-ssl"
|
||||
ingestionIpInfoEnabled: false
|
||||
# local path in Airflow Pod
|
||||
sslCertificatePath: "/no/path"
|
||||
auth:
|
||||
username: admin
|
||||
password:
|
||||
secretRef: airflow-secrets
|
||||
secretKey: openmetadata-airflow-password
|
||||
...
|
||||
pipelineServiceClientConfig:
|
||||
enabled: true
|
||||
className: "org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"
|
||||
# endpoint url for airflow
|
||||
apiEndpoint: http://openmetadata-dependencies-web.default.svc.cluster.local:8080
|
||||
# this will be the api endpoint url of OpenMetadata Server
|
||||
metadataApiEndpoint: http://openmetadata.default.svc.cluster.local:8585/api
|
||||
# possible values are "no-ssl", "ignore", "validate"
|
||||
verifySsl: "no-ssl"
|
||||
ingestionIpInfoEnabled: false
|
||||
# local path in Airflow Pod
|
||||
sslCertificatePath: "/no/path"
|
||||
auth:
|
||||
username: admin
|
||||
password:
|
||||
secretRef: airflow-secrets
|
||||
secretKey: openmetadata-airflow-password
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
@ -18,6 +18,66 @@ Upgrading from 1.0 to 1.1 can be done directly on your instances. This page will
|
||||
|
||||
## Breaking Changes for 1.1 Stable Release
|
||||
|
||||
### OpenMetadata Helm Chart Values
|
||||
|
||||
With `1.1.0` we are moving away from `global.*` helm values under openmetadata helm charts to `openmetadata.config.*`. This change is introduce as helm reserves global chart values across all the helm charts. This conflicted the use of OpenMetadata helm charts along with other helm charts for organizations using common helm values yaml files.
|
||||
|
||||
For example, with `1.0.X` Application version Releases, helm values would look like below -
|
||||
```yaml
|
||||
global:
|
||||
...
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
botPrincipals:
|
||||
- "<service_application_client_id>"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "google"
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
- "http://openmetadata:8585/api/v1/system/config/jwks"
|
||||
authority: "https://accounts.google.com"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
...
|
||||
```
|
||||
|
||||
With OpenMetadata Application version `1.1.0` and above, the above config will need to be updated as
|
||||
```yaml
|
||||
openmetadata:
|
||||
config:
|
||||
authorizer:
|
||||
className: "org.openmetadata.service.security.DefaultAuthorizer"
|
||||
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
|
||||
initialAdmins:
|
||||
- "user1"
|
||||
- "user2"
|
||||
botPrincipals:
|
||||
- "<service_application_client_id>"
|
||||
principalDomain: "open-metadata.org"
|
||||
authentication:
|
||||
provider: "google"
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
- "http://openmetadata:8585/api/v1/system/config/jwks"
|
||||
authority: "https://accounts.google.com"
|
||||
clientId: "{client id}"
|
||||
callbackUrl: "http://localhost:8585/callback"
|
||||
```
|
||||
|
||||
A quick and easy way to update the config is to use [yq](https://mikefarah.gitbook.io/yq/) utility to manipulate YAML files.
|
||||
|
||||
```bash
|
||||
yq -i -e '{"openmetadata": {"config": .global}}' openmetadata.values.yml
|
||||
```
|
||||
|
||||
The above command will update `global.*` with `openmetadata.config.*` yaml config. Please note, the above command is only recommended for users with custom helm values file explicit for OpenMetadata Helm Charts.
|
||||
|
||||
For more information, visit the official helm docs for [global chart values](https://helm.sh/docs/chart_template_guide/subcharts_and_globals/#global-chart-values).
|
||||
|
||||
### Pipeline Service Client Configuration
|
||||
|
||||
If reusing an old YAML configuration file, make sure to add the following inside `pipelineServiceClientConfiguration`:
|
||||
@ -174,3 +234,5 @@ impacts BigQuery, Datalake and any other source where you are directly passing t
|
||||
- Snowflake supports the `clientSessionKeepAlive` parameter to keep the session open for long processes.
|
||||
- Databricks now supports the `useUnityCatalog` parameter to extract the metadata from unity catalog instead of hive metastore.
|
||||
- Kafka and Redpanda now have the `saslMechanism` based on enum values `["PLAIN", "GSSAPI", "SCRAM-SHA-256", "SCRAM-SHA-512", "OAUTHBEARER"]`.
|
||||
- OpenMetadata Server Docker Image now installs the OpenMetadata Libraries under `/opt/openmetadata` directory
|
||||
- Bumped up ElasticSearch version for Docker and Kubernetes OpenMetadata Dependencies Helm Chart to `7.16.3`
|
||||
|
||||
@ -84,11 +84,12 @@ AUTHENTICATION_PUBLIC_KEYS=[https://www.googleapis.com/oauth2/v3/certs, http://l
|
||||
- In the case of **kubernetes**, you have to update `publicKeys` values:
|
||||
|
||||
```yaml
|
||||
global:
|
||||
authentication:
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
- "http://localhost:8585/api/v1/system/config/jwks"
|
||||
openmetadata:
|
||||
config:
|
||||
authentication:
|
||||
publicKeys:
|
||||
- "https://www.googleapis.com/oauth2/v3/certs"
|
||||
- "http://localhost:8585/api/v1/system/config/jwks"
|
||||
```
|
||||
|
||||
**3. Redeploying ingestion pipelines**
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user