diff --git a/openmetadata-docs/content/deployment/docker/index.md b/openmetadata-docs/content/deployment/docker/index.md index 341b0027936..0828e601290 100644 --- a/openmetadata-docs/content/deployment/docker/index.md +++ b/openmetadata-docs/content/deployment/docker/index.md @@ -15,9 +15,26 @@ for data persistence. Learn how to do so [here](/deployment/docker/volumes). To test out your security integration, check out how to [Enable Security](/deployment/docker/security). +## Changing ports + +This docker deployment is powered by `docker compose`, and uses the `docker-compose.yml` files shipped during +each release [example](https://github.com/open-metadata/OpenMetadata/releases/tag/0.11.4-release). + +As with the [Named Volumes](/deployment/docker/volumes), you might want to tune a bit the compose file to modify +the default ports. + +We are shipping the OpenMetadata server and UI at `8585`, and the ingestion container (Airflow) at `8080`. You can +take a look at the official Docker [docs](https://docs.docker.com/compose/compose-file/#ports). As an example, You could +update the ports to serve Airflow at `1234` with: + +```yaml +ports: + - "1234:8080" +``` + # Production Deployment -If instead, you are planning on going to PROD, we recommend the following +If you are planning on going to PROD, we also recommend taking a look at the following deployment strategies: diff --git a/openmetadata-docs/content/deployment/kubernetes/values.md b/openmetadata-docs/content/deployment/kubernetes/values.md index 8be7d8aa030..7ff8b71ca09 100644 --- a/openmetadata-docs/content/deployment/kubernetes/values.md +++ b/openmetadata-docs/content/deployment/kubernetes/values.md @@ -9,7 +9,6 @@ This page list all the supported helm values for OpenMetadata Helm Charts. ## Global Chart Values - | Key | Type | Default | | :---------- | :---------- | :---------- | @@ -75,7 +74,6 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | global.elasticsearch.trustStore.path | string | `Empty String` | | global.elasticsearch.trustStore.password.secretRef | string | `elasticsearch-truststore-secrets` | | global.elasticsearch.trustStore.password.secretKey | string | `openmetadata-elasticsearch-truststore-password` | -| global.fernetKey | string | `jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=` | | global.jwtTokenConfiguration.enabled | bool | `false` | | global.jwtTokenConfiguration.rsapublicKeyFilePath | string | `Empty String` | | global.jwtTokenConfiguration.rsaprivateKeyFilePath | string | `Empty String` | @@ -86,11 +84,9 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | global.openmetadata.host | string | `openmetadata` | | global.openmetadata.port | int | 8585 | -
## Chart Values - | Key | Type | Default | | :---------- | :---------- | :---------- | @@ -129,5 +125,3 @@ This page list all the supported helm values for OpenMetadata Helm Charts. | serviceAccount.name | string | `nil` | | sidecars | list | `[]` | | tolerations | list | `[]` | - -
\ No newline at end of file diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index 1d8cee2398d..5a8e1e36017 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -50,6 +50,8 @@ site_menu: url: /deployment/kubernetes/onprem - category: Deployment / Kubernetes Deployment / Enable Security url: /deployment/kubernetes/security + - category: Deployment / Kubernetes Deployment / Helm Values + url: /deployment/kubernetes/helm-values - category: Deployment / Enable Security url: /deployment/security diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/airflow.md index 63673e90576..0390cb6c799 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/looker/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/cli.md index d95d50bfe94..c27f610a81f 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/looker/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/looker/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/airflow.md index 6bdc146ac12..804bfbedc97 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/metabase/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/cli.md index bf172c50a57..509d08c90b0 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/metabase/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/metabase/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md index f6d1321f40c..c6a3b1983b0 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/powerbi/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md index 39e9778768d..0692cbb9ed1 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/powerbi/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/powerbi/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/airflow.md index 75de7708b03..60831273699 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/redash/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/cli.md index 7301ffb5602..233fe597d4d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/redash/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/redash/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/airflow.md index 62932cdafe2..6a3d45b9564 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/superset/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/cli.md index 816c5a8c2d1..17f5d4e75dd 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/superset/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/superset/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/airflow.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/airflow.md index d63df6da363..858b8e95b79 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/tableau/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/cli.md b/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/cli.md index 5edbfd21772..de86c2b5c85 100644 --- a/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/dashboard/tableau/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/dashboard/tableau/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/athena/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/athena/airflow.md index c3470b6f68d..a50508b1792 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/athena/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/athena/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/athena/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/athena/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/athena/cli.md index cff25392f15..05cdb6a15ca 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/athena/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/athena/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/athena/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md index 602b7e9a05e..62c32acc812 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/azuresql/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/azuresql/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/azuresql/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/azuresql/cli.md index 7cb954091af..188372b513c 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/azuresql/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/azuresql/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/azuresql/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md index 9696b4a0f51..4639d271687 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/airflow.md @@ -7,6 +7,14 @@ slug: /openmetadata/connectors/database/bigquery/airflow + + +If you want to run the Usage Connector, you'll also need to install: + +```bash +pip3 install "openmetadata-ingestion[bigquery-usage]" +``` +

GCP Permissions

To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md index 6ceb6608c08..cee57805857 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/bigquery/cli.md @@ -7,6 +7,14 @@ slug: /openmetadata/connectors/database/bigquery/cli + + +If you want to run the Usage Connector, you'll also need to install: + +```bash +pip3 install "openmetadata-ingestion[bigquery-usage]" +``` +

GCP Permissions

To execute metadata extraction and usage workflow successfully the user or the service account should have enough access to fetch required data. Following table describes the minimum required permissions

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/airflow.md index 45ee91e59c9..f7a83f0dc06 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/clickhouse/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/cli.md index 02e1b62d811..6364e2ae7e5 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/clickhouse/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/clickhouse/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/databricks/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/databricks/airflow.md index 340ac5f70c4..3fbaf5a3abf 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/databricks/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/databricks/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/databricks/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/databricks/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/databricks/cli.md index c6f17c8d417..6ab37f0ed70 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/databricks/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/databricks/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/databricks/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/datalake/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/datalake/airflow.md index 9744c7c13fa..5dad6a49b4b 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/datalake/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/datalake/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/datalake/airflow + + ## Metadata Ingestion All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Datalake. @@ -59,8 +61,6 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada * **schemaFilterPattern** and **tableFilternPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., - - This is a sample config for Datalake using GCS: ```yaml @@ -103,13 +103,13 @@ workflowConfig: ``` -

Source Configuration - Service Connection using GCS

+#### Source Configuration - Service Connection using GCS The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json). * **type**: Credentials type, e.g. `service_account`. * **projectId** -* **privat**eKe**y** +* **privateKey** * **privateKeyId** * **clientEmail** * **clientId** @@ -117,8 +117,8 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada * **tokenUri**: [https://oauth2.googleapis.com/token](https://oauth2.googleapis.com/token) by default * **authProviderX509CertUrl**: [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default * **clientX509CertUrl** -* **bucketName :** name of the bucket in GCS -* **Prefix** : prefix in gcs bucket +* **bucketName**: name of the bucket in GCS +* **Prefix**: prefix in gcs bucket * **schemaFilterPattern** and **tableFilternPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/datalake/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/datalake/cli.md index f11350ab33d..650f1958d2e 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/datalake/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/datalake/cli.md @@ -7,6 +7,7 @@ slug: /openmetadata/connectors/database/datalake/cli + ## Metadata Ingestion All connectors are defined as JSON Schemas. Here you can find the structure to create a connection to Datalake. @@ -45,11 +46,9 @@ workflowConfig: openMetadataServerConfig: hostPort: http://localhost:8585/api authProvider: no-auth - ``` - #### Source Configuration - Source Config using AWS S3 The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json). @@ -59,9 +58,6 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada * **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile. * **schemaFilterPattern** and **tableFilternPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., - - - This is a sample config for Datalake using GCS: ```yaml @@ -104,13 +100,13 @@ workflowConfig: ``` -

Source Configuration - Service Connection using GCS

+#### Source Configuration - Service Connection using GCS The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json). * **type**: Credentials type, e.g. `service_account`. * **projectId** -* **privat**eKe**y** +* **privateKey** * **privateKeyId** * **clientEmail** * **clientId** @@ -118,8 +114,8 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada * **tokenUri**: [https://oauth2.googleapis.com/token](https://oauth2.googleapis.com/token) by default * **authProviderX509CertUrl**: [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default * **clientX509CertUrl** -* **bucketName :** name of the bucket in GCS -* **Prefix** : prefix in gcs bucket +* **bucketName**: name of the bucket in GCS +* **Prefix**: prefix in gcs bucket * **schemaFilterPattern** and **tableFilternPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., diff --git a/openmetadata-docs/content/openmetadata/connectors/database/datalake/index.md b/openmetadata-docs/content/openmetadata/connectors/database/datalake/index.md index 26043a19305..23c970c2ff9 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/datalake/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/datalake/index.md @@ -3,59 +3,18 @@ title: Datalake slug: /openmetadata/connectors/database/datalake --- - + + -## Metadata Ingestion +

Connection Options

-### 1. Visit the Services Page - - -The first step is ingesting the metadata from your sources. Under Settings you will find a **Services** link an external source system to OpenMetadata. Once a service is created, it can be used to configure metadata, usage, and profiler workflows. - -To visit the _Services_ page, select _Services_ from the _Settings_ menu. - - -![Navigate to Settings >> Services]() - - - -### 2. Create a New Service - -Click on the _Add New Service_ button to start the Service creation. - - -create-new-service - -### 3. Select the Service Type - -Select Datalake as the service type and click _Next_. - -select-service - -### 4. Name and Describe your Service - -Provide a name and description for your service as illustrated below. - -describe-service - -#### Service Name - -OpenMetadata uniquely identifies services by their _Service Name_. Provide a name that distinguishes your deployment from other services, including the other Datalake services that you might be ingesting metadata from. - -### 5. Configure the Service Connection - -In this step, we will configure the connection settings required for this connector. Please follow the instructions below to ensure that you've configured the connector to read from your Datalake service as desired. - -**Datalake using AWS S3** + create-account -
- -Connection Options for AWS S3 **AWS Access Key ID** @@ -91,25 +50,12 @@ Enter the details for any additional connection options that can be sent to Dyna Enter the details for any additional connection arguments such as security or protocol configs that can be sent to DynamoDB during the connection. These details must be added as Key-Value pairs. -In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows. + -`"authenticator" : "sso_login_url"` - -In case you authenticate with SSO using an external browser popup, then add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows. - -`"authenticator" : "externalbrowser"` - -
- -**Datalake using GCS** + service-connection-using-gcs - -
- -Connection Options for GCS - **BUCKET NAME** This is the Bucket Name in GCS. @@ -120,7 +66,7 @@ This is the Bucket Name in GCS. **GCS Credentials** -We support two ways of authenticating to BigQuery: +We support two ways of authenticating to GCS: 1. Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery: 1. Credentials type, e.g. `service_account`. @@ -134,126 +80,7 @@ We support two ways of authenticating to BigQuery: 9. Authentication Provider X509 Certificate URL, [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default 10. Client X509 Certificate URL -
- -After hitting Save you will see that your Datalake connector has been added successfully, and you can add an ingestion. - -created-service - -### 6. Configure the Metadata Ingestion - -Once the service is created, we can add a **Metadata Ingestion Workflow**, either directly from the _Add Ingestion_ button in the figure above, or from the Service page: - -service-page - -
- -Metadata Ingestion Options - -**Include (Table Filter Pattern)** - -Use to table filter patterns to control whether or not to include tables as part of metadata ingestion and data profiling. - -Explicitly include tables by adding a list of comma-separated regular expressions to the _Include_ field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded. See the figure above for an example. - -**Exclude (Table Filter Pattern)** - -Explicitly exclude tables by adding a list of comma-separated regular expressions to the _Exclude_ field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included. See the figure above for an example. - -**Include (Schema Filter Pattern)** - -Use to schema filter patterns to control whether or not to include schemas as part of metadata ingestion and data profiling. - -Explicitly include schemas by adding a list of comma-separated regular expressions to the _Include_ field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded. - -**Exclude (Schema Filter Pattern)** - -Explicitly exclude schemas by adding a list of comma-separated regular expressions to the _Exclude_ field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included. - -**Include views (toggle)** - -Set the _Include views_ toggle to the on position to control whether or not to include views as part of metadata ingestion and data profiling. - -Explicitly include views by adding the following key-value pair in the `source.config` field of your configuration file. - -**Enable data profiler (toggle)** - -Glue does not provide querying capabilities, so the data profiler is not supported. - -**Ingest sample data (toggle)** - -Glue does not provide querying capabilities, so sample data is not supported. - -
- - -deploy-service - -### 7. Schedule the Ingestion and Deploy - -Scheduling can be set up at an hourly, daily, or weekly cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date. - -Review your configuration settings. If they match what you intended, click _Deploy_ to create the service and schedule metadata ingestion. - -If something doesn't look right, click the _Back_ button to return to the appropriate step and change the settings as needed. - -
- -Scheduling Options - -**Every** - -Use the _Every_ drop down menu to select the interval at which you want to ingest metadata. Your options are as follows: - -* _Hour_: Ingest metadata once per hour -* _Day_: Ingest metadata once per day -* _Week_: Ingest metadata once per week - -**Day** - -The _Day_ selector is only active when ingesting metadata once per week. Use the _Day_ selector to set the day of the week on which to ingest metadata. - -**Minute** - -The _Minute_ dropdown is only active when ingesting metadata once per hour. Use the _Minute_ drop down menu to select the minute of the hour at which to begin ingesting metadata. - -**Time** - -The _Time_ drop down menus are active when ingesting metadata either once per day or once per week. Use the time drop downs to select the time of day at which to begin ingesting metadata. - -**Start date (UTC)** - -Use the _Start date_ selector to choose the date at which to begin ingesting metadata according to the defined schedule. - -**End date (UTC)** - -Use the _End date_ selector to choose the date at which to stop ingesting metadata according to the defined schedule. If no end date is set, metadata ingestion will continue according to the defined schedule indefinitely. - -
- -After configuring the workflow, you can click on _Deploy_ to create the pipeline. - -schedule-options - - - -### 8. View the Ingestion Pipeline - -Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page. - -ingestion-pipeline - -### 9. Workflow Deployment Error - -If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. - -You can then edit the Ingestion Pipeline and _Deploy_ it again. - -create-account - -From the _Connection_ tab, you can also _Edit_ the Service if needed. - - +
diff --git a/openmetadata-docs/content/openmetadata/connectors/database/db2/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/db2/airflow.md index 3a55bd3ef6e..13291ec1822 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/db2/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/db2/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/db2/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/db2/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/db2/cli.md index f7cfa37b28f..42eb89d8a70 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/db2/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/db2/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/db2/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md index 9f22549072e..013a5bf0cfc 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/deltalake/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md index 5d97dda7a70..33b7d57dafe 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/deltalake/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/deltalake/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/druid/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/druid/airflow.md index 12757fc02fe..b31204823cd 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/druid/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/druid/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/druid/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/druid/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/druid/cli.md index 18a0a97ea1f..71d11f6099f 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/druid/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/druid/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/druid/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/airflow.md index 1277eca674d..1c9cc9c823d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/dynamodb/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/cli.md index ca7deed5c86..65b11f13506 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/dynamodb/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/dynamodb/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/glue/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/glue/airflow.md index 06b5c0bbcc6..c0aa0895f27 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/glue/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/glue/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/glue/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/glue/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/glue/cli.md index 8b77afa95a4..21066be6ce2 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/glue/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/glue/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/glue/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/hive/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/hive/airflow.md index 2ba8f8c0158..a052db2dddd 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/hive/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/hive/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/hive/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/hive/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/hive/cli.md index c149b6992f1..1fa95b18699 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/hive/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/hive/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/hive/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md index f8903ef2a73..e3d92566159 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/mariadb/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md index fdf62233dd8..6d567be63d8 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mariadb/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/mariadb/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mssql/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/mssql/airflow.md index fe3c5671a8b..fa437cfd9d5 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mssql/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mssql/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/mssql/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mssql/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/mssql/cli.md index 76b4c9ec28b..b8b266a77a3 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mssql/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mssql/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/mssql/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mysql/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/mysql/airflow.md index e63b931b044..2c751d6762f 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mysql/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mysql/airflow.md @@ -7,6 +7,10 @@ slug: /openmetadata/connectors/database/mysql/airflow + + +Note that the user should have access to the `INFORMATION_SCHEMA` table. +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mysql/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/mysql/cli.md index a796312917c..9b15ac9542e 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mysql/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mysql/cli.md @@ -7,6 +7,10 @@ slug: /openmetadata/connectors/database/mysql/cli + + +Note that the user should have access to the `INFORMATION_SCHEMA` table. +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/mysql/index.md b/openmetadata-docs/content/openmetadata/connectors/database/mysql/index.md index 93e0b640dc8..2ec836aea5e 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/mysql/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/mysql/index.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/mysql +Note that the user should have access to the `INFORMATION_SCHEMA` table. +

Connection Options

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/oracle/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/oracle/airflow.md index 823e909ca8b..810079aec11 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/oracle/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/oracle/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/oracle/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/oracle/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/oracle/cli.md index bd5044b07ea..b7cbff7fe32 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/oracle/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/oracle/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/oracle/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/postgres/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/postgres/airflow.md index c2dbd7f548c..5d385dc0be2 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/postgres/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/postgres/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/postgres/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/postgres/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/postgres/cli.md index 2815057c507..1bac797c4e0 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/postgres/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/postgres/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/postgres/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/presto/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/presto/airflow.md index f3315c56050..1939a4248a5 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/presto/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/presto/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/presto/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/presto/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/presto/cli.md index 945f5ea7abf..c8321d6d490 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/presto/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/presto/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/presto/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/redshift/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/redshift/airflow.md index 2e5010a8dab..eccd1907406 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/redshift/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/redshift/airflow.md @@ -7,6 +7,14 @@ slug: /openmetadata/connectors/database/redshift/airflow + + +If you want to run the Usage Connector, you'll also need to install: + +```bash +pip3 install "openmetadata-ingestion[redshift-usage]" +``` +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/redshift/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/redshift/cli.md index d2e7be4a5ee..b8c535da1d7 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/redshift/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/redshift/cli.md @@ -7,6 +7,14 @@ slug: /openmetadata/connectors/database/redshift/cli + + +If you want to run the Usage Connector, you'll also need to install: + +```bash +pip3 install "openmetadata-ingestion[redshift-usage]" +``` +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/salesforce/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/salesforce/airflow.md index f8cd1dbb4c9..a2b7db3f7aa 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/salesforce/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/salesforce/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/salesforce/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/salesforce/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/salesforce/cli.md index b9f4f9c63cf..826682b9e7d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/salesforce/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/salesforce/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/salesforce/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/singlestore/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/singlestore/airflow.md index b97a9d72ef6..01a7a3365f4 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/singlestore/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/singlestore/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/singlestore/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/singlestore/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/singlestore/cli.md index 73d61350de2..77a7de0857d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/singlestore/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/singlestore/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/singlestore/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/snowflake/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/snowflake/airflow.md index eca2173f4ed..87a657feaa7 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/snowflake/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/snowflake/airflow.md @@ -7,6 +7,14 @@ slug: /openmetadata/connectors/database/snowflake/airflow + + +If you want to run the Usage Connector, you'll also need to install: + +```bash +pip3 install "openmetadata-ingestion[snowflake-usage]" +``` +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/snowflake/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/snowflake/cli.md index 7bafadf09f8..4d33a2ee45a 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/snowflake/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/snowflake/cli.md @@ -7,6 +7,14 @@ slug: /openmetadata/connectors/database/snowflake/cli + + +If you want to run the Usage Connector, you'll also need to install: + +```bash +pip3 install "openmetadata-ingestion[snowflake-usage]" +``` +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/trino/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/trino/airflow.md index 0ff7413f072..8c7e6e84d05 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/trino/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/trino/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/trino/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/trino/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/trino/cli.md index 20a88d87e60..caba1f55248 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/trino/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/trino/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/trino/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/vertica/airflow.md b/openmetadata-docs/content/openmetadata/connectors/database/vertica/airflow.md index 967cd0f1f2b..e0845d130f0 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/vertica/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/vertica/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/vertica/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/database/vertica/cli.md b/openmetadata-docs/content/openmetadata/connectors/database/vertica/cli.md index fc4bf3ea87c..c19ababc721 100644 --- a/openmetadata-docs/content/openmetadata/connectors/database/vertica/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/database/vertica/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/database/vertica/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/airflow.md b/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/airflow.md index 44dfddd7872..3aeefc4cc67 100644 --- a/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/messaging/kafka/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/cli.md b/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/cli.md index 40024658fec..0883d831e60 100644 --- a/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/messaging/kafka/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/messaging/kafka/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md b/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md index ed4020abf2c..f63440d593d 100644 --- a/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md @@ -9,13 +9,7 @@ In this page, you will learn how to use the `metadata` CLI to run a one-ingestio -## Python requirements - -To run the Amundsen ingestion, you will need to install: - -```commandline -pip3 install "openmetadata-ingestion[amundsen]" -``` + Make sure you are running openmetadata-ingestion version 0.10.2 or above. diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/airflow.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/airflow.md index b84ae5a49d8..04225064e10 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/pipeline/airbyte/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/cli.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/cli.md index cf4b99e6387..6869a2635fd 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/airbyte/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/pipeline/airbyte/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/cli.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/cli.md index 9b4054fcb0e..43bdbb72131 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/cli.md @@ -7,6 +7,13 @@ slug: /openmetadata/connectors/pipeline/airflow/cli + + +Note that this installs the same Airflow version that we ship in the Ingestion Container, which is +Airflow `2.3.3` from Release `0.12`. + +The ingestion using Airflow version 2.3.3 as a source package has been tested against Airflow 2.3.3 and Airflow 2.2.5. +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/index.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/index.md index a5003168639..c7d4ce1e8a9 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/airflow/index.md @@ -3,7 +3,36 @@ title: Airflow slug: /openmetadata/connectors/pipeline/airflow --- - +# Airflow + +In this section, we provide guides and references to use the Airflow connector. + +Configure and schedule Airflow metadata workflow from the OpenMetadata UI: + +If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to +extract metadata directly from your Airflow instance or via the CLI: + + + + + + diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/airflow.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/airflow.md index 03a77095e37..af0389136d9 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/pipeline/fivetran/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/cli.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/cli.md index d18ef01f58f..b48f8636ca1 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/fivetran/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/pipeline/fivetran/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/airflow.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/airflow.md index 283f1ecdef0..7ca8ecff61b 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/airflow.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/airflow.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/pipeline/glue/airflow + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/cli.md b/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/cli.md index f42cfa4794d..fa0948e3e49 100644 --- a/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/cli.md +++ b/openmetadata-docs/content/openmetadata/connectors/pipeline/glue/cli.md @@ -7,6 +7,8 @@ slug: /openmetadata/connectors/pipeline/glue/cli + +

Source Configuration - Service Connection

diff --git a/openmetadata-docs/content/quick-start/local-deployment.md b/openmetadata-docs/content/quick-start/local-deployment.md index bc2d41d3b04..a59e40392e1 100644 --- a/openmetadata-docs/content/quick-start/local-deployment.md +++ b/openmetadata-docs/content/quick-start/local-deployment.md @@ -101,30 +101,142 @@ Follow the instructions [here](https://docs.docker.com/compose/cli-command/#inst
-## Get the latest release and run +## Procedure -From your terminal: +### 1. Create a directory for OpenMetadata -```commandline -mkdir openmetadata && cd "$_" -wget https://github.com/open-metadata/OpenMetadata/releases/download/0.11.3-release/docker-compose.yml -docker compose up -d +Create a new directory for OpenMetadata and navigate into that directory. + +```bash +mkdir openmetadata-docker && cd openmetadata-docker ``` -This will start all the necessary components locally. You can validate that all containers are up -and running with `docker ps`. +### 2. Create a Python virtual environment -```commandline -❯ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -470cc8149826 openmetadata/server:0.11.0 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server -63578aacbff5 openmetadata/ingestion:0.11.0 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion -9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch -08947ab3424b openmetadata/db:0.11.0 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql +Create a virtual environment to avoid conflicts with other Python environments on your host system. +A virtual environment is a self-contained directory tree that contains a Python installation for a particular version +of Python, plus a number of additional packages. + +In a later step you will install the `openmetadata-ingestion` Python module and its dependencies in this virtual environment. + +```bash +python3 -m venv env ``` -In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585): +### 3. Activate the virtual environment +```bash +source env/bin/activate +``` + +### 4. Upgrade pip and setuptools + +```bash +pip3 install --upgrade pip setuptools +``` + +### 5. Install the OpenMetadata Python module using pip + +```bash +pip3 install --upgrade "openmetadata-ingestion[docker]" +``` + +### 6. Ensure the module is installed and ready for use + +```bash +metadata docker --help +``` + +After running the command above, you should see output similar to the following. + +``` +❯ metadata docker --help +Usage: metadata docker [OPTIONS] + + Checks Docker Memory Allocation Run Latest Release Docker - metadata docker + --start Run Local Docker - metadata docker --start -f path/to/docker- + compose.yml + +Options: + --start Start release docker containers + --stop Stops openmetadata docker containers + --pause Pause openmetadata docker containers + --resume Resume/Unpause openmetadata docker + containers + --clean Stops and remove openmetadata docker + containers along with images, volumes, + networks associated + -f, --file-path FILE Path to Local docker-compose.yml + -env-file, --env-file-path FILE + Path to env file containing the environment + variables + --reset-db Reset OpenMetadata Data + --ingest-sample-data Enable the sample metadata ingestion + --help Show this message and exit. +``` + +### 7. Start the OpenMetadata Docker containers + +```bash +metadata docker --start +``` + +This will create a docker network and four containers for the following services: +- MySQL to store the metadata catalog +- Elasticsearch to maintain the metadata index which enables you to search the catalog +- Apache Airflow which OpenMetadata uses for metadata ingestion +- The OpenMetadata UI and API server + +After starting the Docker containers, you should see an output similar to the following. + +``` +[2021-11-18 15:53:52,532] INFO {metadata.cmd:202} - Running Latest Release Docker +[+] Running 5/5 + ⠿ Network tmp_app_net Created 0.3s + ⠿ Container tmp_mysql_1 Started 1.0s + ⠿ Container tmp_elasticsearch_1 Started 1.0s + ⠿ Container tmp_ingestion_1 Started 2.1s + ⠿ Container tmp_openmetadata-server_1 Started 2.2s +[2021-11-18 15:53:55,876] INFO {metadata.cmd:212} - Time took to get containers running: 0:00:03.124889 +....... +``` + +After starting the containers, `metadata` will launch Airflow tasks to ingest sample metadata and usage data for you to +experiment with. This might take several minutes, depending on your system. + + + +- `metadata docker --stop` will stop the Docker containers. +- `metadata docker --clean` will clean/prune the containers, volumes, and networks. + + + +### 8. Wait for metadata ingestion to finish +˚ +Once metadata ingestion has finished and the OpenMetadata UI is ready for use, you will see output similar to the following. + +``` +✅ OpenMetadata is up and running + +Open http://localhost:8585 in your browser to access OpenMetadata.. + +To checkout Ingestion via Airflow, go to http://localhost:8080 +(username: admin, password: admin) + +We are available on Slack , https://slack.open-metadata.org/ . Reach out to us if you have any questions. + +If you like what we are doing, please consider giving us a star on github at https://github.com/open-metadata/OpenMetadata. +It helps OpenMetadata reach wider audience and helps our community. +``` + + + +The `metadata` CLI is very useful for quickly testing when getting started or wanting to try out a new release. + +If you had already set up a release and are trying to test a new one, you might need to run `metadata docker --clean` +to clean up the whole environment and pick up the new ingredients from a fresh start. + + UI @@ -158,3 +270,60 @@ If you want to persist your data, prepare [Named Volumes](/deployment/docker/vol 2. Visit the [Connectors](/openmetadata/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. + +## Troubleshooting + +### Compose is not a docker command + +If you are getting an error such as `"compose" is not a docker command`, you might need to revisit the +installation steps above to make sure that Docker Compose is properly added to your system. + +### metadata CLI issues + +Are you having trouble starting the containers with the `metadata` CLI? While that process is recommended, +you can always run `docker compose` manually after picking up the latest `docker-compose.yml` file from the release: + +```commandline +mkdir openmetadata && cd "$_" +wget https://github.com/open-metadata/OpenMetadata/releases/download/0.11.3-release/docker-compose.yml +docker compose up -d +``` + +This snippet will create a directory named `openmetadata` and download the `docker-compose.yml` file automatically. +Afterwards, it will start the containers. If instead you want to download the file manually to another location, +you can do so from the Releases [page](https://github.com/open-metadata/OpenMetadata/releases). + +This will start all the necessary components locally. You can validate that all containers are up +and running with `docker ps`. + +```commandline +❯ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +470cc8149826 openmetadata/server:0.11.0 "./openmetadata-star…" 45 seconds ago Up 43 seconds 3306/tcp, 9200/tcp, 9300/tcp, 0.0.0.0:8585-8586->8585-8586/tcp openmetadata_server +63578aacbff5 openmetadata/ingestion:0.11.0 "./ingestion_depende…" 45 seconds ago Up 43 seconds 0.0.0.0:8080->8080/tcp openmetadata_ingestion +9f5ee8334f4b docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/local…" 45 seconds ago Up 44 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp openmetadata_elasticsearch +08947ab3424b openmetadata/db:0.11.0 "/entrypoint.sh mysq…" 45 seconds ago Up 44 seconds (healthy) 3306/tcp, 33060-33061/tcp openmetadata_mysql +``` + +In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585): + +### Network openmetadata_app_net Error + +You might see something like: + +``` +The docker command executed was `/usr/local/bin/docker compose --file /var/folders/bl/rm5dhdf127ngm4rr40hvhbq40000gn/T/docker-compose.yml --project-name openmetadata up --detach`. +It returned with code 1 +The content of stdout can be found above the stacktrace (it wasn't captured). +The content of stderr is 'Network openmetadata_app_net Creating +Network openmetadata_app_net Error +failed to create network openmetadata_app_net: Error response from daemon: Pool overlaps with other one on this address space +``` + +A common solution is to run `docker network prune`: + +``` +WARNING! This will remove all custom networks not used by at least one container. +``` + +So be careful if you want to keep up some (unused) networks from your laptop. diff --git a/openmetadata-docs/images/openmetadata/connectors/datalake/service-connection.png b/openmetadata-docs/images/openmetadata/connectors/datalake/service-connection.png new file mode 100644 index 00000000000..1d4bb7fecc2 Binary files /dev/null and b/openmetadata-docs/images/openmetadata/connectors/datalake/service-connection.png differ