diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index ccbd7a0926d..d6043cb1abf 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -337,6 +337,11 @@ site_menu: - category: OpenMetadata / Connectors / Pipeline / Glue / CLI url: /openmetadata/connectors/pipeline/glue/cli + - category: OpenMetadata / Connectors / Metadata + url: /openmetadata/connectors/metadata + - category: OpenMetadata / Connectors / Metadata / Amundsen + url: /openmetadata/connectors/metadata/amundsen + - category: OpenMetadata / Ingestion url: /openmetadata/ingestion - category: OpenMetadata / Ingestion / Workflows diff --git a/openmetadata-docs/content/openmetadata/connectors/index.md b/openmetadata-docs/content/openmetadata/connectors/index.md index e71977f26a9..45e102a8184 100644 --- a/openmetadata-docs/content/openmetadata/connectors/index.md +++ b/openmetadata-docs/content/openmetadata/connectors/index.md @@ -8,6 +8,7 @@ slug: /openmetadata/connectors OpenMetadata can extract metadata from the following list of connectors: ## Database Services + - [Athena](/openmetadata/connectors/database/athena) - [AzureSQL](/openmetadata/connectors/database/azuresql) - [BigQuery](/openmetadata/connectors/database/bigquery) @@ -33,6 +34,7 @@ OpenMetadata can extract metadata from the following list of connectors: - [Vertica](/openmetadata/connectors/database/vertica) ## Dashboard Services + - [Looker](/openmetadata/connectors/dashboard/looker) - [Metabase](/openmetadata/connectors/dashboard/metabase) - [PowerBI](/openmetadata/connectors/dashboard/powerbi) @@ -41,6 +43,7 @@ OpenMetadata can extract metadata from the following list of connectors: - [Tableau](/openmetadata/connectors/dashboard/tableau) ## Messaging Services + - [Kafka](/openmetadata/connectors/messaging/kafka) ## Pipeline Services @@ -48,3 +51,7 @@ OpenMetadata can extract metadata from the following list of connectors: - [Airbyte](/openmetadata/connectors/pipeline/airbyte) - [Airflow](/openmetadata/connectors/pipeline/airflow) - [Glue](/openmetadata/connectors/pipeline/glue) + +## Metadata Services + +- [Amundsen](/openmetadata/connectors/metadata/amundsen) \ No newline at end of file diff --git a/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md b/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md new file mode 100644 index 00000000000..ed4020abf2c --- /dev/null +++ b/openmetadata-docs/content/openmetadata/connectors/metadata/amundsen/index.md @@ -0,0 +1,156 @@ +--- +title: Amundsen +slug: /openmetadata/connectors/metadata/amundsen +--- + +# Amundsen + +In this page, you will learn how to use the `metadata` CLI to run a one-ingestion. + + + +## Python requirements + +To run the Amundsen ingestion, you will need to install: + +```commandline +pip3 install "openmetadata-ingestion[amundsen]" +``` + +Make sure you are running openmetadata-ingestion version 0.10.2 or above. + + +## Create Database Services + +You need to create database services before ingesting the metadata from Amundsen. In the below example we have 5 tables +from 3 data sources i.e., `hive`, `dynamo` & `delta` so in OpenMetadata we have to create database services with the same name +as the source. + +db-service + +To create database service follow these steps: + +### 1. Visit the Services Page + +The first step is ingesting the metadata from your sources. Under Settings, you will find a Services link an external +source system to OpenMetadata. Once a service is created, it can be used to configure metadata, usage, and profiler +workflows.To visit the Services page, select Services from the Settings menu.serv + +db-service + +### 2. Create a New Service + +Click on the Add New Service button to start the Service creation. + +db-service + +### 3. Select the Service Type + +Select the service type which are available on the amundsen and create a service one by one. In this example we will +need to create services for hive, dynamo db & deltalake. Possible service names are `athena`, `bigquery`, `db2`, `druid`, `delta`, +`salesforce`, `oracle`, `glue`, `snowflake` or `hive`. + +db-service + + +db-service + +Note: Adding ingestion in this step is optional, because we will fetch the metadata from Amundsen. After creating all +the database services, `my service` page looks like below, and we are ready to start with the Amundsen ingestion via the CLI. + +db-service + +## Metadata Ingestion + +All connectors are now defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/services/connections/metadata/amundsenConnection.json) +you can find the structure to create a connection to Amundsen. + +In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a +YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. + +The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/services/connections/metadata/amundsenConnection.json). + +### 1. Define the YAML Config + +This is a sample config for Amundsen: + +```yaml +source: + type: amundsen + serviceName: local_amundsen + serviceConnection: + config: + type: Amundsen + username: + password: + hostPort: bolt://localhost:7687 + maxConnectionLifeTime: