diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/athena/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/athena/yaml.md index a726f4e857e..58557f12c1a 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/athena/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/athena/yaml.md @@ -784,9 +784,10 @@ You can learn more about how to ingest lineage [here](/connectors/ingestion/work {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/athena/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/azuresql/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/azuresql/yaml.md index 003fe313eba..f27df5ab90f 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/azuresql/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/azuresql/yaml.md @@ -499,9 +499,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/azuresql/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/db2/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/db2/yaml.md index 171dd37c4cc..e671d2feca3 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/db2/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/db2/yaml.md @@ -502,9 +502,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/db2/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/deltalake/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/deltalake/yaml.md index 41b81a9f41d..89272e2f6c7 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/deltalake/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/deltalake/yaml.md @@ -282,9 +282,10 @@ you will be able to extract metadata from different sources. {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/deltalake/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/domo-database/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/domo-database/yaml.md index 06e8c6c6f24..8d9f65a4db5 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/domo-database/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/domo-database/yaml.md @@ -272,9 +272,10 @@ you will be able to extract metadata from different sources. {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/domo-database/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} \ No newline at end of file diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/dynamodb/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/dynamodb/yaml.md index 3e94e1df012..e77ff4f89fe 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/dynamodb/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/dynamodb/yaml.md @@ -287,9 +287,10 @@ you will be able to extract metadata from different sources. {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/dynamodb/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/hive/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/hive/yaml.md index b1eb08e3d12..eaaeda6f480 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/hive/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/hive/yaml.md @@ -521,9 +521,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/hive/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/impala/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/impala/yaml.md index d43b2574219..87aaf5c05c1 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/impala/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/impala/yaml.md @@ -480,9 +480,10 @@ link="/connectors/ingestion/workflows/dbt" /%} {% tilesContainer %} {% tile -title="Ingest with Airflow" -description="Configure the ingestion using Airflow SDK" -link="/connectors/database/impala/airflow" -/ %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/index.md b/openmetadata-docs/content/v1.1.x/connectors/database/index.md index 0599b0dbed9..c56d2695ddb 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/index.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/index.md @@ -27,8 +27,8 @@ This is the supported list of connectors for Database Services: - [Postgres](/connectors/database/postgres) - [Presto](/connectors/database/presto) - [Redshift](/connectors/database/redshift) -- [Sap Hana](/connectors/database/saphana) - [Salesforce](/connectors/database/salesforce) +- [Sap Hana](/connectors/database/sap-hana) - [SingleStore](/connectors/database/singlestore) - [Snowflake](/connectors/database/snowflake) - [Trino](/connectors/database/trino) diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/mariadb/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/mariadb/yaml.md index ebd4fdb3a5f..5ebe3751272 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/mariadb/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/mariadb/yaml.md @@ -482,9 +482,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/mariadb/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/mongodb/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/mongodb/yaml.md index db882e0bc8d..5df06d728c4 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/mongodb/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/mongodb/yaml.md @@ -233,9 +233,10 @@ you will be able to extract metadata from different sources. {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/mongodb/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/mysql/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/mysql/yaml.md index 479aad38925..7ca0ef7a11b 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/mysql/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/mysql/yaml.md @@ -605,9 +605,10 @@ source: {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/mysql/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/oracle/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/oracle/yaml.md index bf3c5c41cb3..1be404f0d67 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/oracle/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/oracle/yaml.md @@ -526,9 +526,10 @@ You can learn more about how to ingest lineage [here](/connectors/ingestion/work {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/oracle/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/pinotdb/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/pinotdb/yaml.md index ea61a8fbfa5..51dffe7d7f3 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/pinotdb/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/pinotdb/yaml.md @@ -524,9 +524,10 @@ source: {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/pinotdb/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/presto/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/presto/yaml.md index e9d5de3aed5..2e7120ee6ea 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/presto/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/presto/yaml.md @@ -484,9 +484,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/presto/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/salesforce/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/salesforce/yaml.md index 221a06fc18e..9e7b74458e5 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/salesforce/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/salesforce/yaml.md @@ -257,9 +257,10 @@ you will be able to extract metadata from different sources. {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/salesforce/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/sap-hana/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/sap-hana/yaml.md index a0b43b3c249..cb642ae683a 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/sap-hana/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/sap-hana/yaml.md @@ -513,9 +513,10 @@ Note how instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/sap-hana/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/singlestore/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/singlestore/yaml.md index 05e88b39890..786d40e0505 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/singlestore/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/singlestore/yaml.md @@ -476,9 +476,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/singlestore/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/trino/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/trino/yaml.md index 0f571923320..d18b1275e73 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/trino/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/trino/yaml.md @@ -559,9 +559,10 @@ source: {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/trino/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} \ No newline at end of file diff --git a/openmetadata-docs/content/v1.1.x/connectors/database/vertica/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/database/vertica/yaml.md index d2022700482..caf8c8a096b 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/database/vertica/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/database/vertica/yaml.md @@ -521,9 +521,10 @@ Note now instead of running `ingest`, we are using the `profile` command to sele {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/vertica/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/connectors/index.md b/openmetadata-docs/content/v1.1.x/connectors/index.md index ac1493be3a8..b993e2bb003 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/index.md +++ b/openmetadata-docs/content/v1.1.x/connectors/index.md @@ -52,6 +52,7 @@ the following docs to run the Ingestion Framework in any orchestrator externally - [PinotDB](/connectors/database/pinotdb) - [Redshift](/connectors/database/redshift) - [Salesforce](/connectors/database/salesforce) +- [Sap Hana](/connectors/database/sap-hana) - [SingleStore](/connectors/database/singlestore) - [Snowflake](/connectors/database/snowflake) - [SQLite](/connectors/database/sqlite) diff --git a/openmetadata-docs/content/v1.1.x/connectors/ingestion/deployment/index.md b/openmetadata-docs/content/v1.1.x/connectors/ingestion/deployment/index.md index cb1bf3436b8..7482639da08 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/ingestion/deployment/index.md +++ b/openmetadata-docs/content/v1.1.x/connectors/ingestion/deployment/index.md @@ -62,8 +62,8 @@ While the endpoints are directly defined in the `IngestionPipelineResource`, the that decouples how OpenMetadata communicates with the Orchestrator, as different external systems will need different calls and data to be sent. -- You can find the `PipelineServiceClient` abstraction [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/util/PipelineServiceClient.java), -- And the `AirflowRESTClient` implementation [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/airflow/AirflowRESTClient.java). +- You can find the `PipelineServiceClient` abstraction [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/java/org/openmetadata/sdk/PipelineServiceClient.java), +- And the `AirflowRESTClient` implementation [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClient.java). The clients that implement the abstractions from the `PipelineServiceClient` are merely a translation layer between the information received in the shape of an `IngestionPipeline` Entity, and the specific requirements of each Orchestrator. diff --git a/openmetadata-docs/content/v1.1.x/connectors/ingestion/lineage/index.md b/openmetadata-docs/content/v1.1.x/connectors/ingestion/lineage/index.md index ce734c004ea..87deeb3d06b 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/ingestion/lineage/index.md +++ b/openmetadata-docs/content/v1.1.x/connectors/ingestion/lineage/index.md @@ -119,12 +119,12 @@ as well). You might also need to validate if the query logs are available in the You can check the queries being used here: -- [BigQuery](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L428) -- [Snowflake](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L197) -- [MSSQL](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L350) -- [Redshift](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L18) -- [Clickhouse](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L376) -- [Postgres](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L467) +- [BigQuery](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/bigquery/queries.py) +- [Snowflake](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/snowflake/queries.py) +- [MSSQL](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/mssql/queries.py) +- [Redshift](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/redshift/queries.py) +- [Clickhouse](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/clickhouse/queries.py) +- [Postgres](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/postgres/queries.py) By default, we apply a result limit of 1000 records. You might also need to increase that for databases with big volumes of queries. diff --git a/openmetadata-docs/content/v1.1.x/connectors/ml-model/mlflow/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/ml-model/mlflow/yaml.md index 5b66c8eb824..3445aef810b 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/ml-model/mlflow/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/ml-model/mlflow/yaml.md @@ -41,7 +41,7 @@ the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following -[JSON Schema](https://github.com/open-metadata/OpenMetadatablob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/mlmodelServiceMetadataPipeline.json) ### 1. Define the YAML Config diff --git a/openmetadata-docs/content/v1.1.x/connectors/ml-model/sagemaker/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/ml-model/sagemaker/yaml.md index ed2ce1bfecf..21d31c25f91 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/ml-model/sagemaker/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/ml-model/sagemaker/yaml.md @@ -56,7 +56,7 @@ pip3 install "openmetadata-ingestion[sagemaker]" ## Metadata Ingestion All connectors are defined as JSON Schemas. -[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/mlmodel/sagemakerConnection.json) +[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/mlmodel/sageMakerConnection.json) you can find the structure to create a connection to Sagemaker. In order to create and run a Metadata Ingestion workflow, we will follow @@ -64,7 +64,7 @@ the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following -[JSON Schema](https://github.com/open-metadata/OpenMetadatablob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/mlmodelServiceMetadataPipeline.json) ### 1. Define the YAML Config diff --git a/openmetadata-docs/content/v1.1.x/connectors/storage/s3/yaml.md b/openmetadata-docs/content/v1.1.x/connectors/storage/s3/yaml.md index c2a019b3538..1f0fa10cffb 100644 --- a/openmetadata-docs/content/v1.1.x/connectors/storage/s3/yaml.md +++ b/openmetadata-docs/content/v1.1.x/connectors/storage/s3/yaml.md @@ -338,9 +338,10 @@ you will be able to extract metadata from different sources. {% tilesContainer %} {% tile -title="Ingest with Airflow" -description="Configure the ingestion using Airflow SDK" -link="/connectors/database/athena/airflow" -/ %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.1.x/deployment/bare-metal/index.md b/openmetadata-docs/content/v1.1.x/deployment/bare-metal/index.md index a48dac71896..7dae8c7fd9c 100644 --- a/openmetadata-docs/content/v1.1.x/deployment/bare-metal/index.md +++ b/openmetadata-docs/content/v1.1.x/deployment/bare-metal/index.md @@ -237,7 +237,7 @@ installation. ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. \ No newline at end of file diff --git a/openmetadata-docs/content/v1.1.x/deployment/docker/index.md b/openmetadata-docs/content/v1.1.x/deployment/docker/index.md index bfd12a7ee71..41784749ffc 100644 --- a/openmetadata-docs/content/v1.1.x/deployment/docker/index.md +++ b/openmetadata-docs/content/v1.1.x/deployment/docker/index.md @@ -367,7 +367,7 @@ installation. ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. diff --git a/openmetadata-docs/content/v1.1.x/deployment/security/index.md b/openmetadata-docs/content/v1.1.x/deployment/security/index.md index 2262c946959..773a1dfaba4 100644 --- a/openmetadata-docs/content/v1.1.x/deployment/security/index.md +++ b/openmetadata-docs/content/v1.1.x/deployment/security/index.md @@ -11,7 +11,7 @@ OpenMetadata has support for Google SSO, Okta SSO, custom OIDC, Auth0, Azure SSO Enabling Security is only required for your **Production** installation. If you are testing OpenMetadata, it will be easier and faster to set up without security. To get up and running quickly with OpenMetadata (without security), -please follow the [Quickstart](/quick-start/local-deployment) guide. +please follow the [Quickstart](/quick-start) guide. {%inlineCalloutContainer%} {%inlineCallout diff --git a/openmetadata-docs/content/v1.1.x/deployment/security/okta/bare-metal.md b/openmetadata-docs/content/v1.1.x/deployment/security/okta/bare-metal.md index a9792051a43..498cbf72b2a 100644 --- a/openmetadata-docs/content/v1.1.x/deployment/security/okta/bare-metal.md +++ b/openmetadata-docs/content/v1.1.x/deployment/security/okta/bare-metal.md @@ -66,5 +66,5 @@ airflowConfiguration: metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api} ``` -**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for +**Note:** Follow [this](/how-to-guides/quick-start-guide-for-admins/bots) guide to configure the `ingestion-bot` credentials for ingesting data from Airflow. \ No newline at end of file diff --git a/openmetadata-docs/content/v1.1.x/deployment/security/okta/docker.md b/openmetadata-docs/content/v1.1.x/deployment/security/okta/docker.md index 7a9e524c40d..1f092837785 100644 --- a/openmetadata-docs/content/v1.1.x/deployment/security/okta/docker.md +++ b/openmetadata-docs/content/v1.1.x/deployment/security/okta/docker.md @@ -74,7 +74,7 @@ AUTHENTICATION_CLIENT_ID={CLIENT_ID - SPA APP} # Update with your Client ID AUTHENTICATION_CALLBACK_URL=http://localhost:8585/callback ``` -**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for +**Note:** Follow [this](/how-to-guides/quick-start-guide-for-admins/bots) guide to configure the `ingestion-bot` credentials for ingesting data from Airflow. ## 2. Start Docker diff --git a/openmetadata-docs/content/v1.1.x/quick-start/local-docker-deployment.md b/openmetadata-docs/content/v1.1.x/quick-start/local-docker-deployment.md index 175d6775334..69e1959c87b 100644 --- a/openmetadata-docs/content/v1.1.x/quick-start/local-docker-deployment.md +++ b/openmetadata-docs/content/v1.1.x/quick-start/local-docker-deployment.md @@ -254,7 +254,7 @@ installation. ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. \ No newline at end of file diff --git a/openmetadata-docs/content/v1.1.x/quick-start/local-kubernetes-deployment.md b/openmetadata-docs/content/v1.1.x/quick-start/local-kubernetes-deployment.md index ded99232eca..15aa6541acd 100644 --- a/openmetadata-docs/content/v1.1.x/quick-start/local-kubernetes-deployment.md +++ b/openmetadata-docs/content/v1.1.x/quick-start/local-kubernetes-deployment.md @@ -187,7 +187,7 @@ For more information, visit the kubectl logs command line reference documentatio ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. diff --git a/openmetadata-docs/content/v1.1.x/quick-start/sandbox.md b/openmetadata-docs/content/v1.1.x/quick-start/sandbox.md index b32f06e558d..45663563439 100644 --- a/openmetadata-docs/content/v1.1.x/quick-start/sandbox.md +++ b/openmetadata-docs/content/v1.1.x/quick-start/sandbox.md @@ -34,7 +34,7 @@ alt="tour" /%} ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. diff --git a/openmetadata-docs/content/v1.2.x/connectors/database/unity-catalog/index.md b/openmetadata-docs/content/v1.2.x/connectors/database/unity-catalog/index.md index fc0eb2fe224..25576a4b956 100644 --- a/openmetadata-docs/content/v1.2.x/connectors/database/unity-catalog/index.md +++ b/openmetadata-docs/content/v1.2.x/connectors/database/unity-catalog/index.md @@ -41,6 +41,8 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI: {% partial file="/v1.2/connectors/external-ingestion-deployment.md" /%} +## Requirements + {%inlineCallout icon="description" bold="OpenMetadata 0.12 or later" href="/deployment"%} To deploy OpenMetadata, check the Deployment guides. {%/inlineCallout%} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/athena/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/athena/index.md index dc545b50058..7fba3121ba3 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/athena/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/athena/index.md @@ -33,7 +33,7 @@ Configure and schedule Athena metadata and profiler workflows from the OpenMetad - [Requirements](#requirements) - [Metadata Ingestion](#metadata-ingestion) - [Service Name](#service-name) - - [Connection Options](#connection-options) + - [Connection Details](#connection-details) - [Metadata Ingestion Options](#metadata-ingestion-options) - [Troubleshooting](#troubleshooting) - [Workflow Deployment Error](#workflow-deployment-error) diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/azuresql/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/azuresql/yaml.md index 335e61009d8..a2cbb83ff99 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/azuresql/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/azuresql/yaml.md @@ -207,9 +207,10 @@ source: {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/azuresql/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/index.md index 7c130c1fd36..a7cfdd39c4b 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/index.md @@ -47,6 +47,8 @@ Configure and schedule BigQuery metadata and profiler workflows from the OpenMet To deploy OpenMetadata, check the Deployment guides. {%/inlineCallout%} +## Requirements + ### Data Catalog API Permissions - Go to [https://console.cloud.google.com/apis/library/datacatalog.googleapis.com](https://console.cloud.google.com/apis/library/datacatalog.googleapis.com) diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/yaml.md index 94548544181..486c6f11d5a 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/bigquery/yaml.md @@ -98,6 +98,7 @@ link="/connectors/database/bigquery/roles" / %} {% /tilesContainer %} +## Metadata Ingestion ### 1. Define the YAML Config diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/couchbase/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/couchbase/yaml.md index 96182f4997d..fbc1dd6f448 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/couchbase/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/couchbase/yaml.md @@ -156,9 +156,10 @@ source: {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/mongodb/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/databricks/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/databricks/index.md index 6a36249aa70..b472f90954c 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/databricks/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/databricks/index.md @@ -30,7 +30,6 @@ In this section, we provide guides and references to use the Databricks connecto Configure and schedule Databricks metadata and profiler workflows from the OpenMetadata UI: -- [Requirements](#requirements) - [Unity Catalog](#unity-catalog) - [Metadata Ingestion](#metadata-ingestion) - [Query Usage](/connectors/ingestion/workflows/usage) diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/impala/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/impala/index.md index d678e965d2d..236c820b475 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/impala/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/impala/index.md @@ -31,9 +31,9 @@ In this section, we provide guides and references to use the Impala connector. Configure and schedule Impala metadata and profiler workflows from the OpenMetadata UI: - [Requirements](#requirements) - [Metadata Ingestion](#metadata-ingestion) -- [Data Profiler](#data-profiler) -- [Data Quality](#data-quality) -- [dbt Integration](#dbt-integration) +- [Data Profiler](/connectors/ingestion/workflows/profiler) +- [Data Quality](/connectors/ingestion/workflows/data-quality) +- [dbt Integration](/connectors/ingestion/workflows/dbt) {% partial file="/v1.3/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/impala/yaml"} /%} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/presto/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/presto/yaml.md index b85064b354a..0318c1a8f20 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/presto/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/presto/yaml.md @@ -195,9 +195,10 @@ source: {% tilesContainer %} {% tile - title="Ingest with Airflow" - description="Configure the ingestion using Airflow SDK" - link="/connectors/database/presto/airflow" - / %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/unity-catalog/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/unity-catalog/index.md index ef59c25d6fd..bd6ba5925b6 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/unity-catalog/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/unity-catalog/index.md @@ -41,6 +41,8 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI: {% partial file="/v1.3/connectors/external-ingestion-deployment.md" /%} +## Requirements + {%inlineCallout icon="description" bold="OpenMetadata 0.12 or later" href="/deployment"%} To deploy OpenMetadata, check the Deployment guides. {%/inlineCallout%} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/vertica/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/vertica/index.md index 0684adb3d59..ed663c16b84 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/vertica/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/database/vertica/index.md @@ -33,9 +33,9 @@ Configure and schedule Vertica metadata and profiler workflows from the OpenMeta - [Requirements](#requirements) - [Metadata Ingestion](#metadata-ingestion) -- [Data Profiler](#data-profiler) -- [Data Quality](#data-quality) -- [dbt Integration](#dbt-integration) +- [Data Profiler](/connectors/ingestion/workflows/profiler) +- [Data Quality](/connectors/ingestion/workflows/data-quality) +- [dbt Integration](/connectors/ingestion/workflows/dbt) {% partial file="/v1.3/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/vertica/yaml"} /%} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/deployment/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/deployment/index.md index 43deca8a3ba..42d1db7ddbe 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/deployment/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/deployment/index.md @@ -62,8 +62,8 @@ While the endpoints are directly defined in the `IngestionPipelineResource`, the that decouples how OpenMetadata communicates with the Orchestrator, as different external systems will need different calls and data to be sent. -- You can find the `PipelineServiceClient` abstraction [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/util/PipelineServiceClient.java), -- And the `AirflowRESTClient` implementation [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/airflow/AirflowRESTClient.java). +- You can find the `PipelineServiceClient` abstraction [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/java/org/openmetadata/sdk/PipelineServiceClient.java), +- And the `AirflowRESTClient` implementation [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClient.java). The clients that implement the abstractions from the `PipelineServiceClient` are merely a translation layer between the information received in the shape of an `IngestionPipeline` Entity, and the specific requirements of each Orchestrator. @@ -284,7 +284,7 @@ pipelineServiceClient.deployPipeline(ingestionPipeline); ``` Then, the actual deployment logic is handled by the class implementing the Pipeline Service Client. For this example, -it will be the [AirflowRESTClient](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/airflow/AirflowRESTClient.java). +it will be the [AirflowRESTClient](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClient.java). First, let's see what it is needed to instantiate the Airflow REST Client: diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/lineage/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/lineage/index.md index 34c9d2a78b5..6f21e09c6cc 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/lineage/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ingestion/lineage/index.md @@ -119,12 +119,12 @@ as well). You might also need to validate if the query logs are available in the You can check the queries being used here: -- [BigQuery](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L428) -- [Snowflake](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L197) -- [MSSQL](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L350) -- [Redshift](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L18) -- [Clickhouse](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L376) -- [Postgres](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/sql_queries.py#L467) +- [BigQuery](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/bigquery/queries.py) +- [Snowflake](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/snowflake/queries.py) +- [MSSQL](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/mssql/queries.py) +- [Redshift](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/redshift/queries.py) +- [Clickhouse](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/clickhouse/queries.py) +- [Postgres](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/postgres/queries.py) By default, we apply a result limit of 1000 records. You might also need to increase that for databases with big volumes of queries. diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/mlflow/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/mlflow/yaml.md index b5e92546a96..245815fddc5 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/mlflow/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/mlflow/yaml.md @@ -41,7 +41,7 @@ the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following -[JSON Schema](https://github.com/open-metadata/OpenMetadatablob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/mlmodelServiceMetadataPipeline.json) ### 1. Define the YAML Config diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md index 006d19c1bda..65a672d72c6 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md @@ -64,7 +64,7 @@ the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following -[JSON Schema](https://github.com/open-metadata/OpenMetadatablob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) +[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/mlmodelServiceMetadataPipeline.json) ### 1. Define the YAML Config diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/pipeline/dagster/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/pipeline/dagster/index.md index d660d4aabae..0b5b2d0653b 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/pipeline/dagster/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/pipeline/dagster/index.md @@ -14,7 +14,7 @@ Configure and schedule Dagster metadata and profiler workflows from the OpenMeta - [Dagster Versions](#dagster-versions) - [Metadata Ingestion](#metadata-ingestion) - [Service Name](#service-name) - - [Connection Options](#connection-options) + - [Connection Details](#connection-details) - [Metadata Ingestion Options](#metadata-ingestion-options) - [Troubleshooting](#troubleshooting) - [Workflow Deployment Error](#workflow-deployment-error) diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/storage/s3/yaml.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/storage/s3/yaml.md index ee22c1abe5b..264c23ecc7c 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/storage/s3/yaml.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/connectors/storage/s3/yaml.md @@ -299,9 +299,10 @@ source: {% tilesContainer %} {% tile -title="Ingest with Airflow" -description="Configure the ingestion using Airflow SDK" -link="/connectors/database/athena/airflow" -/ %} + icon="mediation" + title="Configure Ingestion Externally" + description="Deploy, configure, and manage the ingestion workflows externally." + link="/deployment/ingestion" + / %} {% /tilesContainer %} diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/docker/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/docker/index.md index 2b52182c39f..20428a57cae 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/docker/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/docker/index.md @@ -107,7 +107,7 @@ wget https://github.com/open-metadata/OpenMetadata/releases/download/1.2.2-relea ### 3. Update Environment Variables required for OpenMetadata Dependencies -In the previous [step](#2-download-docker-compose-file-from-github-release-branch), we download the `docker-compose` file. +In the previous [step](#2-download-docker-compose-file-from-github-releases), we download the `docker-compose` file. Identify and update the environment variables in the file to prepare openmetadata configurations. @@ -367,7 +367,7 @@ installation. ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/index.md index 2262c946959..773a1dfaba4 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/index.md @@ -11,7 +11,7 @@ OpenMetadata has support for Google SSO, Okta SSO, custom OIDC, Auth0, Azure SSO Enabling Security is only required for your **Production** installation. If you are testing OpenMetadata, it will be easier and faster to set up without security. To get up and running quickly with OpenMetadata (without security), -please follow the [Quickstart](/quick-start/local-deployment) guide. +please follow the [Quickstart](/quick-start) guide. {%inlineCalloutContainer%} {%inlineCallout diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/bare-metal.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/bare-metal.md index 6b7b00b810e..0cf9848f10d 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/bare-metal.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/bare-metal.md @@ -67,5 +67,5 @@ airflowConfiguration: metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api} ``` -**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for +**Note:** Follow [this](/developers/bots) guide to configure the `ingestion-bot` credentials for ingesting data from Airflow. \ No newline at end of file diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/docker.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/docker.md index 9878b2fa8b9..3f057f20a7e 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/docker.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/security/okta/docker.md @@ -74,7 +74,7 @@ AUTHENTICATION_CLIENT_ID={CLIENT_ID - SPA APP} # Update with your Client ID AUTHENTICATION_CALLBACK_URL=http://localhost:8585/callback ``` -**Note:** Follow [this](/how-to-guides/feature-configurations/bots) guide to configure the `ingestion-bot` credentials for +**Note:** Follow [this](/developers/bots) guide to configure the `ingestion-bot` credentials for ingesting data from Airflow. ## 2. Start Docker diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/upgrade/kubernetes.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/upgrade/kubernetes.md index e95756971bb..31d8302ad2a 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/upgrade/kubernetes.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/deployment/upgrade/kubernetes.md @@ -108,7 +108,7 @@ openmetadata: - global: Additional property airflow is not allowed ``` -This means the values passed to the helm charts has a section `global.airflow`. As per the breaking changes mentioned [here](/deployment/upgrade/versions/013-to-100#airflow-configuration-&-pipeline-service-client), Airflow configs are replaced with pipelineServiceClient for Helm Charts. +This means the values passed to the helm charts has a section `global.airflow`. As per the breaking changes mentioned [here](/deployment/upgrade/versions/100-to-110#pipeline-service-client-configuration), Airflow configs are replaced with pipelineServiceClient for Helm Charts. The Helm Chart Values JSON Schema helps to catch the above breaking changes and this section will help you resolve and update your configurations for the same. You can read more about JSON Schema with Helm Charts [here](https://helm.sh/docs/topics/charts/#schema-files). diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/admin-guide/how-to-ingest-metadata.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/admin-guide/how-to-ingest-metadata.md index 1d54412afc5..30c6abbf7a5 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/admin-guide/how-to-ingest-metadata.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/admin-guide/how-to-ingest-metadata.md @@ -172,6 +172,6 @@ Admin users can create, edit, or delete services. They can also view the connect color="violet-70" bold="Delete a Service Connection" icon="MdArrowForward" - href="/how-to-guides/admin-guide/how-to-ingest-metadata/delete-service-connection"%} + href="/how-to-guides/admin-guide/delete-service-connection"%} Permanently delete a service connection. {%/inlineCallout%} \ No newline at end of file diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/index.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/index.md index 05c3d33e555..776a0669ce5 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/index.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/index.md @@ -87,7 +87,7 @@ OpenMetadata is a complete package for data teams to break down team silos, shar - Enhance organizational **[Data Culture](/how-to-guides/data-insights)** to gain crucial insights to drive innovation. -- Define your **[Glossary](/how-to-guides/data-governance/glossary-classification)** to build a common understanding of terms within your organization. +- Define your **[Glossary](/how-to-guides/data-governance/glossary)** to build a common understanding of terms within your organization. - Implement **[Data Governance](/how-to-guides/data-governance)** to maintain data integrity, security, and compliance. diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/user-guide-data-users/tags.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/user-guide-data-users/tags.md index d59ffeb79c7..f47eed543c5 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/user-guide-data-users/tags.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/how-to-guides/user-guide-data-users/tags.md @@ -36,7 +36,7 @@ You can view all the tags in the right panel. Data assets can also be classified using Tiers. Learn more about [Tiers](/how-to-guides/data-governance/classification/tiers). -Among the Classification Tags, OpenMetadata has some System Classification. Learn more about the [System Tags](/how-to-guides/data-governance/classification/classification). +Among the Classification Tags, OpenMetadata has some System Classification. Learn more about the [System Tags](/how-to-guides/data-governance/classification/overview#classification-in-openmetadata). ## Auto-Classification in OpenMetadata diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-docker-deployment.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-docker-deployment.md index 373564d5c4d..4dc36216f90 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-docker-deployment.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-docker-deployment.md @@ -254,7 +254,7 @@ installation. ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-kubernetes-deployment.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-kubernetes-deployment.md index ded99232eca..15aa6541acd 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-kubernetes-deployment.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/local-kubernetes-deployment.md @@ -187,7 +187,7 @@ For more information, visit the kubectl logs command line reference documentatio ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs. diff --git a/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/sandbox.md b/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/sandbox.md index 9e0211da9c8..a8e5b88681d 100644 --- a/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/sandbox.md +++ b/openmetadata-docs/content/v1.3.x-SNAPSHOT/quick-start/sandbox.md @@ -34,7 +34,7 @@ alt="tour" /%} ## Next Steps -1. Visit the [Features](/releases/features) overview page and explore the OpenMetadata UI. +1. Refer the [How-to Guides](/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](/swagger.html) documentation and explore the rich set of OpenMetadata APIs.