mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-18 04:05:42 +00:00
Docs: Brand Name Tag Updation (#22760)
* Docs: Updation * Docs: Updation * Docs: Updation --------- Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”> Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
This commit is contained in:
Rounak Dhillon
GitHub
parent
79189e49e0
commit
2b2a4090fc
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Airflow Connector | OpenMetadata Workflow Orchestration
|
||||
description: Connect Apache Airflow to OpenMetadata effortlessly. Complete setup guide, configuration steps, and pipeline metadata extraction for seamless data lineage tracking.
|
||||
title: Airflow Connector | `brandName` Workflow Orchestration
|
||||
description: Connect Apache Airflow to `brandName` effortlessly. Complete setup guide, configuration steps, and pipeline metadata extraction for seamless data lineage tracking.
|
||||
slug: /connectors/pipeline/airflow
|
||||
---
|
||||
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Collate SaaS
|
||||
title: Collate SaaS Setup Guide | Connect Database Services for Metadata Extraction
|
||||
description: Learn how to configure database services in Collate SaaS, allow IP access, test connections, and begin metadata extraction using built-in connectors like Snowflake.
|
||||
slug: /getting-started/day-1/collate-saas
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from your Airflow
|
||||
description: Learn how to run OpenMetadata ingestion workflows using Python, Docker, or Virtualenv operators in Airflow for secure and flexible metadata ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/airflow
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Managing Credentials
|
||||
title: Managing Credentials Securely | Collate Ingestion Best Practices
|
||||
description: Learn secure ways to manage credentials in Collate ingestion workflows using environment variables, Airflow connections, GitHub secrets, and existing services.
|
||||
slug: /getting-started/day-1/hybrid-saas/credentials
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from GCP Composer
|
||||
title: Run the ingestion from GCP Composer
|
||||
description: Learn to run Collate ingestion in GCP Composer using Python or KubernetesPod operators. Install packages or run containers for secure, scalable ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/gcp-composer
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from GitHub Actions
|
||||
description: Automate Collate ingestion with GitHub Actions. Set up YAML configs, secure credentials, schedule workflows, and send Slack alerts on failure.
|
||||
slug: /getting-started/day-1/hybrid-saas/github-actions
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Hybrid Ingestion Runner
|
||||
title: Hybrid Ingestion Runner | Secure Metadata Workflows in Your Cloud
|
||||
description: Learn to configure and manage Hybrid Ingestion Runner to securely execute workflows in your cloud using AWS, Azure, or GCP secrets—without exposing credentials.
|
||||
slug: /getting-started/day-1/hybrid-saas/hybrid-ingestion-runner
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Hybrid SaaS | Secure Metadata Ingestion for Collate
|
||||
description: Learn how to securely ingest metadata using the Collate Ingestion Agent in your own infrastructure. Ideal for private networks and hybrid SaaS setups.
|
||||
slug: /getting-started/day-1/hybrid-saas
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from AWS MWAA
|
||||
description: Set up Collate ingestion workflows on AWS MWAA using Python, ECS, or Virtualenv operators. Compare approaches and configure DAGs for secure metadata ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/mwaa
|
||||
collate: true
|
||||
---
|
||||
@ -35,7 +36,7 @@ To install the package, we need to update the `requirements.txt` file from the M
|
||||
openmetadata-ingestion[<plugin>]==x.y.z
|
||||
```
|
||||
|
||||
Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1.
|
||||
Where `x.y.z` is the version of the Collate ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1.
|
||||
|
||||
The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==1.3.1`.
|
||||
|
||||
@ -78,7 +79,7 @@ def metadata_ingestion_workflow():
|
||||
with DAG(
|
||||
"redshift_ingestion",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
description="An example DAG which runs a Collate ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
@ -293,7 +294,7 @@ with DAG(
|
||||
```
|
||||
|
||||
Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following
|
||||
the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values:
|
||||
the official Collate docs, and the value of the `pipelineType` configuration will need to hold one of the following values:
|
||||
|
||||
- `metadata`
|
||||
- `usage`
|
||||
@ -410,7 +411,7 @@ YAML config
|
||||
with DAG(
|
||||
"redshift_ingestion",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
description="An example DAG which runs a Collate ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
@ -419,7 +420,7 @@ with DAG(
|
||||
task_id="ingest_redshift",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
requirements=['openmetadata-ingestion==1.0.5.0',
|
||||
'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with OpenMetadata's 'tabulate' requirements
|
||||
'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with Collate's 'tabulate' requirements
|
||||
'apache-airflow-providers-amazon==6.0.0', # Amazon Airflow provider is necessary for MWAA
|
||||
'watchtower',],
|
||||
system_site_packages=False,
|
||||
@ -430,7 +431,7 @@ with DAG(
|
||||
Where you can update the YAML configuration and workflow classes accordingly. accordingly. Further examples on how to
|
||||
run the ingestion can be found on the documentation (e.g., [Snowflake](/connectors/database/snowflake)).
|
||||
|
||||
You will also need to determine the OpenMetadata ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`.
|
||||
You will also need to determine the Collate ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`.
|
||||
For Airflow providers, you will want to pull the provider versions from [the matching constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt). Since this example installs Airflow Providers v2.4.3 on Python 3.7, we use that constraints file.
|
||||
|
||||
Also note that the ingestion workflow function must be entirely self-contained as it will run by itself in the virtualenv. Any imports it needs, including the configuration, must exist within the function itself.
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Day 1
|
||||
title: Day 1 Setup Guide | Connect, Ingest & Collaborate in Collate
|
||||
description: Start your Collate journey by connecting data sources, ingesting metadata, inviting users, and exploring key features. Everything you need for Day 1.
|
||||
slug: /getting-started/day-1
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Getting Started
|
||||
title: Getting Started with Collate | Setup & Onboarding Guide
|
||||
description: Quickly set up your Collate environment in 30 minutes. Learn how to connect data sources, explore features, and begin your onboarding journey.
|
||||
slug: /getting-started
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Data Profiler | OpenMetadata Data Profiling Guide
|
||||
title: Data Profiler | `brandName` Data Profiling Guide
|
||||
description: Explore profiling workflows including histogram metrics, null counts, and field-level health.
|
||||
slug: /how-to-guides/data-quality-observability/profiler
|
||||
---
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Airflow Connector | OpenMetadata Workflow Orchestration
|
||||
description: Connect Apache Airflow to OpenMetadata effortlessly. Complete setup guide, configuration steps, and pipeline metadata extraction for seamless data lineage tracking.
|
||||
title: Airflow Connector | `brandName` Workflow Orchestration
|
||||
description: Connect Apache Airflow to `brandName` effortlessly. Complete setup guide, configuration steps, and pipeline metadata extraction for seamless data lineage tracking.
|
||||
slug: /connectors/pipeline/airflow
|
||||
---
|
||||
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Collate SaaS
|
||||
title: Collate SaaS Setup Guide | Connect Database Services for Metadata Extraction
|
||||
description: Learn how to configure database services in Collate SaaS, allow IP access, test connections, and begin metadata extraction using built-in connectors like Snowflake.
|
||||
slug: /getting-started/day-1/collate-saas
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from your Airflow
|
||||
description: Learn how to run OpenMetadata ingestion workflows using Python, Docker, or Virtualenv operators in Airflow for secure and flexible metadata ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/airflow
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Managing Credentials
|
||||
title: Managing Credentials Securely | Collate Ingestion Best Practices
|
||||
description: Learn secure ways to manage credentials in Collate ingestion workflows using environment variables, Airflow connections, GitHub secrets, and existing services.
|
||||
slug: /getting-started/day-1/hybrid-saas/credentials
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from GCP Composer
|
||||
title: Run the ingestion from GCP Composer
|
||||
description: Learn to run Collate ingestion in GCP Composer using Python or KubernetesPod operators. Install packages or run containers for secure, scalable ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/gcp-composer
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from GitHub Actions
|
||||
description: Automate Collate ingestion with GitHub Actions. Set up YAML configs, secure credentials, schedule workflows, and send Slack alerts on failure.
|
||||
slug: /getting-started/day-1/hybrid-saas/github-actions
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Hybrid Ingestion Runner
|
||||
title: Hybrid Ingestion Runner | Secure Metadata Workflows in Your Cloud
|
||||
description: Learn to configure and manage Hybrid Ingestion Runner to securely execute workflows in your cloud using AWS, Azure, or GCP secrets—without exposing credentials.
|
||||
slug: /getting-started/day-1/hybrid-saas/hybrid-ingestion-runner
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Hybrid SaaS | Secure Metadata Ingestion for Collate
|
||||
description: Learn how to securely ingest metadata using the Collate Ingestion Agent in your own infrastructure. Ideal for private networks and hybrid SaaS setups.
|
||||
slug: /getting-started/day-1/hybrid-saas
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from AWS MWAA
|
||||
description: Set up Collate ingestion workflows on AWS MWAA using Python, ECS, or Virtualenv operators. Compare approaches and configure DAGs for secure metadata ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/mwaa
|
||||
collate: true
|
||||
---
|
||||
@ -35,7 +36,7 @@ To install the package, we need to update the `requirements.txt` file from the M
|
||||
openmetadata-ingestion[<plugin>]==x.y.z
|
||||
```
|
||||
|
||||
Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1.
|
||||
Where `x.y.z` is the version of the Collate ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1.
|
||||
|
||||
The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==1.3.1`.
|
||||
|
||||
@ -78,7 +79,7 @@ def metadata_ingestion_workflow():
|
||||
with DAG(
|
||||
"redshift_ingestion",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
description="An example DAG which runs a Collate ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
@ -293,7 +294,7 @@ with DAG(
|
||||
```
|
||||
|
||||
Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following
|
||||
the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values:
|
||||
the official Collate docs, and the value of the `pipelineType` configuration will need to hold one of the following values:
|
||||
|
||||
- `metadata`
|
||||
- `usage`
|
||||
@ -410,7 +411,7 @@ YAML config
|
||||
with DAG(
|
||||
"redshift_ingestion",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
description="An example DAG which runs a Collate ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
@ -419,7 +420,7 @@ with DAG(
|
||||
task_id="ingest_redshift",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
requirements=['openmetadata-ingestion==1.0.5.0',
|
||||
'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with OpenMetadata's 'tabulate' requirements
|
||||
'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with Collate's 'tabulate' requirements
|
||||
'apache-airflow-providers-amazon==6.0.0', # Amazon Airflow provider is necessary for MWAA
|
||||
'watchtower',],
|
||||
system_site_packages=False,
|
||||
@ -430,7 +431,7 @@ with DAG(
|
||||
Where you can update the YAML configuration and workflow classes accordingly. accordingly. Further examples on how to
|
||||
run the ingestion can be found on the documentation (e.g., [Snowflake](/connectors/database/snowflake)).
|
||||
|
||||
You will also need to determine the OpenMetadata ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`.
|
||||
You will also need to determine the Collate ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`.
|
||||
For Airflow providers, you will want to pull the provider versions from [the matching constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt). Since this example installs Airflow Providers v2.4.3 on Python 3.7, we use that constraints file.
|
||||
|
||||
Also note that the ingestion workflow function must be entirely self-contained as it will run by itself in the virtualenv. Any imports it needs, including the configuration, must exist within the function itself.
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Day 1
|
||||
title: Day 1 Setup Guide | Connect, Ingest & Collaborate in Collate
|
||||
description: Start your Collate journey by connecting data sources, ingesting metadata, inviting users, and exploring key features. Everything you need for Day 1.
|
||||
slug: /getting-started/day-1
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Getting Started
|
||||
title: Getting Started with Collate | Setup & Onboarding Guide
|
||||
description: Quickly set up your Collate environment in 30 minutes. Learn how to connect data sources, explore features, and begin your onboarding journey.
|
||||
slug: /getting-started
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Data Profiler | OpenMetadata Data Profiling Guide
|
||||
title: Data Profiler | `brandName` Data Profiling Guide
|
||||
description: Explore profiling workflows including histogram metrics, null counts, and field-level health.
|
||||
slug: /how-to-guides/data-quality-observability/profiler
|
||||
---
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Airflow Connector | OpenMetadata Workflow Orchestration
|
||||
description: Connect Apache Airflow to OpenMetadata effortlessly. Complete setup guide, configuration steps, and pipeline metadata extraction for seamless data lineage tracking.
|
||||
title: Airflow Connector | `brandName` Workflow Orchestration
|
||||
description: Connect Apache Airflow to `brandName` effortlessly. Complete setup guide, configuration steps, and pipeline metadata extraction for seamless data lineage tracking.
|
||||
slug: /connectors/pipeline/airflow
|
||||
---
|
||||
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Collate SaaS
|
||||
title: Collate SaaS Setup Guide | Connect Database Services for Metadata Extraction
|
||||
description: Learn how to configure database services in Collate SaaS, allow IP access, test connections, and begin metadata extraction using built-in connectors like Snowflake.
|
||||
slug: /getting-started/day-1/collate-saas
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from your Airflow
|
||||
description: Learn how to run OpenMetadata ingestion workflows using Python, Docker, or Virtualenv operators in Airflow for secure and flexible metadata ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/airflow
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Managing Credentials
|
||||
title: Managing Credentials Securely | Collate Ingestion Best Practices
|
||||
description: Learn secure ways to manage credentials in Collate ingestion workflows using environment variables, Airflow connections, GitHub secrets, and existing services.
|
||||
slug: /getting-started/day-1/hybrid-saas/credentials
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from GCP Composer
|
||||
title: Run the ingestion from GCP Composer
|
||||
description: Learn to run Collate ingestion in GCP Composer using Python or KubernetesPod operators. Install packages or run containers for secure, scalable ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/gcp-composer
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from GitHub Actions
|
||||
description: Automate Collate ingestion with GitHub Actions. Set up YAML configs, secure credentials, schedule workflows, and send Slack alerts on failure.
|
||||
slug: /getting-started/day-1/hybrid-saas/github-actions
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Hybrid Ingestion Runner
|
||||
title: Hybrid Ingestion Runner | Secure Metadata Workflows in Your Cloud
|
||||
description: Learn to configure and manage Hybrid Ingestion Runner to securely execute workflows in your cloud using AWS, Azure, or GCP secrets—without exposing credentials.
|
||||
slug: /getting-started/day-1/hybrid-saas/hybrid-ingestion-runner
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Hybrid SaaS | Secure Metadata Ingestion for Collate
|
||||
description: Learn how to securely ingest metadata using the Collate Ingestion Agent in your own infrastructure. Ideal for private networks and hybrid SaaS setups.
|
||||
slug: /getting-started/day-1/hybrid-saas
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Run the ingestion from AWS MWAA
|
||||
description: Set up Collate ingestion workflows on AWS MWAA using Python, ECS, or Virtualenv operators. Compare approaches and configure DAGs for secure metadata ingestion.
|
||||
slug: /getting-started/day-1/hybrid-saas/mwaa
|
||||
collate: true
|
||||
---
|
||||
@ -35,7 +36,7 @@ To install the package, we need to update the `requirements.txt` file from the M
|
||||
openmetadata-ingestion[<plugin>]==x.y.z
|
||||
```
|
||||
|
||||
Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1.
|
||||
Where `x.y.z` is the version of the Collate ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1.
|
||||
|
||||
The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==1.3.1`.
|
||||
|
||||
@ -78,7 +79,7 @@ def metadata_ingestion_workflow():
|
||||
with DAG(
|
||||
"redshift_ingestion",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
description="An example DAG which runs a Collate ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
@ -293,7 +294,7 @@ with DAG(
|
||||
```
|
||||
|
||||
Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following
|
||||
the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values:
|
||||
the official Collate docs, and the value of the `pipelineType` configuration will need to hold one of the following values:
|
||||
|
||||
- `metadata`
|
||||
- `usage`
|
||||
@ -410,7 +411,7 @@ YAML config
|
||||
with DAG(
|
||||
"redshift_ingestion",
|
||||
default_args=default_args,
|
||||
description="An example DAG which runs a OpenMetadata ingestion workflow",
|
||||
description="An example DAG which runs a Collate ingestion workflow",
|
||||
start_date=days_ago(1),
|
||||
is_paused_upon_creation=False,
|
||||
catchup=False,
|
||||
@ -419,7 +420,7 @@ with DAG(
|
||||
task_id="ingest_redshift",
|
||||
python_callable=metadata_ingestion_workflow,
|
||||
requirements=['openmetadata-ingestion==1.0.5.0',
|
||||
'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with OpenMetadata's 'tabulate' requirements
|
||||
'apache-airflow==2.4.3', # note, v2.4.3 is the first version that does not conflict with Collate's 'tabulate' requirements
|
||||
'apache-airflow-providers-amazon==6.0.0', # Amazon Airflow provider is necessary for MWAA
|
||||
'watchtower',],
|
||||
system_site_packages=False,
|
||||
@ -430,7 +431,7 @@ with DAG(
|
||||
Where you can update the YAML configuration and workflow classes accordingly. accordingly. Further examples on how to
|
||||
run the ingestion can be found on the documentation (e.g., [Snowflake](/connectors/database/snowflake)).
|
||||
|
||||
You will also need to determine the OpenMetadata ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`.
|
||||
You will also need to determine the Collate ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`.
|
||||
For Airflow providers, you will want to pull the provider versions from [the matching constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt). Since this example installs Airflow Providers v2.4.3 on Python 3.7, we use that constraints file.
|
||||
|
||||
Also note that the ingestion workflow function must be entirely self-contained as it will run by itself in the virtualenv. Any imports it needs, including the configuration, must exist within the function itself.
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Day 1
|
||||
title: Day 1 Setup Guide | Connect, Ingest & Collaborate in Collate
|
||||
description: Start your Collate journey by connecting data sources, ingesting metadata, inviting users, and exploring key features. Everything you need for Day 1.
|
||||
slug: /getting-started/day-1
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Getting Started
|
||||
title: Getting Started with Collate | Setup & Onboarding Guide
|
||||
description: Quickly set up your Collate environment in 30 minutes. Learn how to connect data sources, explore features, and begin your onboarding journey.
|
||||
slug: /getting-started
|
||||
collate: true
|
||||
---
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Data Profiler | OpenMetadata Data Profiling Guide
|
||||
title: Data Profiler | `brandName` Data Profiling Guide
|
||||
description: Explore profiling workflows including histogram metrics, null counts, and field-level health.
|
||||
slug: /how-to-guides/data-quality-observability/profiler
|
||||
---
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Tests - UI Config | OpenMetadata Quality Config Guide
|
||||
title: Tests - UI Config | `brandName` Quality Config Guide
|
||||
description: Define UI tests to validate data quality during ingestion and enforce expectations at column or table level.
|
||||
slug: /how-to-guides/data-quality-observability/quality/tests-ui
|
||||
---
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user