From b2aab148fa6880d11b81a7331ffbe25bad4564ca Mon Sep 17 00:00:00 2001 From: Mayur Singal <39544459+ulixius9@users.noreply.github.com> Date: Tue, 21 Feb 2023 17:26:50 +0530 Subject: [PATCH] Kafka Schema Registry Note (#10255) --- .../content/connectors/messaging/kafka/airflow.md | 7 +++++++ .../content/connectors/messaging/kafka/cli.md | 6 ++++++ .../content/connectors/messaging/kafka/index.md | 6 ++++++ .../content/connectors/messaging/redpanda/airflow.md | 6 ++++++ .../content/connectors/messaging/redpanda/cli.md | 6 ++++++ .../content/connectors/messaging/redpanda/index.md | 6 ++++++ 6 files changed, 37 insertions(+) diff --git a/openmetadata-docs/content/connectors/messaging/kafka/airflow.md b/openmetadata-docs/content/connectors/messaging/kafka/airflow.md index 5762693c03e..8207e352701 100644 --- a/openmetadata-docs/content/connectors/messaging/kafka/airflow.md +++ b/openmetadata-docs/content/connectors/messaging/kafka/airflow.md @@ -83,6 +83,13 @@ workflowConfig: - **consumerConfig**: Confluent Kafka Consumer Config. - **schemaRegistryConfig**:Confluent Kafka Schema Registry Config. + + + +To ingest the topic schema `schemaRegistryURL` must be passed + + + #### Source Configuration - Source Config The sourceConfig is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/messagingServiceMetadataPipeline.json): diff --git a/openmetadata-docs/content/connectors/messaging/kafka/cli.md b/openmetadata-docs/content/connectors/messaging/kafka/cli.md index 1f83872f38d..dc303de511f 100644 --- a/openmetadata-docs/content/connectors/messaging/kafka/cli.md +++ b/openmetadata-docs/content/connectors/messaging/kafka/cli.md @@ -83,6 +83,12 @@ workflowConfig: - **consumerConfig**: Confluent Kafka Consumer Config. - **schemaRegistryConfig**:Confluent Kafka Schema Registry Config. + + +To ingest the topic schema `schemaRegistryURL` must be passed + + + #### Source Configuration - Source Config The sourceConfig is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/messagingServiceMetadataPipeline.json): diff --git a/openmetadata-docs/content/connectors/messaging/kafka/index.md b/openmetadata-docs/content/connectors/messaging/kafka/index.md index fc375232832..41286373baa 100644 --- a/openmetadata-docs/content/connectors/messaging/kafka/index.md +++ b/openmetadata-docs/content/connectors/messaging/kafka/index.md @@ -134,6 +134,12 @@ the changes. - **Consumer Config**: Confluent Kafka Consumer Config. - **Schema Registry Config**:Confluent Kafka Schema Registry Config. + + +To ingest the topic schema `Schema Registry URL` must be passed + + + ### 6. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline, diff --git a/openmetadata-docs/content/connectors/messaging/redpanda/airflow.md b/openmetadata-docs/content/connectors/messaging/redpanda/airflow.md index fda9afbb8c7..eaa88a1f48c 100644 --- a/openmetadata-docs/content/connectors/messaging/redpanda/airflow.md +++ b/openmetadata-docs/content/connectors/messaging/redpanda/airflow.md @@ -83,6 +83,12 @@ workflowConfig: - **consumerConfig**: Redpanda Consumer Config. - **schemaRegistryConfig**: Redpanda Schema Registry Config. + + +To ingest the topic schema `schemaRegistryURL` must be passed + + + #### Source Configuration - Source Config The sourceConfig is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/messagingServiceMetadataPipeline.json): diff --git a/openmetadata-docs/content/connectors/messaging/redpanda/cli.md b/openmetadata-docs/content/connectors/messaging/redpanda/cli.md index bb46b693509..29d24b7a648 100644 --- a/openmetadata-docs/content/connectors/messaging/redpanda/cli.md +++ b/openmetadata-docs/content/connectors/messaging/redpanda/cli.md @@ -83,6 +83,12 @@ workflowConfig: - **consumerConfig**: Redpanda Consumer Config. - **schemaRegistryConfig**: Redpanda Schema Registry Config. + + +To ingest the topic schema `schemaRegistryURL` must be passed + + + #### Source Configuration - Source Config The sourceConfig is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/messagingServiceMetadataPipeline.json): diff --git a/openmetadata-docs/content/connectors/messaging/redpanda/index.md b/openmetadata-docs/content/connectors/messaging/redpanda/index.md index 7eeee49cf8a..933f7f168ee 100644 --- a/openmetadata-docs/content/connectors/messaging/redpanda/index.md +++ b/openmetadata-docs/content/connectors/messaging/redpanda/index.md @@ -134,6 +134,12 @@ the changes. - **Consumer Config**: Redpanda Consumer Config. - **Schema Registry Config**: Redpanda Schema Registry Config. + + +To ingest the topic schema `Schema Registry URL` must be passed + + + ### 6. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline,