mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-07-24 17:59:52 +00:00
876ac24e44
* GitBook: [#50] BigQuery, Glue, MSSQL, Postgres, Redshift, Snowflake - V2 * GitBook: [#62] No subject * GitBook: [#63] No subject * GitBook: [#64] Beta * GitBook: [#65] Make Harsha's requested changes to connectors section organization * GitBook: [#66] Kerberos authentication with Hive * GitBook: [#67] Fix procedure overview links * GitBook: [#68] Fix procedure overview links * GitBook: [#69] correct step reference * GitBook: [#70] Add Kerberos connection troubleshooting * updated json schema and schema docs (#3219) * updated json schema and schema docs * added glossay to readme * GitBook: [#72] Metrics & Tests Co-authored-by: Parth Panchal <parth.panchal@deuexsolutions.com> Co-authored-by: Shilpa V <vernekar.shilpa@gmail.com> Co-authored-by: Shannon Bradshaw <shannon.bradshaw@arrikto.com> Co-authored-by: parthp2107 <83201188+parthp2107@users.noreply.github.com> Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
1.9 KiB
1.9 KiB
| description |
|---|
| This guide will help install Kafka connector and run manually |
Kafka
{% hint style="info" %} Prerequisites
OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
- Python 3.7 or above {% endhint %}
Install from PyPI
{% tabs %} {% tab title="Install Using PyPI" %}
pip install 'openmetadata-ingestion[kafka]'
{% endtab %} {% endtabs %}
Run Manually
metadata ingest -c ./examples/workflows/confluent_kafka.json
Configuration
{% code title="confluent_kafka.json" %}
{
"source": {
"type": "kafka",
"config": {
"service_name": "local_kafka",
"bootstrap_servers": "192.168.1.32:9092",
"schema_registry_url": "http://192.168.1.32:8081",
"filter_pattern": {
"excludes": ["_confluent.*"]
}
}
},
...
{% endcode %}
- service_name - Service Name for this Kafka cluster. If you added Kafka cluster through OpenMetadata UI, make sure the service name matches the same.
- filter_pattern - It contains includes, excludes options to choose which pattern of datasets you want to ingest into OpenMetadata
Publish to OpenMetadata
Below is the configuration to publish Kafka data into the OpenMetadata service.
Add metadata-rest sink along with metadata-server config
{% code title="confluent_kafka.json" %}
{
"source": {
"type": "kafka",
"config": {
"service_name": "local_kafka",
"bootstrap_servers": "192.168.1.32:9092",
"schema_registry_url": "http://192.168.1.32:8081",
"filter_pattern": {
"excludes": ["_confluent.*"]
}
}
},
"sink": {
"type": "metadata-rest",
"config": {
}
},
"metadata_server": {
"type": "metadata-server",
"config": {
"api_endpoint": "http://localhost:8585/api",
"auth_provider_type": "no-auth"
}
}
}
{% endcode %}