datahub/metadata-ingestion/examples/recipes/kafka_to_datahub.yml

---
source:
  type: "kafka"
  config:
    connection.bootstrap: "localhost:9092"
    topic_patterns:
      allow:
        - ".*"
      deny:
        - "^_.+" # deny all tables that start with an underscore

sink:
  type: "datahub-kafka"
  config:
    connection.bootstrap: "localhost:9092"
Firstdrop of ingest (#1) 2021-01-31 22:40:30 -08:00			`---`
			`source:`
			`type: "kafka"`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`config:`
Firstdrop of ingest (#1) 2021-01-31 22:40:30 -08:00			`connection.bootstrap: "localhost:9092"`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`topic_patterns:`
			`allow:`
			`- ".*"`
			`deny:`
			`- "^_.+" # deny all tables that start with an underscore`
Firstdrop of ingest (#1) 2021-01-31 22:40:30 -08:00
			`sink:`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`type: "datahub-kafka"`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`config:`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`connection.bootstrap: "localhost:9092"`