datahub/metadata-ingestion/examples/recipes/kafka_to_datahub_rest.dhub.yaml

---
# see https://datahubproject.io/docs/generated/ingestion/sources/kafka for complete documentation
source:
  type: "kafka"
  config:
    connection:
      bootstrap: "localhost:9092"

# see https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub for complete documentation
sink:
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`---`
fix(doc): update doc url to generated docs (#4860) 2022-05-13 10:19:46 +05:30			`# see https://datahubproject.io/docs/generated/ingestion/sources/kafka for complete documentation`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`source:`
			`type: "kafka"`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`config:`
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 17:32:12 -05:00			`connection:`
feat: Adding support for nested schemas in ingestion and visualization (#3079) 2021-08-11 15:47:18 -07:00			`bootstrap: "localhost:9092"`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
docs(ingest): link to docs from recipes (#3058) 2021-08-09 13:00:47 -04:00			`# see https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub for complete documentation`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`sink:`
			`type: "datahub-rest"`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`config:`
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 17:32:12 -05:00			`server: "http://localhost:8080"`