datahub/metadata-ingestion/examples/recipes/file_to_datahub.dhub.yaml

---
# see https://docs.datahub.com/docs/generated/ingestion/sources/file for complete documentation
source:
  type: "file"
  config:
    filename: ./examples/mce_files/nih_demo.json

# see https://docs.datahub.com/docs/metadata-ingestion/sink_docs/datahub for complete documentation
sink:
  type: datahub-kafka
  config:
    connection:
      schema_registry_url: http://localhost:8081
      bootstrap: localhost:9092
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`---`
doc: Acryl to DataHub, datahubproject.io to datahub.com (#13252) Co-authored-by: Jay <159848059+jayacryl@users.noreply.github.com> 2025-04-28 23:34:33 +09:00			`# see https://docs.datahub.com/docs/generated/ingestion/sources/file for complete documentation`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`source:`
			`type: "file"`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`config:`
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983) Co-authored-by: Dexter Lee <dexter@acryl.io> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> Co-authored-by: Ravindra Lanka <rlanka@acryl.io> Co-authored-by: Shirshanka Das <shirshanka@apache.org> 2021-07-30 17:41:03 -07:00			`filename: ./examples/mce_files/nih_demo.json`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
doc: Acryl to DataHub, datahubproject.io to datahub.com (#13252) Co-authored-by: Jay <159848059+jayacryl@users.noreply.github.com> 2025-04-28 23:34:33 +09:00			`# see https://docs.datahub.com/docs/metadata-ingestion/sink_docs/datahub for complete documentation`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`sink:`
feat(docs): Updating example files with the new ingestion recipe suffix (#5103) 2022-06-08 00:52:26 +02:00			`type: datahub-kafka`
Use type + config everywhere 2021-02-12 12:05:41 -08:00			`config:`
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 17:32:12 -05:00			`connection:`
feat(docs): Updating example files with the new ingestion recipe suffix (#5103) 2022-06-08 00:52:26 +02:00			`schema_registry_url: http://localhost:8081`
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 17:32:12 -05:00			`bootstrap: localhost:9092`