mirror of
https://github.com/datahub-project/datahub.git
synced 2025-07-07 09:11:47 +00:00
2.9 KiB
2.9 KiB
Lineage sample code
The following samples will cover emitting dataset-to-dataset, dataset-to-job-to-dataset, chart-to-dataset, dashboard-to-chart and job-to-dataflow lineages.
- lineage_emitter_mcpw_rest.py - emits simple bigquery table-to-table (dataset-to-dataset) lineage via REST as MetadataChangeProposalWrapper.
- lineage_dataset_job_dataset.py - emits mysql-to-airflow-to-kafka (dataset-to-job-to-dataset) lineage via REST as MetadataChangeProposalWrapper.
- lineage_dataset_chart.py - emits the dataset-to-chart lineage via REST as MetadataChangeProposalWrapper.
- lineage_chart_dashboard.py - emits the chart-to-dashboard lineage via REST as MetadataChangeProposalWrapper.
- lineage_job_dataflow.py - emits the job-to-dataflow lineage via REST as MetadataChangeProposalWrapper.
- lineage_emitter_rest.py - emits simple dataset-to-dataset lineage via REST as MetadataChangeEvent.
- lineage_emitter_kafka.py - emits simple dataset-to-dataset lineage via Kafka as MetadataChangeEvent.
- lineage_emitter_dataset_finegrained.py - emits fine-grained dataset-dataset lineage via REST as MetadataChangeProposalWrapper.
- lineage_emitter_datajob_finegrained.py - emits fine-grained datajob-dataset lineage via REST as MetadataChangeProposalWrapper.
- Datahub Snowflake Lineage - emits Datahub's Snowflake lineage as MetadataChangeProposalWrapper.
- Datahub Bigquery Lineage - emits Datahub's Bigquery lineage as MetadataChangeProposalWrapper.
- Datahub Dbt Lineage - emits Datahub's DBT lineage as MetadataChangeEvent.
NOTE:
- Emitting aspects as MetadataChangeProposalWrapper is recommended over emitting aspects via the MetadataChangeEvent.
- Emitting any aspect associated with an entity completely overwrites the previous value of the aspect associated with the entity. This means that emitting a lineage aspect associated with a dataset will overwrite lineage edges that already exist.