yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-01 22:05:47 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-ingestion/docs/sources/file-based-lineage/datahub-lineage-file.md

1.3 KiB
Raw Blame History

Lineage File Format

The lineage source file should be a .yml file with the following top-level keys:

version: the version of lineage file config the config conforms to. Currently, the only version released is 1.

lineage: the top level key of the lineage file containing a list of EntityNodeConfig objects

EntityNodeConfig:

  • entity: EntityConfig object
  • upstream: (optional) list of child EntityNodeConfig objects

EntityConfig:

  • name : identifier of the entity. Typically name or guid, as used in constructing entity urn.
  • type: type of the entity (only dataset is supported as of now)
  • env: the environment of this entity. Should match the values in the table here
  • platform: a valid platform like kafka, snowflake, etc..
  • platform_instance: optional string specifying the platform instance of this entity

For example if dataset URN is urn:li:dataset:(urn:li:dataPlatform:redshift,userdb.public.customer_table,DEV) then EntityConfig will look like:

name : userdb.public.customer_table
type: dataset
env: DEV
platform: redshift

You can also view an example lineage file checked in here