yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-24 10:00:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/docs/how/data-source-onboarding.md
Mars Lan b0f56de7a8
build: build GitHub Page from /docs directory (#1750) 
- Move top-level MD files to /docs and symlink them back
- Update all absolute links to files in /docs to relative links
2020-07-26 10:22:35 -07:00

1.5 KiB
Raw Blame History

How to onboard a new data source?

In the metadata-ingestion, DataHub provides various kinds of metadata sources onboarding, including Hive, Kafka, LDAP, mySQL, and generic RDBMS as ETL scripts to feed the metadata to the GMS.

1. Extract

The extract process will be specific tight to the data source, hence, the data accessor should be able to reflect the correctness of the metadata from underlying data platforms.

2. Transform

In the transform stage, the extracted metadata should be encapsulated in a valid MetadataChangeEvent under the defined aspects and snapshots.

3. Load

The load part will leverage the Kafka producer to enable the pub-sub event-based ingestion. Meanwhile, the schema validation will be involved to check metadata quality.