mirror of https://github.com/datahub-project/datahub.git synced 2025-09-09 17:20:42 +00:00

History

fix: correct the way to catch the exception (#1727 )

* fix: modify the etl script dependency

* fix: Correct the way to catch the exception

* fix: Compatible with the following kafka cluster when the Kafka Topic message Key cannot be empty

* fix: Adjust the kafka message key; Improve the comment of field

* fix: Avro schema required for key

Co-authored-by: Cobolbaby <Zhang.Xing-Long@inventec.com>

2020-07-10 07:56:19 -07:00

bigquery_etl.py

feature(ingest): add bigquery ETL script (#1711 )

2020-06-25 15:28:13 -07:00

bigquery_etl.txt

feature(ingest): add bigquery ETL script (#1711 )

2020-06-25 15:28:13 -07:00

common.py

fix: correct the way to catch the exception (#1727 )

2020-07-10 07:56:19 -07:00

common.txt

fix: modify the etl script dependency (#1726 )

2020-07-08 21:51:42 -07:00

hive_etl.py

feat(ingest): replace custom hive-etl with sql-based ETL (#1713 )

2020-06-25 19:04:56 -07:00

hive_etl.txt

feat(ingest): replace custom hive-etl with sql-based ETL (#1713 )

2020-06-25 19:04:56 -07:00

hive.env

feat(ingest): replace custom hive-etl with sql-based ETL (#1713 )

2020-06-25 19:04:56 -07:00

hive.yml

feat(ingest): replace custom hive-etl with sql-based ETL (#1713 )

2020-06-25 19:04:56 -07:00

mysql_etl.py

feature(ingest): add bigquery ETL script (#1711 )

2020-06-25 15:28:13 -07:00

mysql_etl.txt

feature(etl): add SQLAlchemy-based ingestion script (#1708 )

2020-06-22 21:25:55 -07:00

postgres_etl.py

feat(ingest): add PostgreSQL ETL script (#1712 )

2020-06-25 15:28:42 -07:00

postgres_etl.txt

feat(ingest): add PostgreSQL ETL script (#1712 )

2020-06-25 15:28:42 -07:00

postgres.yml

feat(ingest): add PostgreSQL ETL script (#1712 )

2020-06-25 15:28:42 -07:00

README.md

Update README.md

2020-06-25 16:32:22 -07:00

snowflake_etl.py

feat(ingest): add snowflake ETL script (#1714 )

2020-06-25 19:05:38 -07:00

snowflake.txt

feat(ingest): add snowflake ETL script (#1714 )

2020-06-25 19:05:38 -07:00

README.md

SQL-Based Metadata Ingestion

This directory contains example ETL scripts that use SQLAlchemy to ingest basic metadata from a wide range of commonly used SQL-based data systems, including MySQL, PostgreSQL, Oracle, MS SQL, Redshift, BigQuery, Snowflake, etc.

Requirements

You'll need to install both the common requirements (common.txt) and the system-specific driver for the script (e.g. mysql_etl.txt for mysql_etl.py). Some drivers also require additional dependencies to be installed so please check the driver's official project page for more details.

Example

Here's an example on how to ingest metadata from MySQL.

Install requirements

pip install --user -r common.txt -r mysql_etl.txt

Modify these variables in mysql_etl.py to match your environment

URL       # Connection URL in the form of mysql+pymysql://username:password@hostname:port
OPTIONS   # Additional conenction options for the driver

Run the ETL script

python mysql_etl.py