198 Commits

Author SHA1 Message Date
Fredrik Sannholm
133577557c
feat(ingest): Looker view and dashboard ingestion (#2493) 2021-05-13 11:42:53 -07:00
Harshal Sheth
8d910b9e39
docs(ingest): improve kafka schema registry config docs (#2544) 2021-05-12 17:54:33 -07:00
Harshal Sheth
a671001824
refactor(ingest): move Airflow into datahub_provider module (#2521) 2021-05-12 15:01:11 -07:00
Albert Franzi
7fce505ffb
feat(ingest): define Redshift as a Postgres Source (#2540) 2021-05-12 10:00:34 -07:00
Harshal Sheth
2811d23e45
feat(ingest): add a transformer for adding ownership (#2532) 2021-05-11 17:46:39 -07:00
Harshal Sheth
36922c1746
docs: enable better syntax highlighting (#2529) 2021-05-11 15:16:12 -07:00
Harshal Sheth
50ea58c32d
docs: improve airflow explanations and examples (#2509) 2021-05-06 19:12:19 -07:00
Vincenzo Lavorini
3132910707
docs: fixed MCE file recipe example (#2505) 2021-05-06 19:05:57 -07:00
Harshal Sheth
1facfbd5a3
feat(ingest): capture table properties if available (#2497) 2021-05-05 14:07:08 -07:00
Harshal Sheth
c32bf494d5
fix(ingest): support https connections with cookies in Hive ingestion (#2489)
Tested locally.
2021-05-04 13:10:52 -07:00
Harshal Sheth
6f1f0a4845
feat(ingest): support hive over http (#2486) 2021-05-03 22:11:50 -07:00
Gabe Lyons
3bf5ffab5d
docs(ingest): Update README.md to add superset source (#2466) 2021-04-26 23:10:27 -07:00
Harshal Sheth
e075acaec9
docs(ingest): clarify options field and fix bigquery sample config (#2456) 2021-04-26 16:30:05 -07:00
Harshal Sheth
663dfe9a7c
fix(ingest): add snowflake warehouse and role to config (#2444) 2021-04-23 23:46:31 -07:00
Harshal Sheth
79daec29b7
fix(ingest): ensure upstreams in airflow lineage emission are entities (#2427) 2021-04-20 20:44:38 -07:00
Harshal Sheth
7d1ec520e5
fix(ingest): include database info for snowflake (#2426) 2021-04-20 20:40:30 -07:00
Thomas Larsson
7869a8f142
feature(ingestion): Adding the concept of transformers (#2411)
Fixes: #2410

Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-18 11:15:05 -07:00
Harshal Sheth
27a672e697
feat(ingest): add Kafka-based emitter example (#2412) 2021-04-16 17:54:50 -07:00
Harshal Sheth
fb6f74b1da
feat(ingest): add generic sqlalchemy source (#2389) 2021-04-13 08:01:38 -07:00
Harshal Sheth
41cd52f9e2
feat(ingest): add Airflow lineage backend (#2368) 2021-04-12 17:40:15 -07:00
Junqiu Zheng
cadcdd936c
fix(docs): fix config example for file sink (#2380) 2021-04-12 11:37:17 -07:00
Harshal Sheth
518de354d9
fix(ingest): support python3 -m datahub (#2359) 2021-04-07 14:58:58 -07:00
Harshal Sheth
cafdd9664a
fix(cli): add --verbose flag for datahub check plugins (#2345) 2021-04-06 15:41:15 -07:00
Harshal Sheth
cfc02ee196
feat(ingest): add Oracle db support (#2347) 2021-04-06 15:38:25 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00
Harshal Sheth
c1f3eaed35
fix(ingest): add support for database and table patterns to glue source (#2339) 2021-04-05 17:14:02 -07:00
amy m
759288161c
feat(ingest): adding support for AWS Glue (#2319)
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
2021-04-04 11:00:27 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes (#2306) 2021-03-26 21:57:05 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source (#2289) 2021-03-23 20:15:44 -07:00
Harshal Sheth
c42ce2e668
docs(ingest): add python versions badge (#2278) 2021-03-22 23:12:41 -07:00
Gary Lucas
922307f8ec
feat(ingest): Add DBT source to metadata-ingestion (#2247) 2021-03-22 23:11:29 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Pedro Silva
6a0c402a58
feat(ingest): Add support for druid (#2235) 2021-03-17 20:06:48 -07:00
Harshal Sheth
6a8fca59f1
feat(ingest): use plugin system based on Python extras (#2224) 2021-03-11 13:41:05 -08:00
Thomas Larsson
53be2078a9
refactor(ingest): improve athena source api and documentation (#2219)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 11:26:29 -08:00
Thomas Larsson
2b470c2d85
feat(ingest): add aws athena ingestion source (#2213)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 10:12:27 -08:00
Thomas Larsson
1f1518ce10
feat(ingest): add option for optimized skipping of schemas (#2209)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 00:08:01 -08:00
Harshal Sheth
6378347f81
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 14:32:12 -08:00
Harshal Sheth
0d374f30c5
docs(ingest): clarify setuptools requirement (#2177) 2021-03-05 12:18:13 -08:00
Harshal Sheth
20bf794ec4
docs: hosted documentation website (#2174) 2021-03-05 00:12:12 -08:00
Harshal Sheth
ec9c739306
docs(ingest): clarify Kafka connection config (#2171) 2021-03-04 11:42:57 -08:00
Harshal Sheth
e066991f54
fix(ingest): bigquery source and dataset naming fixes (#2161) 2021-03-03 19:49:46 -08:00
Harshal Sheth
656d9c771b
feat(ingest): support Postgres PostGIS extensions (#2136) 2021-02-25 09:05:16 -08:00
Harshal Sheth
b2fc27dbe8
docs(ingest): add solutions for common install issues (#2123) 2021-02-18 20:06:30 -08:00
Harshal Sheth
76e0594b8b
feat(ingest): add support for LDAP ingestion (#2122) 2021-02-18 20:05:39 -08:00
Harshal Sheth
02ffa6fd54
docs(ingest): add example DAGs for Airflow (#2116) 2021-02-17 18:01:22 -08:00
Harshal Sheth
fd4878a89d
docs(ingest): clarify docs for new ingestion framework (#2108) 2021-02-16 15:31:13 -08:00
Harshal Sheth
6e0c7ee37f Add architecture diagram 2021-02-15 18:29:27 -08:00
Harshal Sheth
6b750fc4fb Update docs 2021-02-15 18:29:27 -08:00
Harshal Sheth
38f75be8ad gometa -> datahub 2021-02-15 18:29:27 -08:00