3089 Commits

Author SHA1 Message Date
Harshal Sheth
c4d5f3288e
fix(ingest): support datahub --version (#2350) 2021-04-06 18:30:20 -07:00
Harshal Sheth
cafdd9664a
fix(cli): add --verbose flag for datahub check plugins (#2345) 2021-04-06 15:41:15 -07:00
Harshal Sheth
cfc02ee196
feat(ingest): add Oracle db support (#2347) 2021-04-06 15:38:25 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00
Harshal Sheth
c1f3eaed35
fix(ingest): add support for database and table patterns to glue source (#2339) 2021-04-05 17:14:02 -07:00
amy m
759288161c
feat(ingest): adding support for AWS Glue (#2319)
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
2021-04-04 11:00:27 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime (#2327) 2021-04-01 12:15:05 -07:00
Joe Mirizio
f3304bec7c
feat(ingest): dynamically register plugins (#2316)
Co-authored-by: Joe Mirizio <mirizioj@email.chop.edu>
2021-03-31 20:59:45 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes (#2306) 2021-03-26 21:57:05 -07:00
Harshal Sheth
7afe038a5c
fix(ingest): deal with breaking change from confluent_kafka (#2305) 2021-03-26 17:42:57 -07:00
Harshal Sheth
e067103e65
fix(cli): validate only required containers (#2304) 2021-03-26 16:55:03 -07:00
Harshal Sheth
07f4cb1199
feat: datahub check local-docker (#2295) 2021-03-26 10:03:51 -07:00
Pedro Silva
ca2a7fb527
fix(ingest): Add druid-specific identification logic (#2284) 2021-03-23 20:25:05 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source (#2289) 2021-03-23 20:15:44 -07:00
Gary Lucas
922307f8ec
feat(ingest): Add DBT source to metadata-ingestion (#2247) 2021-03-22 23:11:29 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Pedro Silva
6a0c402a58
feat(ingest): Add support for druid (#2235) 2021-03-17 20:06:48 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes (#2232) 2021-03-15 15:27:30 -07:00
Harshal Sheth
6a8fca59f1
feat(ingest): use plugin system based on Python extras (#2224) 2021-03-11 13:41:05 -08:00
Thomas Larsson
53be2078a9
refactor(ingest): improve athena source api and documentation (#2219)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 11:26:29 -08:00
Thomas Larsson
2b470c2d85
feat(ingest): add aws athena ingestion source (#2213)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 10:12:27 -08:00
Thomas Larsson
1f1518ce10
feat(ingest): add option for optimized skipping of schemas (#2209)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 00:08:01 -08:00
Harshal Sheth
4e15feebfd
fix(ingestion): nullable types and timestamp precision (#2212) 2021-03-10 23:29:24 -08:00
Harshal Sheth
6378347f81
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 14:32:12 -08:00
Harshal Sheth
d220647094
feat: add date and time types to SQL model (#2201) 2021-03-09 23:07:20 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. (#2164) 2021-03-07 11:26:47 -08:00
Harshal Sheth
1f082b114b
feat(ingest): bigquery sample data (#2178) 2021-03-05 16:39:34 -08:00
Harshal Sheth
ae9ffde8d0
fix(ingest): loosen Kafka broker validation (#2172) 2021-03-04 23:17:36 -08:00
Harshal Sheth
ec9c739306
docs(ingest): clarify Kafka connection config (#2171) 2021-03-04 11:42:57 -08:00
Harshal Sheth
e066991f54
fix(ingest): bigquery source and dataset naming fixes (#2161) 2021-03-03 19:49:46 -08:00
Harshal Sheth
0310e97b5b
feat(ingest): capture table descriptions (#2157) 2021-03-02 11:49:37 -08:00
Harshal Sheth
dced25fef7
feat(ingest): switch quickstart to Python ingestion (#2158) 2021-03-02 11:48:26 -08:00
Pedro Silva
6c7668115f
feat(ingest): support alternative authentication in sql ingestion (#2156) 2021-03-02 11:47:47 -08:00
Harshal Sheth
656d9c771b
feat(ingest): support Postgres PostGIS extensions (#2136) 2021-02-25 09:05:16 -08:00
Harshal Sheth
ac064584ae
refactor(ingest): cleanup configuration models (#2134) 2021-02-23 15:55:31 -08:00
Harshal Sheth
2a184dd3d5
fix(ingest): gracefully handle unknown types (#2125) 2021-02-19 19:57:13 -08:00
Harshal Sheth
76e0594b8b
feat(ingest): add support for LDAP ingestion (#2122) 2021-02-18 20:05:39 -08:00
Harshal Sheth
dfe00bfee8
feat(ingest): improve error reporting for pipelines (#2121) 2021-02-18 11:15:13 -08:00
Harshal Sheth
38f75be8ad gometa -> datahub 2021-02-15 18:29:27 -08:00
Harshal Sheth
0063c04460 gometa-ingest -> datahub ingest 2021-02-15 18:29:27 -08:00
Harshal Sheth
b91d0cf63b Add bigquery and refactor others 2021-02-15 18:29:27 -08:00
Harshal Sheth
cbbdf0930a Add snowflake 2021-02-15 18:29:27 -08:00
Harshal Sheth
d12497a3ff Add postgres source 2021-02-15 18:29:27 -08:00
Harshal Sheth
5666a50a7b Add hive 2021-02-15 18:29:27 -08:00
Harshal Sheth
8960f4b57d Remove codegen files 2021-02-15 18:29:27 -08:00
Harshal Sheth
08ed46eb69 Add docs for each source/sink 2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8 Use type + config everywhere 2021-02-15 18:29:27 -08:00
Harshal Sheth
9332e6b878 Add isort to CI 2021-02-15 18:29:27 -08:00
Harshal Sheth
92d1368f5e Disable ldap, enable mypy 2021-02-15 18:29:27 -08:00
Harshal Sheth
d483d23fd7 Allow/deny patterns for kafka source 2021-02-15 18:29:27 -08:00