3975 Commits

Author SHA1 Message Date
Thomas Larsson
6610666496
fix(ingestion): dont crash on non-RecordSchema topics (#2372)
Fixes: #2371

Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-09 17:36:01 -07:00
Thomas Larsson
e02a17aecf
fix(ingestion): Support mapping from avro "boolean" and "map" types t… (#2364)
Fixes: #2363

Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-08 14:23:12 -07:00
Thomas Larsson
4215dcd53c
fix(ingestion): properly detect optional fields in avro schemas (#2343)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-08 14:00:01 -07:00
Harshal Sheth
bfe345da42
fix(ingest): add test for avro serialization and deserialization (#2351) 2021-04-07 21:30:21 -07:00
Harshal Sheth
e29082bf55
feat(cli): Add support for checking docker memory usage (#2361) 2021-04-07 16:26:21 -07:00
Harshal Sheth
518de354d9
fix(ingest): support python3 -m datahub (#2359) 2021-04-07 14:58:58 -07:00
Harshal Sheth
455c078773
chore(ingest): remove unused plugin_requirements.txt file (#2353) 2021-04-06 18:37:39 -07:00
Harshal Sheth
b0d8f70354
fix(ingest): bump pybigquery version (#2352) 2021-04-06 18:34:06 -07:00
Harshal Sheth
c4d5f3288e
fix(ingest): support datahub --version (#2350) 2021-04-06 18:30:20 -07:00
Harshal Sheth
cafdd9664a
fix(cli): add --verbose flag for datahub check plugins (#2345) 2021-04-06 15:41:15 -07:00
Harshal Sheth
cfc02ee196
feat(ingest): add Oracle db support (#2347) 2021-04-06 15:38:25 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00
Harshal Sheth
c1f3eaed35
fix(ingest): add support for database and table patterns to glue source (#2339) 2021-04-05 17:14:02 -07:00
amy m
759288161c
feat(ingest): adding support for AWS Glue (#2319)
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
2021-04-04 11:00:27 -07:00
Gabe Lyons
8ab2630fb6
feat(lineage): expandable lineage visualization for dataset <> dataset lineage (#2333) 2021-04-03 11:13:25 -07:00
Gary Lucas
9d00e8bb2e
fix(test): dbt-manifest files (#2330) 2021-04-02 10:59:20 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime (#2327) 2021-04-01 12:15:05 -07:00
Joe Mirizio
f3304bec7c
feat(ingest): dynamically register plugins (#2316)
Co-authored-by: Joe Mirizio <mirizioj@email.chop.edu>
2021-03-31 20:59:45 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes (#2306) 2021-03-26 21:57:05 -07:00
Harshal Sheth
7afe038a5c
fix(ingest): deal with breaking change from confluent_kafka (#2305) 2021-03-26 17:42:57 -07:00
Harshal Sheth
e067103e65
fix(cli): validate only required containers (#2304) 2021-03-26 16:55:03 -07:00
Harshal Sheth
07f4cb1199
feat: datahub check local-docker (#2295) 2021-03-26 10:03:51 -07:00
Harshal Sheth
cc19465f55
fix(ingest): resolve array serialization bug (#2290) 2021-03-24 10:02:46 -07:00
Pedro Silva
ca2a7fb527
fix(ingest): Add druid-specific identification logic (#2284) 2021-03-23 20:25:05 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source (#2289) 2021-03-23 20:15:44 -07:00
Harshal Sheth
1ea450e0e4
fix(ingest): use custom pybigquery ref to get descriptions (#2279) 2021-03-22 23:17:54 -07:00
Harshal Sheth
c42ce2e668
docs(ingest): add python versions badge (#2278) 2021-03-22 23:12:41 -07:00
Gary Lucas
922307f8ec
feat(ingest): Add DBT source to metadata-ingestion (#2247) 2021-03-22 23:11:29 -07:00
Dexter Lee
61c37aa04d
refactor(demo): Add empty global tags to BigQuery demo data (#2258) 2021-03-18 16:03:05 -07:00
Gabe Lyons
039fe597f7
feat(tags): editing tags from react client on datasets, schemas, charts & dashboards (#2248) 2021-03-18 11:52:14 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Pedro Silva
6a0c402a58
feat(ingest): Add support for druid (#2235) 2021-03-17 20:06:48 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes (#2232) 2021-03-15 15:27:30 -07:00
Arun Vasudevan
7750c6120a
feat: MLmodel Graphql Query (#2166) 2021-03-13 08:34:48 -08:00
Harshal Sheth
95c124ffc4
fix(ingest): pin version of avro-gen3 (#2230) 2021-03-12 09:39:38 -08:00
Harshal Sheth
6a8fca59f1
feat(ingest): use plugin system based on Python extras (#2224) 2021-03-11 13:41:05 -08:00
Thomas Larsson
53be2078a9
refactor(ingest): improve athena source api and documentation (#2219)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 11:26:29 -08:00
Thomas Larsson
2b470c2d85
feat(ingest): add aws athena ingestion source (#2213)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 10:12:27 -08:00
Thomas Larsson
1f1518ce10
feat(ingest): add option for optimized skipping of schemas (#2209)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-03-11 00:08:01 -08:00
Harshal Sheth
4e15feebfd
fix(ingestion): nullable types and timestamp precision (#2212) 2021-03-10 23:29:24 -08:00
Harshal Sheth
6378347f81
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 14:32:12 -08:00
Harshal Sheth
d220647094
feat: add date and time types to SQL model (#2201) 2021-03-09 23:07:20 -08:00
Gabe Lyons
0046aae7cf
feat(tags): Enriching sample data for tags (#2194) 2021-03-08 17:26:41 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. (#2164) 2021-03-07 11:26:47 -08:00
Harshal Sheth
1f082b114b
feat(ingest): bigquery sample data (#2178) 2021-03-05 16:39:34 -08:00
Harshal Sheth
0d374f30c5
docs(ingest): clarify setuptools requirement (#2177) 2021-03-05 12:18:13 -08:00
Harshal Sheth
20bf794ec4
docs: hosted documentation website (#2174) 2021-03-05 00:12:12 -08:00
Harshal Sheth
ae9ffde8d0
fix(ingest): loosen Kafka broker validation (#2172) 2021-03-04 23:17:36 -08:00
Harshal Sheth
ec9c739306
docs(ingest): clarify Kafka connection config (#2171) 2021-03-04 11:42:57 -08:00
Harshal Sheth
e066991f54
fix(ingest): bigquery source and dataset naming fixes (#2161) 2021-03-03 19:49:46 -08:00