Harshal Sheth
694c4ce8ec
test(ingest): fix mypy issue in schema util test ( #2465 )
2021-04-26 18:18:37 -07:00
Harshal Sheth
83fdc6417f
feat(ingest): capture default values in Avro schemas ( #2463 )
2021-04-26 17:07:29 -07:00
Harshal Sheth
d415234a8c
fix(ingest): fields with defaults should be optional ( #2461 )
2021-04-26 16:45:48 -07:00
Harshal Sheth
c64196e8cf
fix(ingest): setup pyproject.toml ( #2446 )
2021-04-23 23:50:28 -07:00
Harshal Sheth
a857d3b9d8
fix(ingest): various updates to datahub rest sink ( #2445 )
2021-04-23 23:48:44 -07:00
Harshal Sheth
663dfe9a7c
fix(ingest): add snowflake warehouse and role to config ( #2444 )
2021-04-23 23:46:31 -07:00
Gabe Lyons
851e00ba9f
feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way ( #2397 )
...
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Brian <brianwebtek@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2021-04-23 00:18:39 -07:00
adriaanslechten
1295c44615
feat(ingest) LDAP groups ingestion ( #2434 )
2021-04-22 13:56:30 -07:00
Harshal Sheth
034c33a050
fix(ingest): use entrypoints lib instead of pkg_resources ( #2438 )
2021-04-22 00:13:47 -07:00
Harshal Sheth
ffe49f061a
fix(ingest): fix chart type enum serialization and add tests for rest emitter ( #2429 )
2021-04-21 11:34:24 -07:00
Harshal Sheth
79daec29b7
fix(ingest): ensure upstreams in airflow lineage emission are entities ( #2427 )
2021-04-20 20:44:38 -07:00
Thomas Larsson
7869a8f142
feature(ingestion): Adding the concept of transformers ( #2411 )
...
Fixes : #2410
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-18 11:15:05 -07:00
Harshal Sheth
91a2f69310
fix(ingest): properly handle fieldDiscriminator with restli ( #2408 )
2021-04-16 09:42:52 -07:00
Harshal Sheth
ffe03e6758
fix(ingest): streamline codegen init methods ( #2400 )
2021-04-14 19:25:57 -07:00
Harshal Sheth
2af4603e49
fix(ingest): enable mypy disallow_incomplete_defs and disallow_untyped_decorators ( #2393 )
2021-04-14 13:40:24 -07:00
Harshal Sheth
a11329d5b8
refactor(ingest): update test harness to use a compose file per test ( #2392 )
2021-04-13 17:30:24 -07:00
Harshal Sheth
41cd52f9e2
feat(ingest): add Airflow lineage backend ( #2368 )
2021-04-12 17:40:15 -07:00
Thomas Larsson
6610666496
fix(ingestion): dont crash on non-RecordSchema topics ( #2372 )
...
Fixes : #2371
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-09 17:36:01 -07:00
Thomas Larsson
e02a17aecf
fix(ingestion): Support mapping from avro "boolean" and "map" types t… ( #2364 )
...
Fixes : #2363
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-08 14:23:12 -07:00
Thomas Larsson
4215dcd53c
fix(ingestion): properly detect optional fields in avro schemas ( #2343 )
...
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
2021-04-08 14:00:01 -07:00
Harshal Sheth
bfe345da42
fix(ingest): add test for avro serialization and deserialization ( #2351 )
2021-04-07 21:30:21 -07:00
Harshal Sheth
cafdd9664a
fix(cli): add --verbose flag for datahub check plugins ( #2345 )
2021-04-06 15:41:15 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders ( #2331 )
2021-04-05 19:11:28 -07:00
Harshal Sheth
c1f3eaed35
fix(ingest): add support for database and table patterns to glue source ( #2339 )
2021-04-05 17:14:02 -07:00
amy m
759288161c
feat(ingest): adding support for AWS Glue ( #2319 )
...
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
2021-04-04 11:00:27 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime ( #2327 )
2021-04-01 12:15:05 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes ( #2306 )
2021-03-26 21:57:05 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes ( #2246 )
2021-03-17 23:05:05 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes ( #2232 )
2021-03-15 15:27:30 -07:00
Harshal Sheth
6a8fca59f1
feat(ingest): use plugin system based on Python extras ( #2224 )
2021-03-11 13:41:05 -08:00
Harshal Sheth
6378347f81
feat(ingest): standalone metadata emitters ( #2207 )
2021-03-10 14:32:12 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. ( #2164 )
2021-03-07 11:26:47 -08:00
Harshal Sheth
ac064584ae
refactor(ingest): cleanup configuration models ( #2134 )
2021-02-23 15:55:31 -08:00
Harshal Sheth
dfe00bfee8
feat(ingest): improve error reporting for pipelines ( #2121 )
2021-02-18 11:15:13 -08:00
Harshal Sheth
38f75be8ad
gometa -> datahub
2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8
Use type + config everywhere
2021-02-15 18:29:27 -08:00
Harshal Sheth
9332e6b878
Add isort to CI
2021-02-15 18:29:27 -08:00
Harshal Sheth
d483d23fd7
Allow/deny patterns for kafka source
2021-02-15 18:29:27 -08:00
Harshal Sheth
df3e3da45b
More autofixes
2021-02-15 18:29:27 -08:00
Harshal Sheth
7ca018aaa4
Remove unused imports
2021-02-15 18:29:27 -08:00
Harshal Sheth
43d5fac494
Black
2021-02-15 18:29:27 -08:00
Harshal Sheth
1de2bacca4
Fix serde test
2021-02-15 18:29:27 -08:00
Harshal Sheth
107cf043d9
Add a create method to pipeline
2021-02-15 18:29:27 -08:00
Harshal Sheth
b2449eedaa
Add serde tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
a87161cad7
Run black formatting on tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
2307c59296
Add support for rich kafka config
2021-02-15 18:29:27 -08:00
Harshal Sheth
b759205b52
Start checking files for mysql test
2021-02-15 18:29:27 -08:00
Harshal Sheth
8ca8ef2d23
Fix kafka tests
2021-02-15 18:29:27 -08:00
Harshal Sheth
db1ff07e6d
Add reporter everywhere
2021-02-15 18:29:27 -08:00
Harshal Sheth
0929c7cb77
Add test to check that classes are not abstract
2021-02-15 18:29:27 -08:00
