Harshal Sheth
|
91a2f69310
|
fix(ingest): properly handle fieldDiscriminator with restli (#2408)
|
2021-04-16 09:42:52 -07:00 |
|
Harshal Sheth
|
ffe03e6758
|
fix(ingest): streamline codegen init methods (#2400)
|
2021-04-14 19:25:57 -07:00 |
|
Harshal Sheth
|
2af4603e49
|
fix(ingest): enable mypy disallow_incomplete_defs and disallow_untyped_decorators (#2393)
|
2021-04-14 13:40:24 -07:00 |
|
Harshal Sheth
|
a11329d5b8
|
refactor(ingest): update test harness to use a compose file per test (#2392)
|
2021-04-13 17:30:24 -07:00 |
|
Harshal Sheth
|
41cd52f9e2
|
feat(ingest): add Airflow lineage backend (#2368)
|
2021-04-12 17:40:15 -07:00 |
|
Thomas Larsson
|
6610666496
|
fix(ingestion): dont crash on non-RecordSchema topics (#2372)
Fixes: #2371
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
|
2021-04-09 17:36:01 -07:00 |
|
Thomas Larsson
|
e02a17aecf
|
fix(ingestion): Support mapping from avro "boolean" and "map" types t… (#2364)
Fixes: #2363
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
|
2021-04-08 14:23:12 -07:00 |
|
Thomas Larsson
|
4215dcd53c
|
fix(ingestion): properly detect optional fields in avro schemas (#2343)
Co-authored-by: thomas.larsson <thomas.larsson@klarna.com>
|
2021-04-08 14:00:01 -07:00 |
|
Harshal Sheth
|
bfe345da42
|
fix(ingest): add test for avro serialization and deserialization (#2351)
|
2021-04-07 21:30:21 -07:00 |
|
Harshal Sheth
|
cafdd9664a
|
fix(cli): add --verbose flag for datahub check plugins (#2345)
|
2021-04-06 15:41:15 -07:00 |
|
Harshal Sheth
|
bd78b84bd3
|
feat(ingest): start airflow integration + metadata builders (#2331)
|
2021-04-05 19:11:28 -07:00 |
|
Harshal Sheth
|
c1f3eaed35
|
fix(ingest): add support for database and table patterns to glue source (#2339)
|
2021-04-05 17:14:02 -07:00 |
|
amy m
|
759288161c
|
feat(ingest): adding support for AWS Glue (#2319)
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
|
2021-04-04 11:00:27 -07:00 |
|
Harshal Sheth
|
cb24628886
|
feat(ingest): verify dynamic registry types at runtime (#2327)
|
2021-04-01 12:15:05 -07:00 |
|
Harshal Sheth
|
f57c954fc6
|
feat(ingest): support environment variables in recipes (#2306)
|
2021-03-26 21:57:05 -07:00 |
|
Harshal Sheth
|
b8462028c3
|
feat(ingest): various minor fixes (#2246)
|
2021-03-17 23:05:05 -07:00 |
|
Harshal Sheth
|
aa6bc15cd7
|
fix(ingest): various avro codegen fixes (#2232)
|
2021-03-15 15:27:30 -07:00 |
|
Harshal Sheth
|
6a8fca59f1
|
feat(ingest): use plugin system based on Python extras (#2224)
|
2021-03-11 13:41:05 -08:00 |
|
Harshal Sheth
|
6378347f81
|
feat(ingest): standalone metadata emitters (#2207)
|
2021-03-10 14:32:12 -08:00 |
|
Gabe Lyons
|
adfe60e97a
|
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. (#2164)
|
2021-03-07 11:26:47 -08:00 |
|
Harshal Sheth
|
ac064584ae
|
refactor(ingest): cleanup configuration models (#2134)
|
2021-02-23 15:55:31 -08:00 |
|
Harshal Sheth
|
dfe00bfee8
|
feat(ingest): improve error reporting for pipelines (#2121)
|
2021-02-18 11:15:13 -08:00 |
|
Harshal Sheth
|
38f75be8ad
|
gometa -> datahub
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
f807bb44b8
|
Use type + config everywhere
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
9332e6b878
|
Add isort to CI
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
d483d23fd7
|
Allow/deny patterns for kafka source
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
df3e3da45b
|
More autofixes
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
7ca018aaa4
|
Remove unused imports
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
43d5fac494
|
Black
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
1de2bacca4
|
Fix serde test
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
107cf043d9
|
Add a create method to pipeline
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
b2449eedaa
|
Add serde tests
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
a87161cad7
|
Run black formatting on tests
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
2307c59296
|
Add support for rich kafka config
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
b759205b52
|
Start checking files for mysql test
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
8ca8ef2d23
|
Fix kafka tests
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
db1ff07e6d
|
Add reporter everywhere
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
0929c7cb77
|
Add test to check that classes are not abstract
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
22a4dbb45f
|
Cleanup kafka sink schema setup
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
4b83fc6591
|
adding allow deny patterns to sql config
|
2021-02-15 18:29:27 -08:00 |
|
Shirshanka Das
|
d73d8ff4c2
|
fixing unit tests
|
2021-02-15 18:29:27 -08:00 |
|
Harshal Sheth
|
c7892ada4c
|
Codegen avro + datahub kafka sink (#3)
* Add codegen
* New architecture + setup file -> console pipeline
* Cleanup source loader
* Basic Kafka metadata source
* Kafka source and extractor
* Add kwargs construct interface
* Fix kafka source unit test
* start working on pipeline test
* kafka datahub sink
* Make myself a profile
* Ingest to datahub from kafka
* Update codegen
* Add restli transport
* Fix bug in restli conversion
|
2021-02-15 18:29:27 -08:00 |
|
Shirshanka Das
|
9e61220132
|
checking in testing fixtures. docker still not working
|
2021-02-15 18:29:27 -08:00 |
|