yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-19 15:42:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,004 Commits 201 Branches 121 Tags
Commit Graph

935 Commits

Author SHA1 Message Date
Harshal Sheth
bfe345da42
fix(ingest): add test for avro serialization and deserialization (#2351) 2021-04-07 21:30:21 -07:00
Harshal Sheth
cafdd9664a
fix(cli): add --verbose flag for datahub check plugins (#2345) 2021-04-06 15:41:15 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00
Harshal Sheth
c1f3eaed35
fix(ingest): add support for database and table patterns to glue source (#2339) 2021-04-05 17:14:02 -07:00
amy m
759288161c
feat(ingest): adding support for AWS Glue (#2319) 
Co-authored-by: Harry Nash <harrywilliamnash@gmail.com>
 2021-04-04 11:00:27 -07:00
Harshal Sheth
cb24628886
feat(ingest): verify dynamic registry types at runtime (#2327) 2021-04-01 12:15:05 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes (#2306) 2021-03-26 21:57:05 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes (#2232) 2021-03-15 15:27:30 -07:00
Harshal Sheth
6a8fca59f1
feat(ingest): use plugin system based on Python extras (#2224) 2021-03-11 13:41:05 -08:00
Harshal Sheth
6378347f81
feat(ingest): standalone metadata emitters (#2207) 2021-03-10 14:32:12 -08:00
Gabe Lyons
adfe60e97a
feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend. (#2164) 2021-03-07 11:26:47 -08:00
Harshal Sheth
ac064584ae
refactor(ingest): cleanup configuration models (#2134) 2021-02-23 15:55:31 -08:00
Harshal Sheth
dfe00bfee8
feat(ingest): improve error reporting for pipelines (#2121) 2021-02-18 11:15:13 -08:00
Harshal Sheth
38f75be8ad gometa -> datahub 2021-02-15 18:29:27 -08:00
Harshal Sheth
f807bb44b8 Use type + config everywhere 2021-02-15 18:29:27 -08:00
Harshal Sheth
9332e6b878 Add isort to CI 2021-02-15 18:29:27 -08:00
Harshal Sheth
d483d23fd7 Allow/deny patterns for kafka source 2021-02-15 18:29:27 -08:00
Harshal Sheth
df3e3da45b More autofixes 2021-02-15 18:29:27 -08:00
Harshal Sheth
7ca018aaa4 Remove unused imports 2021-02-15 18:29:27 -08:00
Harshal Sheth
43d5fac494 Black 2021-02-15 18:29:27 -08:00
Harshal Sheth
1de2bacca4 Fix serde test 2021-02-15 18:29:27 -08:00
Harshal Sheth
107cf043d9 Add a create method to pipeline 2021-02-15 18:29:27 -08:00
Harshal Sheth
b2449eedaa Add serde tests 2021-02-15 18:29:27 -08:00
Harshal Sheth
a87161cad7 Run black formatting on tests 2021-02-15 18:29:27 -08:00
Harshal Sheth
2307c59296 Add support for rich kafka config 2021-02-15 18:29:27 -08:00
Harshal Sheth
b759205b52 Start checking files for mysql test 2021-02-15 18:29:27 -08:00
Harshal Sheth
8ca8ef2d23 Fix kafka tests 2021-02-15 18:29:27 -08:00
Harshal Sheth
db1ff07e6d Add reporter everywhere 2021-02-15 18:29:27 -08:00
Harshal Sheth
0929c7cb77 Add test to check that classes are not abstract 2021-02-15 18:29:27 -08:00
Harshal Sheth
22a4dbb45f Cleanup kafka sink schema setup 2021-02-15 18:29:27 -08:00
Harshal Sheth
4b83fc6591 adding allow deny patterns to sql config 2021-02-15 18:29:27 -08:00
Shirshanka Das
d73d8ff4c2 fixing unit tests 2021-02-15 18:29:27 -08:00
Harshal Sheth
c7892ada4c Codegen avro + datahub kafka sink (#3) 
* Add codegen

* New architecture + setup file -> console pipeline

* Cleanup source loader

* Basic Kafka metadata source

* Kafka source and extractor

* Add kwargs construct interface

* Fix kafka source unit test

* start working on pipeline test

* kafka datahub sink

* Make myself a profile

* Ingest to datahub from kafka

* Update codegen

* Add restli transport

* Fix bug in restli conversion
 2021-02-15 18:29:27 -08:00
Shirshanka Das
9e61220132 checking in testing fixtures. docker still not working 2021-02-15 18:29:27 -08:00