yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-12 03:07:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,789 Commits 219 Branches 123 Tags
Commit Graph

3089 Commits

Author SHA1 Message Date
Harshal Sheth
df3e3da45b More autofixes 2021-02-15 18:29:27 -08:00
Harshal Sheth
7ca018aaa4 Remove unused imports 2021-02-15 18:29:27 -08:00
Harshal Sheth
43d5fac494 Black 2021-02-15 18:29:27 -08:00
Harshal Sheth
107cf043d9 Add a create method to pipeline 2021-02-15 18:29:27 -08:00
Harshal Sheth
648f53b553 Fix serde capabilities 2021-02-15 18:29:27 -08:00
Harshal Sheth
ae88fbf727 Streamline serde 2021-02-15 18:29:27 -08:00
Harshal Sheth
0660991fb8 More python 3.6 compat 2021-02-15 18:29:27 -08:00
Harshal Sheth
42320081e1 Python 3.6 compat 2021-02-15 18:29:27 -08:00
Harshal Sheth
2307c59296 Add support for rich kafka config 2021-02-15 18:29:27 -08:00
Harshal Sheth
36874970ce Finish file sink 2021-02-15 18:29:27 -08:00
Harshal Sheth
75b01806ef Use find_packages() in setup 2021-02-15 18:29:27 -08:00
Harshal Sheth
becc953f6e Remove dead code - DynamicFactory 2021-02-15 18:29:27 -08:00
Harshal Sheth
fcd13000ef Throw error when aspects are empty 2021-02-15 18:29:27 -08:00
Harshal Sheth
d801c3e1c4 Handle missing kafka schemas elegantly 2021-02-15 18:29:27 -08:00
Harshal Sheth
8ca8ef2d23 Fix kafka tests 2021-02-15 18:29:27 -08:00
Harshal Sheth
82c29d3c94 Properly extract kafka schema 2021-02-15 18:29:27 -08:00
Harshal Sheth
affd5728e6 Integrate kafka extract into source 2021-02-15 18:29:27 -08:00
Harshal Sheth
1e06bda07c Fix dataclass mypy error 2021-02-15 18:29:27 -08:00
Harshal Sheth
eaac303610 Reduce mypy errors 2021-02-15 18:29:27 -08:00
Harshal Sheth
40b8e5e158 Fixes in pipeline 2021-02-15 18:29:27 -08:00
Harshal Sheth
db1ff07e6d Add reporter everywhere 2021-02-15 18:29:27 -08:00
Harshal Sheth
fd9bc09e67 Start adding reporting 2021-02-15 18:29:27 -08:00
Shirshanka Das
4e17a24dd4 Adding mysql integration test + source 2021-02-15 18:29:27 -08:00
Harshal Sheth
22a4dbb45f Cleanup kafka sink schema setup 2021-02-15 18:29:27 -08:00
Harshal Sheth
d68648670a Add back line that I accidentally removed when testing 2021-02-15 18:29:27 -08:00
Harshal Sheth
4b83fc6591 adding allow deny patterns to sql config 2021-02-15 18:29:27 -08:00
Harshal Sheth
62bb7f012f Quick readme updates 2021-02-15 18:29:27 -08:00
Harshal Sheth
e09528909f python 3.7 compat 2021-02-15 18:29:27 -08:00
Shirshanka Das
7b12fc9827 support for multiple mce-s in a file 2021-02-15 18:29:27 -08:00
Harshal Sheth
58554725aa Add bootstrap mce 2021-02-15 18:29:27 -08:00
Harshal Sheth
4dc8936f27 Add mypy type checks 2021-02-15 18:29:27 -08:00
Shirshanka Das
327705fd0d MS-SQL integ test works 2021-02-15 18:29:27 -08:00
Shirshanka Das
d73d8ff4c2 fixing unit tests 2021-02-15 18:29:27 -08:00
Harshal Sheth
c7892ada4c Codegen avro + datahub kafka sink (#3) 
* Add codegen

* New architecture + setup file -> console pipeline

* Cleanup source loader

* Basic Kafka metadata source

* Kafka source and extractor

* Add kwargs construct interface

* Fix kafka source unit test

* start working on pipeline test

* kafka datahub sink

* Make myself a profile

* Ingest to datahub from kafka

* Update codegen

* Add restli transport

* Fix bug in restli conversion
 2021-02-15 18:29:27 -08:00
Shirshanka Das
6b5bbbdc5f workaround for docker exec, waiting for 5 more seconds 2021-02-15 18:29:27 -08:00
Harshal Sheth
4fb673925c Start using avro producer 2021-02-15 18:29:27 -08:00
Shirshanka Das
1ddbdee60c Support for SQL databases (MySQL + MS-SQL) (#2) 
* adding sql source + mysql

* adding sql support

* MSSQL support, basic integration test

* file sink and pipeline context
 2021-02-15 18:29:27 -08:00
Shirshanka Das
faf472aa64 adding some TODOs 2021-02-15 18:29:27 -08:00
Shirshanka Das
128781942d Firstdrop of ingest (#1) 2021-02-15 18:29:27 -08:00