yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-04 15:50:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,753 Commits 189 Branches 118 Tags
Commit Graph

578 Commits

Author SHA1 Message Date
Joe Mirizio
f3304bec7c
feat(ingest): dynamically register plugins (#2316) 
Co-authored-by: Joe Mirizio <mirizioj@email.chop.edu>
 2021-03-31 20:59:45 -07:00
Harshal Sheth
f57c954fc6
feat(ingest): support environment variables in recipes (#2306) 2021-03-26 21:57:05 -07:00
Harshal Sheth
07f4cb1199
feat: datahub check local-docker (#2295) 2021-03-26 10:03:51 -07:00
Harshal Sheth
cc19465f55
fix(ingest): resolve array serialization bug (#2290) 2021-03-24 10:02:46 -07:00
Harshal Sheth
a921d0deae
feat(ingest): MongoDB ingestion source (#2289) 2021-03-23 20:15:44 -07:00
Harshal Sheth
1ea450e0e4
fix(ingest): use custom pybigquery ref to get descriptions (#2279) 2021-03-22 23:17:54 -07:00
Harshal Sheth
b8462028c3
feat(ingest): various minor fixes (#2246) 2021-03-17 23:05:05 -07:00
Pedro Silva
6a0c402a58
feat(ingest): Add support for druid (#2235) 2021-03-17 20:06:48 -07:00
Harshal Sheth
aa6bc15cd7
fix(ingest): various avro codegen fixes (#2232) 2021-03-15 15:27:30 -07:00
Harshal Sheth
95c124ffc4
fix(ingest): pin version of avro-gen3 (#2230) 2021-03-12 09:39:38 -08:00
Harshal Sheth
6a8fca59f1
feat(ingest): use plugin system based on Python extras (#2224) 2021-03-11 13:41:05 -08:00
Harshal Sheth
dced25fef7
feat(ingest): switch quickstart to Python ingestion (#2158) 2021-03-02 11:48:26 -08:00
Harshal Sheth
347148b79b Update python workflow 2021-02-15 18:29:27 -08:00
Harshal Sheth
38f75be8ad gometa -> datahub 2021-02-15 18:29:27 -08:00
Harshal Sheth
0063c04460 gometa-ingest -> datahub ingest 2021-02-15 18:29:27 -08:00
Harshal Sheth
b91d0cf63b Add bigquery and refactor others 2021-02-15 18:29:27 -08:00
Harshal Sheth
d12497a3ff Add postgres source 2021-02-15 18:29:27 -08:00
Harshal Sheth
3678368ed3 Prune deps 2021-02-15 18:29:27 -08:00
Harshal Sheth
9332e6b878 Add isort to CI 2021-02-15 18:29:27 -08:00
Harshal Sheth
0660991fb8 More python 3.6 compat 2021-02-15 18:29:27 -08:00
Harshal Sheth
75b01806ef Use find_packages() in setup 2021-02-15 18:29:27 -08:00
Shirshanka Das
9c6ea3b9e9 Adding 3.9 to setup.py classifiers 2021-02-15 18:29:27 -08:00
Shirshanka Das
024745bcc4 removing py 3.6, adding 3.9 for ci 2021-02-15 18:29:27 -08:00
Harshal Sheth
c7892ada4c Codegen avro + datahub kafka sink (#3) 
* Add codegen

* New architecture + setup file -> console pipeline

* Cleanup source loader

* Basic Kafka metadata source

* Kafka source and extractor

* Add kwargs construct interface

* Fix kafka source unit test

* start working on pipeline test

* kafka datahub sink

* Make myself a profile

* Ingest to datahub from kafka

* Update codegen

* Add restli transport

* Fix bug in restli conversion
 2021-02-15 18:29:27 -08:00
Harshal Sheth
4fb673925c Start using avro producer 2021-02-15 18:29:27 -08:00
Shirshanka Das
1ddbdee60c Support for SQL databases (MySQL + MS-SQL) (#2) 
* adding sql source + mysql

* adding sql support

* MSSQL support, basic integration test

* file sink and pipeline context
 2021-02-15 18:29:27 -08:00
Shirshanka Das
faf472aa64 adding some TODOs 2021-02-15 18:29:27 -08:00
Shirshanka Das
128781942d Firstdrop of ingest (#1) 2021-02-15 18:29:27 -08:00