24 Commits

Author SHA1 Message Date
Harshal Sheth
a1e62c723e
docs(ingest): add airflow docs that use the PythonVirtualenvOperator (#6604) 2022-12-02 19:56:17 +01:00
Harshal Sheth
3e907ab0d1
feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ (#6204)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-11 15:04:36 -05:00
Harshal Sheth
b4687ffceb
feat(ingest): drop plugin support for airflow 1.x (#6331) 2022-11-01 21:12:34 -07:00
Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support (#5521) 2022-08-10 15:00:31 -07:00
Tamas Nemeth
f99d27fd8c
feat(ingest): airflow - add support to capture airflow executions, add high level dataflow jobs api to python sdk (#4615)
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-04-12 23:19:39 -07:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing (#4142)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
Harshal Sheth
b43945ce55
test(airflow): remove unneeded execution_date parameter from test (#3368) 2021-10-12 23:25:27 -07:00
Swaroop Jagadish
7dab06519d
fix(ci): fix task instance constructor in airflow tests (#3366) 2021-10-11 20:58:37 -07:00
Dexter Lee
8316316daa
fix(ingest): add default configurable timeout for rest emitter (#3055) 2021-08-08 22:30:55 -07:00
Harshal Sheth
14bca4c2f6
fix(ingestion): isolate dependency requirements of airflow hooks (#2977) 2021-07-28 14:23:06 -07:00
Harshal Sheth
fa22b8e17b
feat(ingest): add Airflow TaskFlow example (#2958) 2021-07-26 13:09:25 -07:00
Harshal Sheth
7a1ce19d2b
fix(ingestion): resolve test bugs for 3.6 (#2953) 2021-07-23 17:07:13 -07:00
Harshal Sheth
958fe8ea83
feat(ingest): populate inputDatajobs field in airflow integration (#2606) 2021-05-25 22:47:00 -07:00
Harshal Sheth
3dfe3d375b
feat(ingest): add options for Airflow lineage backend (#2557) 2021-05-13 20:02:47 -07:00
Harshal Sheth
a671001824
refactor(ingest): move Airflow into datahub_provider module (#2521) 2021-05-12 15:01:11 -07:00
Harshal Sheth
cd588baccb
build(ingest): include package data in sdist (#2513) 2021-05-07 15:21:43 -07:00
Harshal Sheth
d0ca3191c9
build(ingest): add metadata-ingestion to gradle build (#2510) 2021-05-06 22:10:49 -07:00
Harshal Sheth
7f0656fd5e
fix(ingest): replace ImportError with ModuleNotFoundError (#2498)
Using the more specific exception will prevent us from accidentally
ignoring errors that should be handled.
2021-05-05 14:05:16 -07:00
Harshal Sheth
e48a74b80a
test(ingest): add test names and IDs using pytest (#2476) 2021-04-29 23:18:55 -07:00
Harshal Sheth
50aee5c05a
fix(ingest): support Airflow 1.10.x style lineage in Airflow 2 (#2455) 2021-04-26 23:08:43 -07:00
Harshal Sheth
a857d3b9d8
fix(ingest): various updates to datahub rest sink (#2445) 2021-04-23 23:48:44 -07:00
Harshal Sheth
79daec29b7
fix(ingest): ensure upstreams in airflow lineage emission are entities (#2427) 2021-04-20 20:44:38 -07:00
Harshal Sheth
41cd52f9e2
feat(ingest): add Airflow lineage backend (#2368) 2021-04-12 17:40:15 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders (#2331) 2021-04-05 19:11:28 -07:00