Harshal Sheth
432feaa16d
feat(ingest): mark database_alias and env as deprecated ( #6901 )
2023-01-09 19:58:19 +05:30
Harshal Sheth
31260888fc
feat(ingest/airflow): support raw dataset urns in airflow lineage ( #6854 )
...
* feat(ingest/airflow): support dataset Urns in airflow lineage
This PR also
- resolves a reported circular import issue
- refactors the Airflow tests to reduce duplication
* fix test
2022-12-27 08:59:26 +01:00
Harshal Sheth
a1e62c723e
docs(ingest): add airflow docs that use the PythonVirtualenvOperator ( #6604 )
2022-12-02 19:56:17 +01:00
Harshal Sheth
3e907ab0d1
feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ ( #6204 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-11-11 15:04:36 -05:00
Harshal Sheth
b4687ffceb
feat(ingest): drop plugin support for airflow 1.x ( #6331 )
2022-11-01 21:12:34 -07:00
Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support ( #5521 )
2022-08-10 15:00:31 -07:00
Tamas Nemeth
f99d27fd8c
feat(ingest): airflow - add support to capture airflow executions, add high level dataflow jobs api to python sdk ( #4615 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-04-12 23:19:39 -07:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing ( #4142 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
Harshal Sheth
b43945ce55
test(airflow): remove unneeded execution_date parameter from test ( #3368 )
2021-10-12 23:25:27 -07:00
Swaroop Jagadish
7dab06519d
fix(ci): fix task instance constructor in airflow tests ( #3366 )
2021-10-11 20:58:37 -07:00
Dexter Lee
8316316daa
fix(ingest): add default configurable timeout for rest emitter ( #3055 )
2021-08-08 22:30:55 -07:00
Harshal Sheth
14bca4c2f6
fix(ingestion): isolate dependency requirements of airflow hooks ( #2977 )
2021-07-28 14:23:06 -07:00
Harshal Sheth
fa22b8e17b
feat(ingest): add Airflow TaskFlow example ( #2958 )
2021-07-26 13:09:25 -07:00
Harshal Sheth
7a1ce19d2b
fix(ingestion): resolve test bugs for 3.6 ( #2953 )
2021-07-23 17:07:13 -07:00
Harshal Sheth
958fe8ea83
feat(ingest): populate inputDatajobs field in airflow integration ( #2606 )
2021-05-25 22:47:00 -07:00
Harshal Sheth
3dfe3d375b
feat(ingest): add options for Airflow lineage backend ( #2557 )
2021-05-13 20:02:47 -07:00
Harshal Sheth
a671001824
refactor(ingest): move Airflow into datahub_provider module ( #2521 )
2021-05-12 15:01:11 -07:00
Harshal Sheth
cd588baccb
build(ingest): include package data in sdist ( #2513 )
2021-05-07 15:21:43 -07:00
Harshal Sheth
d0ca3191c9
build(ingest): add metadata-ingestion to gradle build ( #2510 )
2021-05-06 22:10:49 -07:00
Harshal Sheth
7f0656fd5e
fix(ingest): replace ImportError with ModuleNotFoundError ( #2498 )
...
Using the more specific exception will prevent us from accidentally
ignoring errors that should be handled.
2021-05-05 14:05:16 -07:00
Harshal Sheth
e48a74b80a
test(ingest): add test names and IDs using pytest ( #2476 )
2021-04-29 23:18:55 -07:00
Harshal Sheth
50aee5c05a
fix(ingest): support Airflow 1.10.x style lineage in Airflow 2 ( #2455 )
2021-04-26 23:08:43 -07:00
Harshal Sheth
a857d3b9d8
fix(ingest): various updates to datahub rest sink ( #2445 )
2021-04-23 23:48:44 -07:00
Harshal Sheth
79daec29b7
fix(ingest): ensure upstreams in airflow lineage emission are entities ( #2427 )
2021-04-20 20:44:38 -07:00
Harshal Sheth
41cd52f9e2
feat(ingest): add Airflow lineage backend ( #2368 )
2021-04-12 17:40:15 -07:00
Harshal Sheth
bd78b84bd3
feat(ingest): start airflow integration + metadata builders ( #2331 )
2021-04-05 19:11:28 -07:00
