128 Commits

Author SHA1 Message Date
Aseem Bansal
2a75a981ca
chore(ci): upgrade ruff version (#13125) 2025-04-09 11:24:07 +05:30
Sergio Gómez Villamor
4e48e098dc
fix(ingestion): fixes missing platform instance aspect for DataFlow entitiy (#13080) 2025-04-06 08:19:47 +02:00
Rasnar
38e240e916
feat(ingest/airflow): platform_instance support in Airflow plugin (#12751)
Co-authored-by: rasnar <11248833+Rasnar@users.noreply.github.com>
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
2025-04-04 09:26:58 +02:00
Harshal Sheth
d3ef859802
fix(airflow): drop airflow 2.3 and 2.4 support (#13004) 2025-03-28 12:58:50 -07:00
kyungryun choi
26d9962a98
fix(airflow): fix logging typo (#13000) 2025-03-27 13:41:37 -07:00
Tamas Nemeth
0093aa8185
chore(ingestion/airflow): Example custom operator with sql parsing (#12959) 2025-03-27 09:18:29 +01:00
Aseem Bansal
d1494c2252
dev: upgrade ruff, remove 2 ruff ignore rules (#12702) 2025-02-24 14:07:57 +05:30
Aseem Bansal
effd4825f3
dev: reduce log in gradle commands (#12677) 2025-02-20 13:47:48 +05:30
Harshal Sheth
26e7743cfe
fix(dagster): support dagster 1.10.0 (#12648) 2025-02-16 22:23:57 -08:00
Harshal Sheth
79f844b208
chore(py): add unused-noqa lint rule (#12615) 2025-02-13 08:28:48 -08:00
Harshal Sheth
7472c535d3
feat(ci): add pytest hooks for updating golden files (#12581) 2025-02-12 15:32:31 -08:00
Andrew Sikowitz
b51327e0be
feat(versioning): Add V2 UI; make backend more synchronous; add to component library (#12542) 2025-02-04 09:57:41 -08:00
Tamas Nemeth
22e012a209
feat(ingestion/airflow): Add support for mutiple datahub emitter (#12398) 2025-01-29 16:44:31 +01:00
Harshal Sheth
a8d6c54965
feat(sdk): move version info to dedicated file (#12456) 2025-01-24 11:35:39 -08:00
Aseem Bansal
262dd76518
dev: remove black in favor of ruff for formatting (#12378) 2025-01-18 15:06:20 +05:30
Aseem Bansal
436b74cd3a
dev: move from flake8,isort to ruff (#12375) 2025-01-17 23:50:13 +05:30
Aseem Bansal
76e46b89db
dev(ingest): move modules from isort,flake8 to ruff (#12373) 2025-01-17 21:38:29 +05:30
Harshal Sheth
03e3f46175
fix(airflow): fix tests with Airflow 2.4 (#12279) 2025-01-07 07:46:35 -08:00
Harshal Sheth
ed8639e401
chore(python): test with python 3.11 (#11280)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2024-12-27 10:46:49 -08:00
Sergio Gómez Villamor
66df362c0f
fix(dagster): support dagster v1.9.6 (#12189) 2024-12-20 09:57:53 +01:00
Tamas Nemeth
b3a76cb308
feat(ingest/airflow): Add way to disable Airflow plugin without a restart (#12098) 2024-12-16 11:29:13 -08:00
Harshal Sheth
ecf6c8c588
feat(airflow): add DatahubRestHook.make_graph method (#12116) 2024-12-15 10:27:50 -05:00
Tamas Nemeth
ee82a88a75
fix(ingest/dagster): Fix Dagster build (#12121) 2024-12-13 11:51:42 -05:00
sagar-salvi-apptware
57b12bd9cb
fix(ingest): replace sqllineage/sqlparse with our SQL parser (#12020) 2024-12-10 08:36:01 -08:00
Harshal Sheth
2babbe6b0f
feat(airflow): drop Airflow < 2.3 support + make plugin v2 the default (#12056) 2024-12-09 11:08:49 -08:00
Steffen Grohsschmiedt
49b6284ebf
fix(airflow): fix AthenaOperator extraction (#11857)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-12-03 16:16:44 -08:00
Chakru
85b42e3ea5
build(coverage): enable code coverage for java and python (#11992)
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
2024-12-02 19:27:43 -06:00
Harshal Sheth
411e1a30ff
feat(airflow): show dag/task logs in CI (#11981) 2024-12-02 09:53:02 -08:00
JaeMoo Han
418259b8c4
fix(airflow): remove trino to presto mapping (#11925)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-11-25 12:13:26 -08:00
Tamas Nemeth
435792ca2e
fix(ingest/airflow): Add log to dag emit (#11880) 2024-11-18 17:00:22 +01:00
Mayuri Nehate
64355b490a
fix(ingest): update GX version having name arg (#11849) 2024-11-13 15:27:08 +01:00
Mayuri Nehate
5094dabdf4
feat(ingest/gx): support gx version 0.18.0 (#11823) 2024-11-11 14:38:15 +01:00
Tamas Nemeth
f6760436b7
feat(ingestion/dagster): Dagster assetless ingestion (#11262)
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
2024-10-24 16:06:00 +02:00
Harshal Sheth
35f30b7d3c
feat(ingest): use mainline sqlglot (#11693) 2024-10-22 19:57:46 -07:00
Tamas Nemeth
16f9075869
fix(ingest/prefect): Fix prefect mypy errors (#11680) 2024-10-21 12:37:22 -05:00
Tamas Nemeth
ba7a43f530
fix(ingest/dagster): Fix JobSnapshot import is broken (#11672) 2024-10-18 20:16:08 +02:00
Harshal Sheth
d34717fd82
fix(ingest): remove default value from DatahubClientConfig.server (#11570) 2024-10-16 13:50:33 -07:00
Harshal Sheth
be1b8806a7
fix(airflow): fix lint related to dag_run field (#11616) 2024-10-15 14:09:41 +05:30
dushayntAW
f13dae1cd2
fix(airflow): add dag AllowDenyPattern config (#11472)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-10-11 10:46:01 -07:00
Harshal Sheth
771cc9ffe4
feat(airflow): add airflow 2.10 to test matrix (#11579) 2024-10-10 09:30:43 -07:00
Harshal Sheth
0414443b77
feat(airflow): add a render_templates config parameter (#11537) 2024-10-09 18:24:38 -07:00
David Schmidt
a87c123611
fix(dagster-plugin): Fix in/outs format and source config (#11481) 2024-10-01 20:59:23 -07:00
Andrew Sikowitz
3e3a05f09c
test(ingest/mcp_diff): Fallback to overwriting file on more complicated diffs (#11407)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-18 16:40:08 -07:00
dushayntAW
fca86bc5a9
fix(airflow): added support for jinja template for datahub emitter operator (#11300) 2024-09-17 12:57:45 -07:00
Harshal Sheth
1d83131601
fix(ingest/airflow): simplify env configuration (#11371) 2024-09-17 12:54:03 -07:00
Harshal Sheth
dfa97758a1
chore(py): cleanup python CI (#11324) 2024-09-11 10:49:16 -07:00
Tamas Nemeth
c05b89541d
fix(ingest/prefect): Temporary pinning Prefect 2.x until we can upgrade to 3.x (#11302) 2024-09-04 15:00:24 -07:00
dushayntAW
6204cba2bb
fix(ingestion/prefect-plugin): Prefect plugin (#10643)
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-08-29 15:40:10 +02:00
Aseem Bansal
15c1cfc386
ci(build): update outdated action & pin deepdiff lib (#11260) 2024-08-28 19:53:42 +05:30
Tamas Nemeth
1893b7b2dc
feat(ingest/dagster): Add automatic snowflake_pandas_io_manager asset capture (#11189)
Co-authored-by: shubhamjagtap639 <shubham.jagtap@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
2024-08-28 09:22:11 +02:00