12 Commits

Author SHA1 Message Date
Pere Menal-Ferrer
ca812852d6
ci/nox-setup-testing (#21377)
* Make pytest to user code from src rather than from install package

* Fix test_amundsen: missing None

* Update pytest configuration to use importlib mode

* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings

* Refactor referencedByQueries validation to use field_validator as per deprecation warning

* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning

* Move superset test to integration test as they are using testcontainers

* Update coverage source path

* Fix wrong import.

* Add install_dev_env target to Makefile for development dependencies

* Add test-unit as extra in setup.py

* Modify dependencies in dev environment.

* Ignore all airflow tests

* Remove coverage in unit_ingestion_dev_env. Revert coverage source to prevent broken CI.

* Add nox for running unit test

* FIx PowerBI integration test to use pathlib for resource paths and not os.getcwd to prevent failures when not executed from the right path

* Move test_helpers.py to unit test, as it is not an integration test.

* Remove utils empty folder in integration tests

* Refactor testcontainers configuration to avoid pitfalls with max_tries setting

* Add nox unit testing basic setup

* Add format check session

* Refactor nox-unit and add plugins tests

* Add GHA for py-nox-ci

* Add comment to GHA

* Restore conftest.py file

* Clarify comment

* Simplify function

* Fix matrix startegy and nox mismatch

* Improve python version strategy with nox and GHA

---------

Co-authored-by: Pere Menal <pere.menal@getcollate.io>
2025-05-27 10:56:52 +02:00
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
Pere Miquel Brull
f4369ded8d
MINOR - Allow dict-based xlets in Airflow (#16262) 2024-05-15 15:08:09 +02:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets (#14191)
* Add OMEntity model

* Test OMEntity

* Update repr

* Fix __str__

* Add entity ref map

* Test serializer for backend

* Fix tests

* Fix serializer

* Test runner

* Add runner tests

* Update docs

* Format
2023-12-01 06:29:44 +01:00
Pere Miquel Brull
8cf8720a9d
Clean Airflow Lineage Backend and migrate status to millis (#13666)
* Clean Airflow Lineage Backend and migrate status to millis

* Format

* chore(ui): update executions startTs and endTs to millis

* Remove lineage providers

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-10-20 15:42:38 +02:00
Pere Miquel Brull
a3bff293a4
Fix #12190 - Bring support for Airflow 2.6 & Use Airflow 2.6.3 as the new ingestion base (#12398)
* Bump Airflow version

* Support Airflow 2.6

* Lint

* Bump airflow version

* Fix airflow 2.6 lineage

* Fix airflow 2.6 lineage
2023-07-13 06:41:59 +02:00
Pere Miquel Brull
4626363fd8
Fix parsing for Storage (#11663) 2023-05-19 09:36:44 +02:00
Pere Miquel Brull
cc3d2b448b
Clean Airflow lineage test (#10745) 2023-03-27 11:46:34 +02:00
NiharDoshi99
205e5c8c4f
Aligned airflow source with lineage (#10596)
* aligned airflow source with lineage

* fix python test

* changes as per comment
2023-03-15 19:29:24 +05:30
Pere Miquel Brull
fb15c896b3
Handle XLets in groups for AirflowLineageRunner (#10114)
* Handle XLets in groups

* Linting

* Linting
2023-02-07 06:49:46 +01:00
Pere Miquel Brull
479a8de486
Fix #5885 - Provider loading fails for airflow <2.3 (#5927)
Fix #5885 - Provider loading fails for airflow <2.3 (#5927)
2022-07-14 15:07:39 +02:00