12 Commits

Author SHA1 Message Date
Pere Menal-Ferrer
ca812852d6
ci/nox-setup-testing (#21377)
* Make pytest to user code from src rather than from install package

* Fix test_amundsen: missing None

* Update pytest configuration to use importlib mode

* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings

* Refactor referencedByQueries validation to use field_validator as per deprecation warning

* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning

* Move superset test to integration test as they are using testcontainers

* Update coverage source path

* Fix wrong import.

* Add install_dev_env target to Makefile for development dependencies

* Add test-unit as extra in setup.py

* Modify dependencies in dev environment.

* Ignore all airflow tests

* Remove coverage in unit_ingestion_dev_env. Revert coverage source to prevent broken CI.

* Add nox for running unit test

* FIx PowerBI integration test to use pathlib for resource paths and not os.getcwd to prevent failures when not executed from the right path

* Move test_helpers.py to unit test, as it is not an integration test.

* Remove utils empty folder in integration tests

* Refactor testcontainers configuration to avoid pitfalls with max_tries setting

* Add nox unit testing basic setup

* Add format check session

* Refactor nox-unit and add plugins tests

* Add GHA for py-nox-ci

* Add comment to GHA

* Restore conftest.py file

* Clarify comment

* Simplify function

* Fix matrix startegy and nox mismatch

* Improve python version strategy with nox and GHA

---------

Co-authored-by: Pere Menal <pere.menal@getcollate.io>
2025-05-27 10:56:52 +02:00
Pere Menal-Ferrer
6ea630d7ef
DevEx: Ingestion development improvement (focus on unit testing) (#21362)
* Fix test_amundsen: missing None

* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings

* Refactor referencedByQueries validation to use field_validator as per deprecation warning

* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning

* Move superset test to integration test as they are using testcontainers

* Add install_dev_env target to Makefile for development dependencies

* Add test-unit as extra in setup.py

* Skip failing IT test. Requires further investigation.
2025-05-26 10:38:17 +02:00
Pere Miquel Brull
4cddee89ad
FIX - Ingestion Airflow Image constraints (#17296)
* FIX - Ingestion constraints

* FIX - Ingestion constraints

* FIX - Ingestion constraints
2024-08-06 06:44:27 +02:00
Ayush Shah
e9f66e26b4
Bump Setuptools version (#17225) 2024-07-30 10:20:05 +05:30
Ayush Shah
1962ffad52
Fixes #17221: Setuptools Version Fix, Resolve Ingestion Setup Failure (#17216) 2024-07-29 15:29:25 +05:30
Ayush Shah
b676f72b89
chore: Update setuptools version in Makefile and pyproject.toml (#17143) 2024-07-24 10:44:20 +05:30
IceS2
14e475cefe
MINOR: Add PyRight TypeCheck to our Python Project (#17060)
* Add PyRight TypeCheck to our Python Project

* Change pyright for basedpyright

* Fix PyRight
2024-07-18 11:52:56 +02:00
Imri Paran
5b963a5d9e
MINOR: remove metadata/__init__.py (#15750) 2024-03-29 15:15:52 +05:30
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status (#14462)
* Register StackTraceError in spec

* Register StackTraceError in spec

* Register StackTraceError in spec

* Add todos

* Update status

* docs

* format

* Fix tests

* Fix tests

* Fix tests

* Ignore generated

* Fix tests

* Fix tests

* Tests

* Try constants

* Try constants

* Print

* Print

* Print

* order

* Fix service name

* fix ui error

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Pere Miquel Brull
caaf0e7a1d
Fix #12436 - Migrate to pyproject.toml (#14025)
* test

* Use pyproject.toml

* Fix pylint

* makefile

* makefile

* Fix pylint

* isort

* pyproject

* Airflow apis pyproject

* Remove ingestion core

* isort

* Fix makefile help
2023-11-22 07:10:37 +01:00
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework (#13862)
* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Handle app runner

* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Allow deployment

* Fix PII APP

* Fix lint

* Fix PII APP

* Fix PII APP

* Prepare config-based external apps

* Prepare config-based external apps

* Fix lint

* Prepare config-based external apps

* Fix DI errors

* Amend comments
2023-11-13 08:58:38 +01:00
Tom Bushell
33fe3b5ee5
Fixes #7858: Moved ingestion related make commands into Makefile in ingestion directory (#13677)
* Moved more recipes into ingestion/Makefile

* Removed some recipes into ingestion/Makefile and added import statement

* Modified file paths so that 'make generate' works from the ingestion directory

* Modified checks for current directory

* Fixed function names to be in snake case

* Reverted function names back to camel case

* Reverted changes to js_antlr and py_antlr and moved generate command back into root directory Makefile

* Updated run_ometa_integration_testsrecipe in ingestion/Makefile

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-30 08:23:47 +01:00