* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Fix Test
* Fix Profile Test Connection
* Remove unit test
* Remove comment
* Fix tests and missing changes
* refactor: removed testSuite field from CreateTestCase
BREAKING CHANGE: when creating a test case, testsuite is now derived from entityLink (fetch or created)
* feat: allow setting tags when creating a test case
* style: ran linters
* fix: compiling error
* fix: failing test case
* fix: failing tests
* removed testSuite from required filed
* fixed ui side
* style: ran java linting
* deprecation: remove testSuite param from ingestion
* fix: remove test suite filed
* fix: remove test_suite field
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* Add PIICategoryTags and some utilities on top of them.
* Fix static-check
* Add test for fqn representation
* Add NEREntityGeneralTags.json from Collate
* Add test to check PIICategoryTags agree with the ones used by OM server
* Add LabelExtractor
* Fix style
* Add ignore superflous-parens for pylint
* Ass comment as per PR review
* Fix not-updated PII-IT
* Remove duplicated IT test for PII
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* deprecation: remove testCaseResults endpoint from testCaseResource
* fix: path in test e2e test
* fix: endpoint name to testCaseResults
* style: fix java linting
* Make pytest to user code from src rather than from install package
* Fix test_amundsen: missing None
* Update pytest configuration to use importlib mode
* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings
* Refactor referencedByQueries validation to use field_validator as per deprecation warning
* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning
* Move superset test to integration test as they are using testcontainers
* Update coverage source path
* Fix wrong import.
* Add install_dev_env target to Makefile for development dependencies
* Add test-unit as extra in setup.py
* Modify dependencies in dev environment.
* Ignore all airflow tests
* Remove coverage in unit_ingestion_dev_env. Revert coverage source to prevent broken CI.
* Add nox for running unit test
* FIx PowerBI integration test to use pathlib for resource paths and not os.getcwd to prevent failures when not executed from the right path
* Move test_helpers.py to unit test, as it is not an integration test.
* Remove utils empty folder in integration tests
* Refactor testcontainers configuration to avoid pitfalls with max_tries setting
* Add nox unit testing basic setup
* Add format check session
* Refactor nox-unit and add plugins tests
* Add GHA for py-nox-ci
* Add comment to GHA
* Restore conftest.py file
* Clarify comment
* Simplify function
* Fix matrix startegy and nox mismatch
* Improve python version strategy with nox and GHA
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
* Fix test_amundsen: missing None
* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings
* Refactor referencedByQueries validation to use field_validator as per deprecation warning
* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning
* Move superset test to integration test as they are using testcontainers
* Add install_dev_env target to Makefile for development dependencies
* Add test-unit as extra in setup.py
* Skip failing IT test. Requires further investigation.
* Add PII Tag and Sensitivity Level enums.
* Add feature-extraction for PII classification tasks
* Add faker as test dependency
* Add unit tests for presidio tag extractor
* Add PIISensitivityTags enum and update sensitivity mapping logic
* Add Presidio utility functions for PII analysis
* Extend column name regexs for PII
* Add tests for PAN, NIF, SSN entities
* Fix version of faker to prevent flaky tests. Fix failing tests.
* Add Generated to State enum
* Integrate PIISensitive classifier to PIIProcessor
* fix: add support for GX 0.18.22 and GX 1.4.x
* fix: add support for GX 0.18.22 and GX 1.4.x
* style: ran python linting
* fix: skip test if GX version is not installed
* Search: improve relevancy for plural/singular words, partial matches, exact matches
* apply to all indexes
* Fix other query patterns
* Revert changes of database and databaseSchema fields in TableIndex.getFields() and table index mapping
* add missing boost query builder in es
* fix ci
* add max_ngram_diff setting in di-assets index
* fix TestCaseResourceTest mvn test failure
---------
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* feat(app): add config to status
add config to the reported status of the ingestion pipeline
* added separate pipeline service client call for external apps
* fix masking of pydantic model
* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py
* fix: execution time
* fix: mask secrets in dump json
* fix: for python3.8
* fix: for python3.8
* fix: use mask_secrets=False when dumping a model for create
* format
* fix: update mask_secrets=False for workflow configurations
* fix: use context directly when using model_dump_json
* fix: default behavior when dumping json
* format
* fixed tests
* Fix#17723: Generate Incremental Change Events even when consolidation of events applied
* Fix#17723: Generate Incremental Change Events even when consolidation of events applied
* fix tests
* Fix tests
* clean policy tests
* update search methods to use incrementalChangeDescription part-1
* Fix the version page playwrights
* update search methods to use incrementalChangeDescription part-2
* introduce new field incrementalChangeDescription for search part-3
* fix mvn endpoint test
* fix followers and page search test
* fix following of assets
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
* fix: wrong attribute name in SampleConfig model
* fix: test attribute
* fix: failing tests
* fix: trino filter error + adjust test to take into account null value
* fix: mssql and azuresql tablesample on views
* backend
* format & tests
* rename backend
* migrations and ingestion
* format & tests
* format & tests
* tests
* format & tests
* tests
* updated ui side of changes
* addressing comment
* fixed failing unit test
* fix test list
* added e2e test, and fixed existing test
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* MINOR: User search should only look in name & displayname
* py_format
* pyformat
---------
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
* ref(data-quality): modularized test case validator import
- removed test_suite_factory
- implemented TestCaseImporter
- removed SQAValidatorBuilder and PandasValidatorBuilder in favor of a SourceType enum
- removed the orm table creation from test suite source
* format
* IValidatorBuilder -> ValidatorBuilder
* use the table from the sampler in the test suite interface
* linting
* fixed the profiler with similar solution
* removed unused inheritance
* removed unneeded super().__init__()
* removed all instances of orm_table
* fixed tests
* add reportExplicitAny=false
* fixed tests
