* ref(profiler): redshift system metrics
- moved redshift system metrics to the redshift source module
- use Timestamp in data quality
- added plugin feature to test utils
* use timezone.utc
* format
* reverted unintended snowflake changes
* fixed import test_system_metrics.py
* revert
* fixed import in tests
* fix(profiler): snowflake
resolve tables using the snowflake engine instead of OpenMetadata
* added env for cleaning up dbs in E2E
* moved system metric method to profiler. all the rest says in snowflake
* format
* revert unnecessary changes
* removed test for previous resolution method
* use shutdown39
* tests: refactor
refactor tests and consolidate common functionality in integrations.conftest
this enables writing tests more concisely.
demonstrated with postgres and mssql.
will migrate more
* format
* removed helpers
* changed scope of fictures
* changed scope of fixtures
* added profiler test for mssql
* fixed import in data_quality test
* json safe serialization
* format
* set MARS_Connection
* use SerializableTableData instead of TableData
* deleted file test_postgres.py
* fixed tests
* added more test cases
* format
* changed name test_models.py
* removed the logic for serializing table data
* wip
* changed mapping in common type map
* changed mapping in common type map
* reverted TableData imports
* reverted TableData imports
* reverted TableData imports
* feat(table-diff): added column validation
added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report.
* fixed unit tests
* format
* - resolve column types more robustly
- changed test result metric to include "rows" or "columns"
* feat: add tableDiff test case
This changed introduces a "table diff" test case which
compares two tables and fails if they are not identical.
The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections).
1. Added the `tableDiff` test definition.
2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime.
3. Integration tests (because of course).
This feature was not tested end-to-end yet because "array" data
* pydantic v2
* format
* format
* format and added data diff to setup.py
* format
* fixed param issue which has type ARRAY
* fixed runtime_parameter_setter
* moved models to parent directory
* handle errors in table diff
* fixed issue with edit test case
* format
* added more details to pytest skip
* format
* refactor: Improve createTestCaseParameters function in DataQualityUtils
* fixed unit test
* removed unused fixture
* removed validator.py
* fixed tests
* added validate kwarg to tests_mixin
* removed "postgres" data diff extra as they interfere with psycopg2-binary
* fixed tests
* pinned tenacity for tests
* reverted tenacity pinning
* added ui support for test diff
* fixed dq cypress and added edit flow
* organized the test case
* added dialect support
* fixed tests
* option style fix
* fixed calculation for passing/failing rows
* restrict the tableDiff test to limited services
* set where to None if blank string
* fixed where clause
* fixed tests for where clause
* use displayName in place of name in edit form
* added docs for RuntimeParameterSetter
* fixed cypress
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* Added Test Case for Matching Enum
1. Implemented the test case using the `matchEnum` parameter.
2. Added integration tests.
3. Added migrations.
* fix tests
* fixed tests
* format
* fixed tests
* clear search cache before running ingestion
* format
* changed scopt of aws fixture
* moved migrations to 1.5.0
* Mantain the OperationType Order when considering the dividing groups
* Remove reordering the jsonpatch operations from the backend
* Fix checkstyle
* Fix UnitTests to comply with no reordering
* Initial idea on how to fix our current jsonpatch builder from python
* fix(JsonUtils): Change JSONPatch library used
When creating a JSONPatch by using the 'createDiff' method, the library
we are using is not returning a correct JSONPatch when removing multiple
items from an array.
Since the library doesn't provide good ways to override this behavior
and fix it, we decided to move away from it and use the json-patch
library only for this specific operation.
* Fix linters
* Add docstrings
* Refactor patch updated on ingestion framework
* Add UnitTests
* Fix linters
* fix: update antlr regex for entity fqn
* fix: update antlr rule to allow single character
* style: ran python linting
* fix: updated antlr token for NAME_OR_FQN
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
* Fix supported characters in SM
* Update SM
* Fixes
* Fixes
* Improve class conversion exceptions
* Comments
* Rename noop to db secrets manager providee
* Update sm
* Fix
* db SM
* db SM
* Fix test
* UI
* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts
* update default
* fix: logic for test suite config workflow
* fix: added caching for system metrics (snflk and bq)
* fix: linting
* fix: added tearDown logic for tests suite/case
* Change how status is handled after running workflow
* Reset changes in config files
* Add auxiliary Summary class
* Improve failures handling
* Pylint error
* Pylint error
* Show result in table
* Add test
* Fix setup.py
* Add comments
* Removed part of the secrets manager implementation in the server side
* Removed part of the secrets manager implementation in the openmetadata python library side
* Remove deprecated test
* Address pylint checks
* Address new pylint checks
* Address PR comments
* Fix import on airflows apis
* Clear singleton instances for running Ometa secrets manager test
* Fix bot creation and edition logic
* Fix minor error creating user
* Fix failing test
* Minor fixes
* Add missing tests for new flows
* Fix put_failIfBotHasARelationshipToAnotherUser test
* Changes after manual testing
* Move where auth_provider is retrieved in the secret manager
* Encode entityLink string when processing request
* Added logic to decode column type from entityLink
* mvn code formating
* Extracted unquote step into its own function
* Integrated DBT tests
* Fixed py tests
* Review comments addressed
* Changed test suite name default value
* Added fixes
* update ui changes for new field
* send data along with api
* update name
* fixes for UI
* fixes for UI
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* Implementation of AWS SSM as secrets manager
* Remove dead code
* Minor fixes
* Allow using default credentials in AWS client
* Fixed py style
* Fixed tests imports
* Minor changes
* Add patch to failing test
* Fix how we were storing dbtConfigSource
* Address PR comments
* Add cluster name in the app configuration and start using it to create secrets id
* Update secret manager client in openmetadata for using default auth provider
* Add missing property in test config file
* Implementation for stop sending connection credentials when user is BOT
* Change way we add the connection to the service in the Secret Manager
* Services connection is not required as we want to stop returning it when SM is configured
