15 Commits

Author SHA1 Message Date
Teddy
63a55437ae
GEN-1412: Implement load test logic (#19155)
* feat: implemented load test logic

* style: ran python linting

* fix: added locust dependency in test

* fix: skip locust in 3.8 as not supported

* fix: update gcsfs version

* fix: revert gcsfs versionning

* fix: fix gcsf version to 2023.10

* fix: dagster graphql and gx versions

* fix: dagster version to 1.8 for py8 compatibility

* fix: fix clickhouse to 0.2 as 0.3 requires SQA 2+

* fix: revert changes from main

* fix: revert changes compared to main
2025-04-24 16:08:38 +02:00
Imri Paran
b92b950060
Fix 18434: feat(statistics-profiler): use statistics tables to profile trino tables (#18433)
* feat(statistics-profiler): use statistics tables to profile trino tables

- implemented the collaborative root class
- added the "useStatistics" profiler parameter
- added the "supportsStatistics" database connection property
- implemented the ProfilerWithStatistics and StoredStatisticsSource to add this functionality to specific profilers
- implemented TrinoStoredStatisticsSource for specific trino statistics logic

* added ABC to terminal classes in collaborative root

* fixed docstring for TestSuiteInterface

* reverted unintended changes

* typo
2024-11-07 18:37:31 +01:00
Imri Paran
68e71cb3dc
GEN-970: Refactor redshift system metrics to support freshness test (#17981)
* ref(profiler): redshift system metrics

- moved redshift system metrics to the redshift source module
- use Timestamp in data quality
- added plugin feature to test utils

* use timezone.utc

* format

* reverted unintended snowflake changes

* fixed import test_system_metrics.py

* revert

* fixed import in tests
2024-10-10 08:32:07 +02:00
Pere Miquel Brull
d26449576a
GEN-1234 - Clean up suggestions when a user is deleted (#17988)
* GEN-1234 - Clean up suggestions when a user is deleted

* add method

* add method

* fix postgres query
2024-09-26 16:22:36 +02:00
Imri Paran
84be1a3162
Fix 17698: use resolution logic for snowflake system metrics profiler (#17699)
* fix(profiler): snowflake

resolve tables using the snowflake engine instead of OpenMetadata

* added env for cleaning up dbs in E2E

* moved system metric method to profiler. all the rest says in snowflake

* format

* revert unnecessary changes

* removed test for previous resolution method

* use shutdown39
2024-09-06 07:25:10 +00:00
Imri Paran
5133c31d31
MINOR: kafka integration tests (#17457)
* tests: kafka integration

kafka integration tests with schema registry

* added ignore kafka for python 3.8

* fixed tests
2024-08-21 16:05:09 +05:30
Imri Paran
5da7bb049c
MINOR: fix table profiler on empty tables in trino (#17471)
* fix(profiler): trino

coalesce row count to 0 if result is null. this value gets returned for empty tables

* fixed test_metadata.py
2024-08-20 08:42:10 +00:00
Imri Paran
7508848376
fix(dq): data types for unique columns (#17431)
1. remove json and array from supported data types of unique column test.
2. migrations.
3. tests.
2024-08-19 14:28:42 +02:00
Mayur Singal
8b9d8aad91
MINOR: Fix postgres conftest import (#17210) 2024-07-27 18:08:42 +05:30
Imri Paran
cb57941b5f
MINOR: copy data to sql server container instead of using volumes (#17191)
* copy the data files to the containers instead of using volume binds

* format
2024-07-26 10:12:04 +00:00
Imri Paran
0fee79b200
MINOR: fix sample data issue with Pydantic v2 and refactor python integration tests (#16943)
* tests: refactor

refactor tests and consolidate common functionality in integrations.conftest

this enables writing tests more concisely.
demonstrated with postgres and mssql.
will migrate more

* format

* removed helpers

* changed scope of fictures

* changed scope of fixtures

* added profiler test for mssql

* fixed import in data_quality test

* json safe serialization

* format

* set MARS_Connection

* use SerializableTableData instead of TableData

* deleted file test_postgres.py

* fixed tests

* added more test cases

* format

* changed name test_models.py

* removed the logic for serializing table data

* wip

* changed mapping in common type map

* changed mapping in common type map

* reverted TableData imports

* reverted TableData imports

* reverted TableData imports
2024-07-17 08:11:34 +02:00
Imri Paran
43f46196a4
tests: postgres container (#16924)
added track_commit_timestamp=on so we can track table updates in tests

https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-TRACK-COMMIT-TIMESTAMP
2024-07-05 11:34:56 +02:00
Imri Paran
9b5ce3560c
MINOR: Pydantic equal assert util (#16918)
* tests: pydantic object assertion

added util for comparing pydantic objects

* fixed test_data_diff.py
2024-07-04 09:59:46 +05:30
Imri Paran
5e5c811ef2
moved int_admin_ometa to a dedicated module (#16768) 2024-06-25 11:21:22 +05:30
Teddy
141ceb4c8d
MNINOR add common test elements to _openmetadata_testutils module (#16758)
* fix: add common test to _testutils module

* fix: renamed _testutils to _openmetadata_testutils
2024-06-21 15:11:34 +02:00