165 Commits

Author SHA1 Message Date
harshsoni2024
69d0e3ce88
powerbi e2e count fix (#19626) 2025-02-03 10:24:59 +05:30
Imri Paran
e003a5f9ed
tests(e2e): data quality (#19513)
* tests(e2e): data quality

- use the data quality pipeline new schema with a list of service configurations

* format
2025-01-24 14:01:54 +00:00
Imri Paran
5fc20c1ab2
tests: fix sample row count for bigquery (#19511)
- changed method name to expected_sample_count
2025-01-24 10:44:21 +01:00
Suman Maharana
89dae42407
Fix mysql e2e count (#19315) 2025-01-10 21:41:38 +05:30
Akash Verma
4cad5762ad
Added lineage in cli e2e (#19216) 2025-01-09 14:59:49 +05:30
harshsoni2024
63aa484fec
MINOR: powerbi e2e fix (#19240) 2025-01-09 13:49:59 +05:30
IceS2
3d83f41b82
Remove the sample_data from assert in profiler since it was moved to the classification workflow (#19153) 2024-12-20 10:48:56 +01:00
IceS2
efbdf21098
Fix e2e tests (#19130)
* Fix E2E

* Fix E2E

* Fix MySQL count

* Tentative Fix

* Tentative Fix

* Revert PowerBI changes

* Revert PowerBI changes
2024-12-19 10:39:30 +01:00
Imri Paran
e30571cf4e
[GEN-2187] fix(data-diff): added MD5 handling for bigquery (#18904)
* fix(data-diff): added nd5 handling for bigquery

- added MD5 handling for bigquery
- use URL instead of Engine because it requires less steps and less prone to failure

* added e2e test for data diff with sampling in bigquery
2024-12-06 14:21:33 +01:00
Pere Miquel Brull
61021be98a
TEST - Add autoClassification for e2e (#18722) 2024-11-21 15:07:04 +01:00
Suman Maharana
a218bbf5cb
Minor: Fix Mysql cli Update table count (#18582) 2024-11-15 14:27:02 +05:30
harshsoni2024
cd3fcb5d22
MINOR: quicksight e2e fix (#18629) 2024-11-14 16:31:11 +05:30
Ayush Shah
6fa03ee66a
Fixes GEN-1994: Remove View Lineage from Metadata Ingestion flow (#18558) 2024-11-13 00:08:55 +05:30
Mayur Singal
f4fdafeb8a
MINOR: Athena & Tableau E2E fix (#18596) 2024-11-12 19:14:45 +05:30
Imri Paran
70c7880dfa
fixed bigquery system metrics e2e test (#18601) 2024-11-12 14:06:54 +01:00
Imri Paran
a6d97b67a8
MINOR: fix system profile return types (#18470)
* fix(redshift-system): redshift return type

* fixed bigquery profiler

* fixed snowflake profiler

* job id action does not support matrix. using plain action summary.

* reverted gha change
2024-11-11 10:49:42 +01:00
Imri Paran
84391e7078
MINOR: tests: fix Tuple in bigquery e2e cli (#18499)
* tests: fix Tuple in bigquery e2e cli

* tests: fix Tuple in bigquery e2e cli

* fix workflow condition
2024-11-04 09:54:10 -08:00
Imri Paran
95982b9395
[GEN-356] Use ServiceSpec for loading sources based on connectors (#18322)
* ref(profiler): use di for system profile

- use source classes that can be overridden in system profiles
- use a manifest class instead of factory to specify which class to resolve for connectors
- example usage can be seen in redshift and snowflake

* - added manifests for all custom profilers
- used super() dependency injection in order for system metrics source
- formatting

* - implement spec for all source types
- added docs for the new specification
- added some pylint ignores in the importer module

* remove TYPE_CHECKING in core.py

* - deleted valuedispatch function
- deleted get_system_metrics_by_dialect
- implemented BigQueryProfiler with a system metrics source
- moved import_source_class to BaseSpec

* - removed tests related to the profiler factory

* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger

* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger

* fixed tests

* format

* bigquery system profile e2e tests

* fixed module docstring

* - removed import_side_effects from redshift. we still use it in postgres for the orm conversion maps.
- removed leftover methods

* - tests for BaseSpec
- moved get_class_path to importer

* - moved constructors around to get rid of useless kwargs

* - changed test_system_metric

* - added linage and usage to service_spec
- fixed postgres native lineage test

* add comments on collaborative constructors
2024-10-24 07:47:50 +02:00
Suman Maharana
69b34684b5
Fixed mysql E2E (#18229) 2024-10-11 10:49:03 +00:00
Imri Paran
25284e0232
MINOR: fix snowflake system metrics (#17989)
* fix snowflake system metrics

* format

* add link to logs and commit
fixed the dq cli test

* reverted bad formatting

* fixed models.py

* removed version pinning for data diff in tests
2024-09-26 11:55:17 +00:00
IceS2
d36f01abf6
Fix tearDown by using the proper file loader (#17994) 2024-09-25 17:37:56 +02:00
Imri Paran
21af02d8d5
Fix 17903: fix(data-quality): snowflake data diff (#17907)
* fix(data-quality): snowflake data diff

- fixed schema in snowflake URL for data diff
- added e2e for snowflake data quality

* reverted unintended change
2024-09-19 17:55:48 +02:00
Imri Paran
84be1a3162
Fix 17698: use resolution logic for snowflake system metrics profiler (#17699)
* fix(profiler): snowflake

resolve tables using the snowflake engine instead of OpenMetadata

* added env for cleaning up dbs in E2E

* moved system metric method to profiler. all the rest says in snowflake

* format

* revert unnecessary changes

* removed test for previous resolution method

* use shutdown39
2024-09-06 07:25:10 +00:00
Ayush Shah
b2f21fa070
Fixes #17085: Update Oracle count & unique count function to handle HexByteString/Blob types (#17596) 2024-09-06 12:42:41 +05:30
Pere Miquel Brull
2180a6c7f1
FIX - profiler interface system metrics validation & e2e YAML includeDDL (#17562) 2024-08-23 09:00:18 +02:00
Suman Maharana
056affbe0b
Fix mysql e2e test (#17265) 2024-08-02 20:59:33 +05:30
Onkar Ravgan
7272281cf0
fixed tableau owner and e2e counts (#17266) 2024-08-02 15:17:09 +05:30
Suman Maharana
71bd67a503
Fix DBT e2e (#17260) 2024-08-01 15:34:17 +05:30
Suman Maharana
80aa9fd80f
Fix: bigquery e2e test fix (#17192) 2024-07-26 16:14:31 +05:30
Suman Maharana
77529d4fa0
Minor: Fix tableau e2e changed filter count (#17180) 2024-07-25 21:01:04 +05:30
IceS2
d145341026
MINOR: Fix Oracle E2E Tests (#17084)
* Fix Oracle E2E Tests

* Fix Checkstyle

* Add link to issue in the reason
2024-07-19 10:32:22 +02:00
IceS2
aa3d54aede
Fix Vertica E2E Counts (#17071) 2024-07-18 14:54:15 +02:00
IceS2
f845663317
Update test_cli_metabase.py (#17026) 2024-07-16 10:05:27 +02:00
Ayush Shah
eeda6d24ae
Fixes #16697: Modify the Query to avoid Numeric Data Overflow (#16920) 2024-07-12 17:23:53 +05:30
Onkar Ravgan
49876b9cd6
fixed e2e for mysql and quicksight (#16959) 2024-07-08 20:50:02 +05:30
Pere Miquel Brull
7e98ece3e5
MINOR - Pydantic V2 warnings and better exception msg (#16916) 2024-07-04 14:54:41 +02:00
Imri Paran
1b833e1a01
MINOR: exclude information_schema in redshift (#16818)
* fix(redshift): exclude information_schema

exclude information_schema in tests and document it as best practice.

* format
2024-06-28 08:01:52 +02:00
IceS2
3c5d705793
Fixed expected result for schemas exclude (#16757) 2024-06-21 09:42:35 +00:00
IceS2
7020c065ec
Fix Tableau E2E Tests (#16740) 2024-06-20 14:11:03 +00:00
IceS2
640d7c7532
MINOR: Fix Redash e2e filtered_mix test (#16684)
* Fix E2E count

* Fix E2E Redash filtered mix test
2024-06-17 17:46:47 -07:00
IceS2
e3c3f2ac33
Fix E2E count (#16654) 2024-06-14 08:21:07 +00:00
IceS2
070bd2570e
MINOR: Fix e2e (#16594)
* Fix Metabase E2E Test

* Add 'debug' input to python e2e tests

* Fix 'debug' default to be 'false'

* Standardized all Metabase IDs to MetabaseStrId

* Fix Metabase expected filtered sink mix value

* Fix wrong parameter being passed to the config

* Fix powerBI e2e tests

* Fix one Redash e2e test

* Fix checkstyle
2024-06-11 13:19:25 +05:30
Pere Miquel Brull
93f976f6e4
FIX - e2e test for pydantic v2 (#16576)
* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2
2024-06-08 10:07:58 +00:00
Pere Miquel Brull
f5996b2f3a
FIX - e2e test for pydantic v2 (#16572) 2024-06-07 22:52:16 +02:00
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 (#16551)
* Fix - e2e tests for pydantic V2

* add correct default

* add correct default

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* fix apis

* format
2024-06-06 19:36:17 -07:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Onkar Ravgan
0106963e7e
fixed tableau e2e (#16534) 2024-06-05 14:21:35 +05:30
Onkar Ravgan
29028eabb3
fixed oracle e2e count (#16530) 2024-06-04 20:37:21 +05:30
Onkar Ravgan
3c083cdb68
updated pbi e2e counts (#16109) 2024-05-03 06:57:15 +02:00
IceS2
19fa15f010
fix e2e (#15981) 2024-04-22 09:57:06 +02:00