176 Commits

Author SHA1 Message Date
harshsoni2024
17dd182cbb
e2e fix (#20952) 2025-04-24 15:23:43 +05:30
Keshav Mohta
1063e019ba
Fixes: Bigquery E2E (#20863) 2025-04-17 11:43:14 +05:30
Keshav Mohta
1a6224824b
Fixes: BQ Multiple Project E2E (#20797)
* fix: bq e2e lineage and counts

* fix: bigquery multiple project classify

* fix: tests count from 19 to 17
2025-04-15 17:35:22 +05:30
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
Ayush Shah
60974e4ea1
Revert "Fixes #17660: Oracle handle quotes for lowercase columns in workflow agents (#20309)" (#20364) 2025-03-20 21:02:58 +05:30
Ayush Shah
20ab64d1f1
Fixes #17660: Oracle handle quotes for lowercase columns in workflow agents (#20309) 2025-03-18 15:48:58 +05:30
harshsoni2024
dba37820d7
MINOR: e2e fixes (#20301) 2025-03-17 21:00:26 +05:30
harshsoni2024
9bf1ce53ec
MINOR: fix-e2e-tests (#20233) 2025-03-13 20:32:06 +05:30
harshsoni2024
aedbe8be2d
fix pbi, vertica, metabase tests (#20190) 2025-03-11 16:40:56 +01:00
Mayur Singal
365fe7daa0
Minor: Fix mysql & snowflake lineage e2e (#19767) 2025-02-13 08:44:01 +05:30
harshsoni2024
f4fd9b6855
MINOR: vertica e2e fix (#19708) 2025-02-07 17:33:04 +05:30
harshsoni2024
69d0e3ce88
powerbi e2e count fix (#19626) 2025-02-03 10:24:59 +05:30
Imri Paran
e003a5f9ed
tests(e2e): data quality (#19513)
* tests(e2e): data quality

- use the data quality pipeline new schema with a list of service configurations

* format
2025-01-24 14:01:54 +00:00
Imri Paran
5fc20c1ab2
tests: fix sample row count for bigquery (#19511)
- changed method name to expected_sample_count
2025-01-24 10:44:21 +01:00
Suman Maharana
89dae42407
Fix mysql e2e count (#19315) 2025-01-10 21:41:38 +05:30
Akash Verma
4cad5762ad
Added lineage in cli e2e (#19216) 2025-01-09 14:59:49 +05:30
harshsoni2024
63aa484fec
MINOR: powerbi e2e fix (#19240) 2025-01-09 13:49:59 +05:30
IceS2
3d83f41b82
Remove the sample_data from assert in profiler since it was moved to the classification workflow (#19153) 2024-12-20 10:48:56 +01:00
IceS2
efbdf21098
Fix e2e tests (#19130)
* Fix E2E

* Fix E2E

* Fix MySQL count

* Tentative Fix

* Tentative Fix

* Revert PowerBI changes

* Revert PowerBI changes
2024-12-19 10:39:30 +01:00
Imri Paran
e30571cf4e
[GEN-2187] fix(data-diff): added MD5 handling for bigquery (#18904)
* fix(data-diff): added nd5 handling for bigquery

- added MD5 handling for bigquery
- use URL instead of Engine because it requires less steps and less prone to failure

* added e2e test for data diff with sampling in bigquery
2024-12-06 14:21:33 +01:00
Pere Miquel Brull
61021be98a
TEST - Add autoClassification for e2e (#18722) 2024-11-21 15:07:04 +01:00
Suman Maharana
a218bbf5cb
Minor: Fix Mysql cli Update table count (#18582) 2024-11-15 14:27:02 +05:30
harshsoni2024
cd3fcb5d22
MINOR: quicksight e2e fix (#18629) 2024-11-14 16:31:11 +05:30
Ayush Shah
6fa03ee66a
Fixes GEN-1994: Remove View Lineage from Metadata Ingestion flow (#18558) 2024-11-13 00:08:55 +05:30
Mayur Singal
f4fdafeb8a
MINOR: Athena & Tableau E2E fix (#18596) 2024-11-12 19:14:45 +05:30
Imri Paran
70c7880dfa
fixed bigquery system metrics e2e test (#18601) 2024-11-12 14:06:54 +01:00
Imri Paran
a6d97b67a8
MINOR: fix system profile return types (#18470)
* fix(redshift-system): redshift return type

* fixed bigquery profiler

* fixed snowflake profiler

* job id action does not support matrix. using plain action summary.

* reverted gha change
2024-11-11 10:49:42 +01:00
Imri Paran
84391e7078
MINOR: tests: fix Tuple in bigquery e2e cli (#18499)
* tests: fix Tuple in bigquery e2e cli

* tests: fix Tuple in bigquery e2e cli

* fix workflow condition
2024-11-04 09:54:10 -08:00
Imri Paran
95982b9395
[GEN-356] Use ServiceSpec for loading sources based on connectors (#18322)
* ref(profiler): use di for system profile

- use source classes that can be overridden in system profiles
- use a manifest class instead of factory to specify which class to resolve for connectors
- example usage can be seen in redshift and snowflake

* - added manifests for all custom profilers
- used super() dependency injection in order for system metrics source
- formatting

* - implement spec for all source types
- added docs for the new specification
- added some pylint ignores in the importer module

* remove TYPE_CHECKING in core.py

* - deleted valuedispatch function
- deleted get_system_metrics_by_dialect
- implemented BigQueryProfiler with a system metrics source
- moved import_source_class to BaseSpec

* - removed tests related to the profiler factory

* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger

* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger

* fixed tests

* format

* bigquery system profile e2e tests

* fixed module docstring

* - removed import_side_effects from redshift. we still use it in postgres for the orm conversion maps.
- removed leftover methods

* - tests for BaseSpec
- moved get_class_path to importer

* - moved constructors around to get rid of useless kwargs

* - changed test_system_metric

* - added linage and usage to service_spec
- fixed postgres native lineage test

* add comments on collaborative constructors
2024-10-24 07:47:50 +02:00
Suman Maharana
69b34684b5
Fixed mysql E2E (#18229) 2024-10-11 10:49:03 +00:00
Imri Paran
25284e0232
MINOR: fix snowflake system metrics (#17989)
* fix snowflake system metrics

* format

* add link to logs and commit
fixed the dq cli test

* reverted bad formatting

* fixed models.py

* removed version pinning for data diff in tests
2024-09-26 11:55:17 +00:00
IceS2
d36f01abf6
Fix tearDown by using the proper file loader (#17994) 2024-09-25 17:37:56 +02:00
Imri Paran
21af02d8d5
Fix 17903: fix(data-quality): snowflake data diff (#17907)
* fix(data-quality): snowflake data diff

- fixed schema in snowflake URL for data diff
- added e2e for snowflake data quality

* reverted unintended change
2024-09-19 17:55:48 +02:00
Imri Paran
84be1a3162
Fix 17698: use resolution logic for snowflake system metrics profiler (#17699)
* fix(profiler): snowflake

resolve tables using the snowflake engine instead of OpenMetadata

* added env for cleaning up dbs in E2E

* moved system metric method to profiler. all the rest says in snowflake

* format

* revert unnecessary changes

* removed test for previous resolution method

* use shutdown39
2024-09-06 07:25:10 +00:00
Ayush Shah
b2f21fa070
Fixes #17085: Update Oracle count & unique count function to handle HexByteString/Blob types (#17596) 2024-09-06 12:42:41 +05:30
Pere Miquel Brull
2180a6c7f1
FIX - profiler interface system metrics validation & e2e YAML includeDDL (#17562) 2024-08-23 09:00:18 +02:00
Suman Maharana
056affbe0b
Fix mysql e2e test (#17265) 2024-08-02 20:59:33 +05:30
Onkar Ravgan
7272281cf0
fixed tableau owner and e2e counts (#17266) 2024-08-02 15:17:09 +05:30
Suman Maharana
71bd67a503
Fix DBT e2e (#17260) 2024-08-01 15:34:17 +05:30
Suman Maharana
80aa9fd80f
Fix: bigquery e2e test fix (#17192) 2024-07-26 16:14:31 +05:30
Suman Maharana
77529d4fa0
Minor: Fix tableau e2e changed filter count (#17180) 2024-07-25 21:01:04 +05:30
IceS2
d145341026
MINOR: Fix Oracle E2E Tests (#17084)
* Fix Oracle E2E Tests

* Fix Checkstyle

* Add link to issue in the reason
2024-07-19 10:32:22 +02:00
IceS2
aa3d54aede
Fix Vertica E2E Counts (#17071) 2024-07-18 14:54:15 +02:00
IceS2
f845663317
Update test_cli_metabase.py (#17026) 2024-07-16 10:05:27 +02:00
Ayush Shah
eeda6d24ae
Fixes #16697: Modify the Query to avoid Numeric Data Overflow (#16920) 2024-07-12 17:23:53 +05:30
Onkar Ravgan
49876b9cd6
fixed e2e for mysql and quicksight (#16959) 2024-07-08 20:50:02 +05:30
Pere Miquel Brull
7e98ece3e5
MINOR - Pydantic V2 warnings and better exception msg (#16916) 2024-07-04 14:54:41 +02:00
Imri Paran
1b833e1a01
MINOR: exclude information_schema in redshift (#16818)
* fix(redshift): exclude information_schema

exclude information_schema in tests and document it as best practice.

* format
2024-06-28 08:01:52 +02:00
IceS2
3c5d705793
Fixed expected result for schemas exclude (#16757) 2024-06-21 09:42:35 +00:00
IceS2
7020c065ec
Fix Tableau E2E Tests (#16740) 2024-06-20 14:11:03 +00:00