843 Commits

Author SHA1 Message Date
IceS2
e3c3f2ac33
Fix E2E count (#16654) 2024-06-14 08:21:07 +00:00
Trs
fc9033b953
Fixes(ingestion/source/dbt): Handle None Type in get_tag_labels Function for DBT Metadata Processing (#16648)
* fix condition

* fix

* lint
2024-06-13 17:19:46 +05:30
Mayur Singal
7359d6210c
MINOR: Fix Profiler for SSL Enabled Source (#16613) 2024-06-12 11:40:30 +05:30
IceS2
070bd2570e
MINOR: Fix e2e (#16594)
* Fix Metabase E2E Test

* Add 'debug' input to python e2e tests

* Fix 'debug' default to be 'false'

* Standardized all Metabase IDs to MetabaseStrId

* Fix Metabase expected filtered sink mix value

* Fix wrong parameter being passed to the config

* Fix powerBI e2e tests

* Fix one Redash e2e test

* Fix checkstyle
2024-06-11 13:19:25 +05:30
Pere Miquel Brull
93f976f6e4
FIX - e2e test for pydantic v2 (#16576)
* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2
2024-06-08 10:07:58 +00:00
Pere Miquel Brull
f5996b2f3a
FIX - e2e test for pydantic v2 (#16572) 2024-06-07 22:52:16 +02:00
Onkar Ravgan
38e2793705
MINOR: Enabled pbit file test (#16531)
* Enabled pbit file test

* Updated test

* update test

* fixed pylint
2024-06-07 17:59:45 +05:30
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 (#16551)
* Fix - e2e tests for pydantic V2

* add correct default

* add correct default

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* fix apis

* format
2024-06-06 19:36:17 -07:00
Mohit Yadav
9ec3d94e3b
[FIX] GlossaryTerm reviewers should be user or team only (#16372)
* add teams as reviewer

* Check Users to be reviewers

* Reviewers can be a team or user

* Fix check by id or name

* Review can be team or user both

* Validate Reviewers

* add multi select control

* - Fix Reviewers

* - Centralize Reviewer Relationship to EntityRepository

* - Sort

* add team as reviewer for glossary terms

* locales

* cleanup

* - Update Reviewer should remove existing reviewers

* fix selectable owner control

* fix code smells

* fix reviewer issue

* add glossary cypress

* fix patch issue on reviewers set to null

* update cypress tests

* fix cypress

* fix cypress

* fix reviewers in glossary task and supported cypress

* fix pytest

* Fix

* fix cypress

* fix code smells

* Inherited Reviewers need to be present always

* filter out inherited users

* fix cypress

* fix backend tests failure

* fix backend tests failure -checkstyle

* restrict owner to accept task in case of reviewer present

* fix pytest

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-06-06 20:23:37 +05:30
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Onkar Ravgan
0106963e7e
fixed tableau e2e (#16534) 2024-06-05 14:21:35 +05:30
Onkar Ravgan
29028eabb3
fixed oracle e2e count (#16530) 2024-06-04 20:37:21 +05:30
Ayush Shah
a98f6b8dd0
MINOR: Kafka Setup SSL Arg Fix (#16469) 2024-05-30 16:03:03 +05:30
Imri Paran
067fb510ab
MINOR: test case for usage->delete->usage (#16409)
* test: added test case to demonstrate cache issue
2024-05-28 11:23:43 +02:00
Imri Paran
a4c516d2c7
Fixes 16305: Added Test Case for Matching Enum (#16362)
* Added Test Case for Matching Enum

1. Implemented the test case using the `matchEnum` parameter.
2. Added integration tests.
3. Added migrations.

* fix tests

* fixed tests

* format

* fixed tests

* clear search cache before running ingestion

* format

* changed scopt of aws fixture

* moved migrations to 1.5.0
2024-05-28 09:30:30 +02:00
juntao
8dd613caa5
Fixes #16235: need quote fullyQualifiedName in Ingestion Framework (#16273)
* Fixes #16235: need quote fullyQualifiedName in Ingestion Framework

* MINOR: fix UT issue

* revert: fix UT issue

* revert code

* revert code

* format code
2024-05-23 17:45:47 +02:00
Imri Paran
d5bf30ccd3
MINOR: trino integration test (#16291)
* added trino integration test

* - removed warnings for classes which are not real tests
- removed "helpers" as its being used

* use a docker network instead of host

* print logs for hive failure

* removed superset unit tests

* try pinning requests for test

* try pinning requests for test

* wait for hive to be ready

* fix trino fixture

* - reduced testcontainers_config.max_tries to 5
- remove intermediate containers

* print with logs

* disable capture logging

* updated db host

* removed debug stuff

* removed debug stuff

* removed version pin for requests

* reverted superset

* ignore trino integration on python 3.8
2024-05-22 15:12:00 +00:00
Mayur Singal
89829949ce
MINOR: Fix flaky pytests (#16379) 2024-05-22 14:12:34 +05:30
Suman Maharana
bd3f47a563
MINOR - Added quicksight pydantic models (#16269)
* Added quicksight pydantic models

* pyformat

* resolved type hints

* Renamed sheet -> chart in models
2024-05-17 08:40:20 +02:00
harshsoni2024
d6046c811c
MINOR: superset pytest fix (#16299)
* superset testcontainer image tag fix

* update chart data acc. to runtime
2024-05-17 07:27:36 +02:00
Imri Paran
7af7f9322f
MINOR: skip pbit test (#16312)
* skip superset test in CI

* fixed pytest mark

* format

* format

* skip pbi

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-05-17 07:27:12 +02:00
Pere Miquel Brull
53185fd30b
MINOR - Add Integration Test for S3 Storage (#16277)
* MINOR - Add Integration Test for S3 Storage

* MINOR - Add Integration Test for S3 Storage

* MINOR - Add Integration Test for S3 Storage

* format

* format
2024-05-16 10:03:27 +02:00
Pere Miquel Brull
f4369ded8d
MINOR - Allow dict-based xlets in Airflow (#16262) 2024-05-15 15:08:09 +02:00
Mayur Singal
a677910982
Fix #16211: Fix special charechter encoding in trino username (#16258)
* Fix #16211: Fix special charechter encoding in trino username

* fix test and add comments
2024-05-15 14:48:22 +05:30
Mayur Singal
945cd35148
MINOR: Fix Oracle SP Lineage for begin...end SP call (#16240) 2024-05-15 14:46:57 +05:30
Imri Paran
c277233ef1
MINOR: use archive instead of volume for postgres test (#16245)
* using archive instead of volume for postgres test

* format

* remove usage of request
2024-05-14 09:11:16 +00:00
Suman Maharana
0e2736ee74
MINOR: Removed supportsDDL from json schemas (#16171) 2024-05-10 17:40:12 +05:30
Pere Miquel Brull
39eed12f32
MINOR - Version match logic update & Airflow docs (#16157)
* airflow docs

* update version validation

* MINOR - docs and version match
2024-05-08 07:37:14 +02:00
Onkar Ravgan
4a6849a05d
MINOR: Added custom property EntityReference support to python sdk (#16132)
* Added cust prop entityref to python sdk

* Added name and displayName fields to entityref
2024-05-07 17:35:39 +05:30
Mohit Yadav
0769d71ee7
Fixes Test Suite Reference in Table Schema (#16129)
* Fixes Test Suite Reference in Table Schema

* fix: fix test suite to interact with entity reference

---------

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2024-05-06 19:03:23 +05:30
Suman Maharana
488078da8a
Add DDL query ingest (#15860) 2024-05-06 18:03:50 +05:30
IceS2
795879d776
MINOR: Fix issue with SQLAlchemy types not being correctly mapped to OM Type on the profiler (#16122)
* Fix issue with SQLAlchemy types not being correctly mapped to OM Types on the profiler

* Fix checkstyle
2024-05-03 17:05:52 +02:00
Onkar Ravgan
3c083cdb68
updated pbi e2e counts (#16109) 2024-05-03 06:57:15 +02:00
Onkar Ravgan
87c8254c38
Fix #15454: Added protobuf parser complex schema support (#16071)
* Added protobuf parser complex schema support

* Added options keyword in proto testing
2024-04-30 17:59:27 +05:30
Onkar Ravgan
ceaa9d3e8a
Fix #15611 Parse PowerBI Dax files for lineage (#15975) 2024-04-29 14:55:06 +05:30
harshsoni2024
68e036418c
Fix #15719: Improve unit test to increase coverage. (#15905)
* issue-15719: unit test for superset db source

* issue-15719: use testcontainers for superset_api client test

* issue-15719: superset-api yield data changes

* fix failed test cases due to testcontainer version

* issue-15719: postgres container version fix

* issue-15719: setup & teardown with testcontainers

* issue-15719: remove more patch code
2024-04-29 08:00:39 +02:00
Teddy
4ed87a4d08
Fix #15341 - Test Case reference as inherited field for Test Case Incident (#16027)
* fix: unique test computation to scalar_subquery

* fix: make test case reference an inherited field

* style: ran java linting

* fix: added test case resolution migration

* style: ran java linting
2024-04-25 17:31:11 +02:00
Onkar Ravgan
828e9abc97
Added enum support in custom prop python sdk (#16026) 2024-04-25 14:46:55 +05:30
Ayush Shah
3621407642
Fixes #15732: Modify Reference for Tags to EntityName (#15938) 2024-04-25 11:53:46 +05:30
Ayush Shah
a15da7ec98
Issue #14812: Add support for empty string as missing count (#16017) 2024-04-25 09:45:26 +05:30
Ayush Shah
0963a111fe
Fixes #12127: Add Support for Complex types of Databricks & UnityCatalog in profiler (#15976) 2024-04-23 15:54:36 +05:30
Pere Miquel Brull
df5d5e1866
MINOR - Fix datamodel lineage call (#15991)
* MINOR - Fix datamodel lineage call

* amend merge
2024-04-23 09:56:24 +02:00
Mayur Singal
85b6983eee
Fix #15062 & #14810: Fix Column level lineage overwrites pipeline Lineage & manual col lineage (#15897) 2024-04-23 09:37:43 +05:30
Teddy
449a5f2de3
FIX #11951 - ingestion logic for global profiler config (#15948)
* feat: add global metric configuration for the profiler

* style: ran java linting

* fix: renamed disable to disabled

* style: ran java linting

* feat: ometa sdk for profiler setting

* test: ingestion profiler global config tests

* fix: update metric name to use MetricType Enum

* fix: allow bot to retrieve settings

* fix: exclude GX artifacts

* feat: implement global profiler setting logic for ingestion side

* fix: exclude metrics if Metric is empty

* style: ran python linting

* style: ran python linting

* fix: skip empty metrics

* style: ran python linting

* fix: moved GET profiler config to seperate endpoint in system resource

* fix: moved compute metric filter to MetricFilter + renamed container

* fix: test failures

* fix: profiler test case
2024-04-22 22:35:37 +02:00
Imri Paran
93ec391f5c
MINOR: Dynamodb sample data (#15264)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* feat(profiler): add dynamodb row count

* feat(profiler): add dynamodb row count

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format

* feat(nosql-profiler): added sample data

1. Implemented the NoSQL sampler.
2. Some naming changes to the NoSQL adaptor to avoid fixing names with the profiler interface.
3. Tests.

* added default sample limit

* formatting

* fixed import

* feat(profiler): dynamodb sample data

* tests for dynamo db sample data

* format

* format

* use service connection for nosql adaptor factory

* fixed tests

* format

* fixed after merge
2024-04-22 17:46:40 +02:00
IceS2
cb801dedb4
FIXES 13209: Add Sagemaker Model Storage (#15986)
* Add Sagemaker Model Storage

* Fix checkstyle

* Sagemaker unittest

* Small refactor to be less verbose
2024-04-22 16:53:25 +02:00
IceS2
08c114c340
FIXES 15626: Fix issue with not url model store (#15974)
* Changed the MLModelStore storage type to string

* fix checkstyle

* remove unused files

* Update requirements

* fix checkstyle

* Skipping MLFlow intergration on python 3.8

* Hack to allow pytest to parse the mlflow integrations test on python 3.8

* Fix checkstyle
2024-04-22 15:50:44 +02:00
IceS2
19fa15f010
fix e2e (#15981) 2024-04-22 09:57:06 +02:00
Imri Paran
0a1018648c
Fixes #15566: add dynamodb row count (#15204)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* feat(profiler): add dynamodb row count

* feat(profiler): add dynamodb row count

* formatting

* fixed import

* format

* dded dynamodb row count

* format

* removed unused factory file

* removed "validate"

* migrations

* removed validations

* format

* linting

* fixed: test_amundsen.py

* Update schemaChanges.sql
2024-04-22 09:14:52 +02:00
Imri Paran
d8781bbef2
MINOR: postgres integration test (#15929)
* implemented postgres-integration-tests

* format

* format

* - disable ryuk
- disabled verbose sqlfluff logging

* query usage assertion
2024-04-19 10:00:37 -07:00