1157 Commits

Author SHA1 Message Date
Onkar Ravgan
ceaf4bf08a
MINOR: Add method to list custom properties for a entity for python sdk (#16753)
* List custom properties for a entity

* added test

* fixed test
2024-06-21 16:34:49 +05:30
IceS2
3c5d705793
Fixed expected result for schemas exclude (#16757) 2024-06-21 09:42:35 +00:00
Matt Chamberlin
ac6ddbf6c4
MINOR: support JSONL datalake file types (#16614)
* fix: support JSONL datalake file types

* add jsonl zip file types

* update fileFormat enum in table schema

* add tests

* fix test data ref

* reformat

* fix tests

---------

Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-06-21 09:54:19 +02:00
Imri Paran
b960b60965
Fix #16421: add tableDiff test case (#16554)
* feat: add tableDiff test case

This changed introduces a "table diff" test case which
compares two tables and fails if they are not identical.
The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections).

1. Added the `tableDiff` test definition.
2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime.
3. Integration tests (because of course).

This feature was not tested end-to-end yet because "array" data

* pydantic v2

* format

* format

* format and added data diff to setup.py

* format

* fixed param issue which has type ARRAY

* fixed runtime_parameter_setter

* moved models to parent directory

* handle errors in table diff

* fixed issue with edit test case

* format

* added more details to pytest skip

* format

* refactor: Improve createTestCaseParameters function in DataQualityUtils

* fixed unit test

* removed unused fixture

* removed validator.py

* fixed tests

* added validate kwarg to tests_mixin

* removed "postgres" data diff extra as they interfere with psycopg2-binary

* fixed tests

* pinned tenacity for tests

* reverted tenacity pinning

* added ui support for test diff

* fixed dq cypress and added edit flow

* organized the test case

* added dialect support

* fixed tests

* option style fix

* fixed calculation for passing/failing rows

* restrict the tableDiff test to limited services

* set where to None if blank string

* fixed where clause

* fixed tests for where clause

* use displayName in place of name in edit form

* added docs for RuntimeParameterSetter

* fixed cypress

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
IceS2
7020c065ec
Fix Tableau E2E Tests (#16740) 2024-06-20 14:11:03 +00:00
Ayush Shah
c9a017d8db
#16720: Add Support for Salesforce SSL (#16719) 2024-06-20 12:10:41 +05:30
IceS2
f0049853ec
FIXES 14885: Initial deltalake implementation for s3 (#16665)
* Initial deltalake implementation for s3

* Fix styles

* Fix test_amundsen

* Fix UnitTests

* Fix Checkstyle

* Fix integration tests due to datalake client refactor

* Fix unit tests

* Fix tests

* Fix Integration DeltaLake Storage test

* Skip delta storage integration test for python 3.8

* DeltaLake JSONSchema changes migrations

* Update import name

* Add some comments based on sonarcloud suggestions

* Update DeltaLake documentation

* Resolve some comments
2024-06-20 12:08:21 +05:30
Imri Paran
c79c5da00a
Fixes #16632: Add testCases property to a test suite source config (#16631)
* feat: test-suites

enable specifying a subset of test cases to run in a test suite pipeline

* improved property description
2024-06-19 12:47:38 +02:00
Mayur Singal
57e51df05f
MINOR: Fix superset cypress error (#16689) 2024-06-18 11:36:51 +05:30
IceS2
640d7c7532
MINOR: Fix Redash e2e filtered_mix test (#16684)
* Fix E2E count

* Fix E2E Redash filtered mix test
2024-06-17 17:46:47 -07:00
Imri Paran
95d2d0f82f
skip mssql test for python (#16683) 2024-06-17 13:19:04 +00:00
Imri Paran
18206393e2
MINOR: added lineage ingestion test for mssql (#16436)
* added test case for SQL Server lineage

* reasons for failing tests
2024-06-17 08:56:28 +02:00
Ayush Shah
b3eae8c1b9
Minor: Fix Deprecated utcnow to timezone support (#16607) 2024-06-14 15:23:51 +05:30
Mayur Singal
e3fa340c8f
MINOR: Pydantic fixes for redshift & kafka (#16638) 2024-06-14 14:08:59 +05:30
IceS2
e3c3f2ac33
Fix E2E count (#16654) 2024-06-14 08:21:07 +00:00
Trs
fc9033b953
Fixes(ingestion/source/dbt): Handle None Type in get_tag_labels Function for DBT Metadata Processing (#16648)
* fix condition

* fix

* lint
2024-06-13 17:19:46 +05:30
Mayur Singal
7359d6210c
MINOR: Fix Profiler for SSL Enabled Source (#16613) 2024-06-12 11:40:30 +05:30
IceS2
070bd2570e
MINOR: Fix e2e (#16594)
* Fix Metabase E2E Test

* Add 'debug' input to python e2e tests

* Fix 'debug' default to be 'false'

* Standardized all Metabase IDs to MetabaseStrId

* Fix Metabase expected filtered sink mix value

* Fix wrong parameter being passed to the config

* Fix powerBI e2e tests

* Fix one Redash e2e test

* Fix checkstyle
2024-06-11 13:19:25 +05:30
Pere Miquel Brull
93f976f6e4
FIX - e2e test for pydantic v2 (#16576)
* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2
2024-06-08 10:07:58 +00:00
Pere Miquel Brull
f5996b2f3a
FIX - e2e test for pydantic v2 (#16572) 2024-06-07 22:52:16 +02:00
Onkar Ravgan
38e2793705
MINOR: Enabled pbit file test (#16531)
* Enabled pbit file test

* Updated test

* update test

* fixed pylint
2024-06-07 17:59:45 +05:30
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 (#16551)
* Fix - e2e tests for pydantic V2

* add correct default

* add correct default

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* fix apis

* format
2024-06-06 19:36:17 -07:00
Mohit Yadav
9ec3d94e3b
[FIX] GlossaryTerm reviewers should be user or team only (#16372)
* add teams as reviewer

* Check Users to be reviewers

* Reviewers can be a team or user

* Fix check by id or name

* Review can be team or user both

* Validate Reviewers

* add multi select control

* - Fix Reviewers

* - Centralize Reviewer Relationship to EntityRepository

* - Sort

* add team as reviewer for glossary terms

* locales

* cleanup

* - Update Reviewer should remove existing reviewers

* fix selectable owner control

* fix code smells

* fix reviewer issue

* add glossary cypress

* fix patch issue on reviewers set to null

* update cypress tests

* fix cypress

* fix cypress

* fix reviewers in glossary task and supported cypress

* fix pytest

* Fix

* fix cypress

* fix code smells

* Inherited Reviewers need to be present always

* filter out inherited users

* fix cypress

* fix backend tests failure

* fix backend tests failure -checkstyle

* restrict owner to accept task in case of reviewer present

* fix pytest

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-06-06 20:23:37 +05:30
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Onkar Ravgan
0106963e7e
fixed tableau e2e (#16534) 2024-06-05 14:21:35 +05:30
Onkar Ravgan
29028eabb3
fixed oracle e2e count (#16530) 2024-06-04 20:37:21 +05:30
Ayush Shah
a98f6b8dd0
MINOR: Kafka Setup SSL Arg Fix (#16469) 2024-05-30 16:03:03 +05:30
Imri Paran
067fb510ab
MINOR: test case for usage->delete->usage (#16409)
* test: added test case to demonstrate cache issue
2024-05-28 11:23:43 +02:00
Imri Paran
a4c516d2c7
Fixes 16305: Added Test Case for Matching Enum (#16362)
* Added Test Case for Matching Enum

1. Implemented the test case using the `matchEnum` parameter.
2. Added integration tests.
3. Added migrations.

* fix tests

* fixed tests

* format

* fixed tests

* clear search cache before running ingestion

* format

* changed scopt of aws fixture

* moved migrations to 1.5.0
2024-05-28 09:30:30 +02:00
juntao
8dd613caa5
Fixes #16235: need quote fullyQualifiedName in Ingestion Framework (#16273)
* Fixes #16235: need quote fullyQualifiedName in Ingestion Framework

* MINOR: fix UT issue

* revert: fix UT issue

* revert code

* revert code

* format code
2024-05-23 17:45:47 +02:00
Imri Paran
d5bf30ccd3
MINOR: trino integration test (#16291)
* added trino integration test

* - removed warnings for classes which are not real tests
- removed "helpers" as its being used

* use a docker network instead of host

* print logs for hive failure

* removed superset unit tests

* try pinning requests for test

* try pinning requests for test

* wait for hive to be ready

* fix trino fixture

* - reduced testcontainers_config.max_tries to 5
- remove intermediate containers

* print with logs

* disable capture logging

* updated db host

* removed debug stuff

* removed debug stuff

* removed version pin for requests

* reverted superset

* ignore trino integration on python 3.8
2024-05-22 15:12:00 +00:00
Mayur Singal
89829949ce
MINOR: Fix flaky pytests (#16379) 2024-05-22 14:12:34 +05:30
Suman Maharana
bd3f47a563
MINOR - Added quicksight pydantic models (#16269)
* Added quicksight pydantic models

* pyformat

* resolved type hints

* Renamed sheet -> chart in models
2024-05-17 08:40:20 +02:00
harshsoni2024
d6046c811c
MINOR: superset pytest fix (#16299)
* superset testcontainer image tag fix

* update chart data acc. to runtime
2024-05-17 07:27:36 +02:00
Imri Paran
7af7f9322f
MINOR: skip pbit test (#16312)
* skip superset test in CI

* fixed pytest mark

* format

* format

* skip pbi

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-05-17 07:27:12 +02:00
Pere Miquel Brull
53185fd30b
MINOR - Add Integration Test for S3 Storage (#16277)
* MINOR - Add Integration Test for S3 Storage

* MINOR - Add Integration Test for S3 Storage

* MINOR - Add Integration Test for S3 Storage

* format

* format
2024-05-16 10:03:27 +02:00
Pere Miquel Brull
f4369ded8d
MINOR - Allow dict-based xlets in Airflow (#16262) 2024-05-15 15:08:09 +02:00
Mayur Singal
a677910982
Fix #16211: Fix special charechter encoding in trino username (#16258)
* Fix #16211: Fix special charechter encoding in trino username

* fix test and add comments
2024-05-15 14:48:22 +05:30
Mayur Singal
945cd35148
MINOR: Fix Oracle SP Lineage for begin...end SP call (#16240) 2024-05-15 14:46:57 +05:30
Imri Paran
c277233ef1
MINOR: use archive instead of volume for postgres test (#16245)
* using archive instead of volume for postgres test

* format

* remove usage of request
2024-05-14 09:11:16 +00:00
Suman Maharana
0e2736ee74
MINOR: Removed supportsDDL from json schemas (#16171) 2024-05-10 17:40:12 +05:30
Pere Miquel Brull
39eed12f32
MINOR - Version match logic update & Airflow docs (#16157)
* airflow docs

* update version validation

* MINOR - docs and version match
2024-05-08 07:37:14 +02:00
Onkar Ravgan
4a6849a05d
MINOR: Added custom property EntityReference support to python sdk (#16132)
* Added cust prop entityref to python sdk

* Added name and displayName fields to entityref
2024-05-07 17:35:39 +05:30
Mohit Yadav
0769d71ee7
Fixes Test Suite Reference in Table Schema (#16129)
* Fixes Test Suite Reference in Table Schema

* fix: fix test suite to interact with entity reference

---------

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2024-05-06 19:03:23 +05:30
Suman Maharana
488078da8a
Add DDL query ingest (#15860) 2024-05-06 18:03:50 +05:30
IceS2
795879d776
MINOR: Fix issue with SQLAlchemy types not being correctly mapped to OM Type on the profiler (#16122)
* Fix issue with SQLAlchemy types not being correctly mapped to OM Types on the profiler

* Fix checkstyle
2024-05-03 17:05:52 +02:00
Onkar Ravgan
3c083cdb68
updated pbi e2e counts (#16109) 2024-05-03 06:57:15 +02:00
Onkar Ravgan
87c8254c38
Fix #15454: Added protobuf parser complex schema support (#16071)
* Added protobuf parser complex schema support

* Added options keyword in proto testing
2024-04-30 17:59:27 +05:30
Onkar Ravgan
ceaa9d3e8a
Fix #15611 Parse PowerBI Dax files for lineage (#15975) 2024-04-29 14:55:06 +05:30
harshsoni2024
68e036418c
Fix #15719: Improve unit test to increase coverage. (#15905)
* issue-15719: unit test for superset db source

* issue-15719: use testcontainers for superset_api client test

* issue-15719: superset-api yield data changes

* fix failed test cases due to testcontainer version

* issue-15719: postgres container version fix

* issue-15719: setup & teardown with testcontainers

* issue-15719: remove more patch code
2024-04-29 08:00:39 +02:00