2535 Commits

Author SHA1 Message Date
k.nakagaki
9a31a35296
Fixes 9875: supporting gcp secret manager (#16505)
* Split ExternalSecretsManagerTest to new ExternalSecretsManagerTest and AWSBasedSecretsManagerTest

* implement SecretsManagerFactory to create GCPSecretsManager

* implement GCPSecretsManager

* implements gcp secret manager.

* Fix it for the GCP's rule.

* create a template of GCP

* fix compile error

* implements to use project_id

* add library for the google cloud secret manager

* add test code for using google credential in the docker container

* modify docker-compose.yml for GCP

* add google_crc32c module

* modify ways to get project id

* create a new docker-compose.yml for Google Cloud

* create a new document

* create compose file for gcp secret manager

* fix invalid styles and formats

* downgrade google library to avoid conflicting protoc versions
2024-06-28 21:09:02 -07:00
Pere Miquel Brull
166d9ef43d
#10192 - Validate both include and excludes filters (#16826) 2024-06-28 07:11:18 +02:00
Ayush Shah
fe04b0a201
Fixes #16435: Fix pyodbc error for mssql (#16800) 2024-06-27 15:46:06 +05:30
Suman Maharana
2477bbc9ca
Fixes #16574 : Fix Databricks pipeline task status ingestion (#16802)
* Fixed Databricks pipeline task issue

* updated test cases

* addressed review
2024-06-27 12:07:04 +05:30
Mayur Singal
bd2bf4a044
MINOR: Include default values in custom pydantic model (#16795) 2024-06-26 20:31:02 +05:30
Ayush Shah
2e694b5180
Minor: Fixes Bigquery External Account AttributeError (#16801) 2024-06-26 16:49:48 +02:00
harshsoni2024
3f5bc1948d
Fix #14676: Athena S3 Lineage (#16426)
* get table ddl for athena tables

* changes in method to get all table ddls

* external table/container lineage for athena

* column lineage for external table lineage

* unittest for athena

* pyformat changes

* add external table lineage unit test

* fix unittest with pydantic v2 changes

* fix unittest formating

* fix code smell
2024-06-26 19:53:36 +05:30
harshsoni2024
afec7703cc
Fix #11888: Apache flink connector (#16601) 2024-06-26 19:21:16 +05:30
Imri Paran
46560e0b47
fixed lineage error (#16790) 2024-06-25 12:36:13 +00:00
Teddy
d38194f4a7
fix: removed __init__ in cli module for namespace path (#16775) 2024-06-25 07:36:01 +00:00
Imri Paran
5e5c811ef2
moved int_admin_ometa to a dedicated module (#16768) 2024-06-25 11:21:22 +05:30
Imri Paran
54ca82f64d
MINOR: raise lineage error when table does not exist (#16756)
* raise lineage error when table does not exist

* added test case for partial success

* format

* format

* fixed tests
2024-06-24 21:41:59 +05:30
Teddy
38fe061227
MINOR -- Add Test Definition Dimension (#16769)
* feat: added test defintion dimension + sample data for bounds

* chore: added migration for definition dimension

* style: ran python linting

* fix: rename dimension to dataQualityDimension

* fix: test definition dimension key
2024-06-24 15:01:12 +00:00
Ayush Shah
e9c7c398c5
Fix return type annotation (#16772) 2024-06-24 13:57:59 +00:00
Teddy
141ceb4c8d
MNINOR add common test elements to _openmetadata_testutils module (#16758)
* fix: add common test to _testutils module

* fix: renamed _testutils to _openmetadata_testutils
2024-06-21 15:11:34 +02:00
Suman Maharana
f425d9bf73
Fixes #16591 : DBT Glossary Term linking Independent from Tag rule (#16745) 2024-06-21 17:36:25 +05:30
Suman Maharana
5bd48fcc34
Fixes #14065 : Added DBT Cloud connector (#16705) 2024-06-21 17:16:47 +05:30
Ikko Eltociear Ashimine
6539c95d3f
chore: update percentille_mixin.py (#16743) 2024-06-21 17:14:27 +05:30
Onkar Ravgan
ceaf4bf08a
MINOR: Add method to list custom properties for a entity for python sdk (#16753)
* List custom properties for a entity

* added test

* fixed test
2024-06-21 16:34:49 +05:30
Matt Chamberlin
ac6ddbf6c4
MINOR: support JSONL datalake file types (#16614)
* fix: support JSONL datalake file types

* add jsonl zip file types

* update fileFormat enum in table schema

* add tests

* fix test data ref

* reformat

* fix tests

---------

Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-06-21 09:54:19 +02:00
IceS2
7411f9e0e1
Enable Primary Key Constraint for DB2 (#16755) 2024-06-21 09:45:43 +02:00
Imri Paran
b960b60965
Fix #16421: add tableDiff test case (#16554)
* feat: add tableDiff test case

This changed introduces a "table diff" test case which
compares two tables and fails if they are not identical.
The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections).

1. Added the `tableDiff` test definition.
2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime.
3. Integration tests (because of course).

This feature was not tested end-to-end yet because "array" data

* pydantic v2

* format

* format

* format and added data diff to setup.py

* format

* fixed param issue which has type ARRAY

* fixed runtime_parameter_setter

* moved models to parent directory

* handle errors in table diff

* fixed issue with edit test case

* format

* added more details to pytest skip

* format

* refactor: Improve createTestCaseParameters function in DataQualityUtils

* fixed unit test

* removed unused fixture

* removed validator.py

* fixed tests

* added validate kwarg to tests_mixin

* removed "postgres" data diff extra as they interfere with psycopg2-binary

* fixed tests

* pinned tenacity for tests

* reverted tenacity pinning

* added ui support for test diff

* fixed dq cypress and added edit flow

* organized the test case

* added dialect support

* fixed tests

* option style fix

* fixed calculation for passing/failing rows

* restrict the tableDiff test to limited services

* set where to None if blank string

* fixed where clause

* fixed tests for where clause

* use displayName in place of name in edit form

* added docs for RuntimeParameterSetter

* fixed cypress

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
Ayush Shah
c9a017d8db
#16720: Add Support for Salesforce SSL (#16719) 2024-06-20 12:10:41 +05:30
IceS2
f0049853ec
FIXES 14885: Initial deltalake implementation for s3 (#16665)
* Initial deltalake implementation for s3

* Fix styles

* Fix test_amundsen

* Fix UnitTests

* Fix Checkstyle

* Fix integration tests due to datalake client refactor

* Fix unit tests

* Fix tests

* Fix Integration DeltaLake Storage test

* Skip delta storage integration test for python 3.8

* DeltaLake JSONSchema changes migrations

* Update import name

* Add some comments based on sonarcloud suggestions

* Update DeltaLake documentation

* Resolve some comments
2024-06-20 12:08:21 +05:30
IceS2
6b7e9238f7
Update ischema_names for db2 to take BOOLEAN into account (#16718) 2024-06-19 14:03:02 +02:00
Imri Paran
c79c5da00a
Fixes #16632: Add testCases property to a test suite source config (#16631)
* feat: test-suites

enable specifying a subset of test cases to run in a test suite pipeline

* improved property description
2024-06-19 12:47:38 +02:00
Imri Paran
955ed5afe1
MINOR: fix sql lineage with timezone change (#16685)
* skip mssql test for python

* fixed mssql lineage
2024-06-18 07:15:52 +00:00
Mayur Singal
57e51df05f
MINOR: Fix superset cypress error (#16689) 2024-06-18 11:36:51 +05:30
Mayur Singal
72ed09d6f6
MINOR: Fix column merge error lineage (#16670)
* MINOR: Fix Column merge error

* append only when col lineage available

* pyformat
2024-06-16 20:32:21 -07:00
Ayush Shah
b3eae8c1b9
Minor: Fix Deprecated utcnow to timezone support (#16607) 2024-06-14 15:23:51 +05:30
Mayur Singal
e3fa340c8f
MINOR: Pydantic fixes for redshift & kafka (#16638) 2024-06-14 14:08:59 +05:30
Trs
fc9033b953
Fixes(ingestion/source/dbt): Handle None Type in get_tag_labels Function for DBT Metadata Processing (#16648)
* fix condition

* fix

* lint
2024-06-13 17:19:46 +05:30
Ayush Shah
90d9ef639e
Fix Arg for qliksense (#16626) 2024-06-13 09:01:45 +05:30
IceS2
328ed2bf11
MINOR: Fix e2e (#16627)
* Fix Metabase E2E Test

* Add 'debug' input to python e2e tests

* Fix 'debug' default to be 'false'

* Standardized all Metabase IDs to MetabaseStrId

* Fix Metabase expected filtered sink mix value

* Fix wrong parameter being passed to the config

* Fix powerBI e2e tests

* Fix one Redash e2e test

* Fix checkstyle

* Fix Dashboard create patch_request not using EntityReferenceList

* Fix Redash E2E test value

* Add logging to create patch request

* Fix checkstyle and linting

* Fix default debug value

* Fix e2e workflow

* Fix e2e workflow

* Fix e2e workflow

* Fix metabase and powerbi e2e values
2024-06-12 19:32:45 +02:00
Mayur Singal
b7ef13bc95
MINOR: Fix ssl connection in usage & lineage (#16625) 2024-06-12 16:29:24 +05:30
Ayush Shah
2f8de77140
Add Test Suite SSL (#16619) 2024-06-12 14:00:24 +05:30
Mayur Singal
7359d6210c
MINOR: Fix Profiler for SSL Enabled Source (#16613) 2024-06-12 11:40:30 +05:30
Onkar Ravgan
4f50e0b6a0
cleaned dbt code after pydantic update (#16605) 2024-06-12 11:24:29 +05:30
Matt Chamberlin
5236950b6d
Fixes 16595: Parse Iceberg REST table FQN from identifier (#16596) 2024-06-11 15:07:57 +05:30
IceS2
070bd2570e
MINOR: Fix e2e (#16594)
* Fix Metabase E2E Test

* Add 'debug' input to python e2e tests

* Fix 'debug' default to be 'false'

* Standardized all Metabase IDs to MetabaseStrId

* Fix Metabase expected filtered sink mix value

* Fix wrong parameter being passed to the config

* Fix powerBI e2e tests

* Fix one Redash e2e test

* Fix checkstyle
2024-06-11 13:19:25 +05:30
Teddy
ff00175fb1
MINOR - Data Quality Preparatory Work (#16523)
* chore: added min/max bound and dynamic support flag

* chore: added migration for dynamic attribute

* style: fix pythin checkstyle
2024-06-10 11:03:55 +02:00
Pere Miquel Brull
d2bba3b8e0
FIX - e2e test for pydantic v2 (#16577)
* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2
2024-06-10 08:04:36 +02:00
Pere Miquel Brull
93f976f6e4
FIX - e2e test for pydantic v2 (#16576)
* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2

* FIX - e2e test for pydantic v2
2024-06-08 10:07:58 +00:00
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 (#16551)
* Fix - e2e tests for pydantic V2

* add correct default

* add correct default

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* fix apis

* format
2024-06-06 19:36:17 -07:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Onkar Ravgan
e9ec4367ee
Fixed quicksight conn (#16537) 2024-06-05 08:54:41 +00:00
Suman Maharana
cdca199ec8
Fix #16370 - Added view/DDL definition rule (#16486)
* added rule includeDDL

* code optimize

* review changes
2024-06-05 11:11:04 +05:30
Teddy
656da03b14
fix: None type is not iterable (#16496) 2024-05-31 10:17:28 +00:00
Mayur Singal
22be0edd79
MINOR: Ignore Cluster Information from columns (#16495) 2024-05-31 15:41:12 +05:30
Ayush Shah
a98f6b8dd0
MINOR: Kafka Setup SSL Arg Fix (#16469) 2024-05-30 16:03:03 +05:30