harshsoni2024
3f5bc1948d
Fix #14676 : Athena S3 Lineage ( #16426 )
...
* get table ddl for athena tables
* changes in method to get all table ddls
* external table/container lineage for athena
* column lineage for external table lineage
* unittest for athena
* pyformat changes
* add external table lineage unit test
* fix unittest with pydantic v2 changes
* fix unittest formating
* fix code smell
2024-06-26 19:53:36 +05:30
harshsoni2024
afec7703cc
Fix #11888 : Apache flink connector ( #16601 )
2024-06-26 19:21:16 +05:30
Imri Paran
46560e0b47
fixed lineage error ( #16790 )
2024-06-25 12:36:13 +00:00
Teddy
d38194f4a7
fix: removed __init__ in cli module for namespace path ( #16775 )
2024-06-25 07:36:01 +00:00
Imri Paran
5e5c811ef2
moved int_admin_ometa to a dedicated module ( #16768 )
2024-06-25 11:21:22 +05:30
Imri Paran
54ca82f64d
MINOR: raise lineage error when table does not exist ( #16756 )
...
* raise lineage error when table does not exist
* added test case for partial success
* format
* format
* fixed tests
2024-06-24 21:41:59 +05:30
Teddy
38fe061227
MINOR -- Add Test Definition Dimension ( #16769 )
...
* feat: added test defintion dimension + sample data for bounds
* chore: added migration for definition dimension
* style: ran python linting
* fix: rename dimension to dataQualityDimension
* fix: test definition dimension key
2024-06-24 15:01:12 +00:00
Ayush Shah
e9c7c398c5
Fix return type annotation ( #16772 )
2024-06-24 13:57:59 +00:00
Teddy
141ceb4c8d
MNINOR add common test elements to _openmetadata_testutils module ( #16758 )
...
* fix: add common test to _testutils module
* fix: renamed _testutils to _openmetadata_testutils
2024-06-21 15:11:34 +02:00
Suman Maharana
f425d9bf73
Fixes #16591 : DBT Glossary Term linking Independent from Tag rule ( #16745 )
2024-06-21 17:36:25 +05:30
Suman Maharana
5bd48fcc34
Fixes #14065 : Added DBT Cloud connector ( #16705 )
2024-06-21 17:16:47 +05:30
Ikko Eltociear Ashimine
6539c95d3f
chore: update percentille_mixin.py ( #16743 )
2024-06-21 17:14:27 +05:30
Onkar Ravgan
ceaf4bf08a
MINOR: Add method to list custom properties for a entity for python sdk ( #16753 )
...
* List custom properties for a entity
* added test
* fixed test
2024-06-21 16:34:49 +05:30
Matt Chamberlin
ac6ddbf6c4
MINOR: support JSONL datalake file types ( #16614 )
...
* fix: support JSONL datalake file types
* add jsonl zip file types
* update fileFormat enum in table schema
* add tests
* fix test data ref
* reformat
* fix tests
---------
Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-06-21 09:54:19 +02:00
IceS2
7411f9e0e1
Enable Primary Key Constraint for DB2 ( #16755 )
2024-06-21 09:45:43 +02:00
Imri Paran
b960b60965
Fix #16421 : add tableDiff test case ( #16554 )
...
* feat: add tableDiff test case
This changed introduces a "table diff" test case which
compares two tables and fails if they are not identical.
The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections).
1. Added the `tableDiff` test definition.
2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime.
3. Integration tests (because of course).
This feature was not tested end-to-end yet because "array" data
* pydantic v2
* format
* format
* format and added data diff to setup.py
* format
* fixed param issue which has type ARRAY
* fixed runtime_parameter_setter
* moved models to parent directory
* handle errors in table diff
* fixed issue with edit test case
* format
* added more details to pytest skip
* format
* refactor: Improve createTestCaseParameters function in DataQualityUtils
* fixed unit test
* removed unused fixture
* removed validator.py
* fixed tests
* added validate kwarg to tests_mixin
* removed "postgres" data diff extra as they interfere with psycopg2-binary
* fixed tests
* pinned tenacity for tests
* reverted tenacity pinning
* added ui support for test diff
* fixed dq cypress and added edit flow
* organized the test case
* added dialect support
* fixed tests
* option style fix
* fixed calculation for passing/failing rows
* restrict the tableDiff test to limited services
* set where to None if blank string
* fixed where clause
* fixed tests for where clause
* use displayName in place of name in edit form
* added docs for RuntimeParameterSetter
* fixed cypress
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
Ayush Shah
c9a017d8db
#16720 : Add Support for Salesforce SSL ( #16719 )
2024-06-20 12:10:41 +05:30
IceS2
f0049853ec
FIXES 14885: Initial deltalake implementation for s3 ( #16665 )
...
* Initial deltalake implementation for s3
* Fix styles
* Fix test_amundsen
* Fix UnitTests
* Fix Checkstyle
* Fix integration tests due to datalake client refactor
* Fix unit tests
* Fix tests
* Fix Integration DeltaLake Storage test
* Skip delta storage integration test for python 3.8
* DeltaLake JSONSchema changes migrations
* Update import name
* Add some comments based on sonarcloud suggestions
* Update DeltaLake documentation
* Resolve some comments
2024-06-20 12:08:21 +05:30
IceS2
6b7e9238f7
Update ischema_names for db2 to take BOOLEAN into account ( #16718 )
2024-06-19 14:03:02 +02:00
Imri Paran
c79c5da00a
Fixes #16632 : Add testCases property to a test suite source config ( #16631 )
...
* feat: test-suites
enable specifying a subset of test cases to run in a test suite pipeline
* improved property description
2024-06-19 12:47:38 +02:00
Imri Paran
955ed5afe1
MINOR: fix sql lineage with timezone change ( #16685 )
...
* skip mssql test for python
* fixed mssql lineage
2024-06-18 07:15:52 +00:00
Mayur Singal
57e51df05f
MINOR: Fix superset cypress error ( #16689 )
2024-06-18 11:36:51 +05:30
Mayur Singal
72ed09d6f6
MINOR: Fix column merge error lineage ( #16670 )
...
* MINOR: Fix Column merge error
* append only when col lineage available
* pyformat
2024-06-16 20:32:21 -07:00
Ayush Shah
b3eae8c1b9
Minor: Fix Deprecated utcnow to timezone support ( #16607 )
2024-06-14 15:23:51 +05:30
Mayur Singal
e3fa340c8f
MINOR: Pydantic fixes for redshift & kafka ( #16638 )
2024-06-14 14:08:59 +05:30
Trs
fc9033b953
Fixes(ingestion/source/dbt): Handle None Type in get_tag_labels Function for DBT Metadata Processing ( #16648 )
...
* fix condition
* fix
* lint
2024-06-13 17:19:46 +05:30
Ayush Shah
90d9ef639e
Fix Arg for qliksense ( #16626 )
2024-06-13 09:01:45 +05:30
IceS2
328ed2bf11
MINOR: Fix e2e ( #16627 )
...
* Fix Metabase E2E Test
* Add 'debug' input to python e2e tests
* Fix 'debug' default to be 'false'
* Standardized all Metabase IDs to MetabaseStrId
* Fix Metabase expected filtered sink mix value
* Fix wrong parameter being passed to the config
* Fix powerBI e2e tests
* Fix one Redash e2e test
* Fix checkstyle
* Fix Dashboard create patch_request not using EntityReferenceList
* Fix Redash E2E test value
* Add logging to create patch request
* Fix checkstyle and linting
* Fix default debug value
* Fix e2e workflow
* Fix e2e workflow
* Fix e2e workflow
* Fix metabase and powerbi e2e values
2024-06-12 19:32:45 +02:00
Mayur Singal
b7ef13bc95
MINOR: Fix ssl connection in usage & lineage ( #16625 )
2024-06-12 16:29:24 +05:30
Ayush Shah
2f8de77140
Add Test Suite SSL ( #16619 )
2024-06-12 14:00:24 +05:30
Mayur Singal
7359d6210c
MINOR: Fix Profiler for SSL Enabled Source ( #16613 )
2024-06-12 11:40:30 +05:30
Onkar Ravgan
4f50e0b6a0
cleaned dbt code after pydantic update ( #16605 )
2024-06-12 11:24:29 +05:30
Matt Chamberlin
5236950b6d
Fixes 16595: Parse Iceberg REST table FQN from identifier ( #16596 )
2024-06-11 15:07:57 +05:30
IceS2
070bd2570e
MINOR: Fix e2e ( #16594 )
...
* Fix Metabase E2E Test
* Add 'debug' input to python e2e tests
* Fix 'debug' default to be 'false'
* Standardized all Metabase IDs to MetabaseStrId
* Fix Metabase expected filtered sink mix value
* Fix wrong parameter being passed to the config
* Fix powerBI e2e tests
* Fix one Redash e2e test
* Fix checkstyle
2024-06-11 13:19:25 +05:30
Teddy
ff00175fb1
MINOR - Data Quality Preparatory Work ( #16523 )
...
* chore: added min/max bound and dynamic support flag
* chore: added migration for dynamic attribute
* style: fix pythin checkstyle
2024-06-10 11:03:55 +02:00
Pere Miquel Brull
d2bba3b8e0
FIX - e2e test for pydantic v2 ( #16577 )
...
* FIX - e2e test for pydantic v2
* FIX - e2e test for pydantic v2
* FIX - e2e test for pydantic v2
2024-06-10 08:04:36 +02:00
Pere Miquel Brull
93f976f6e4
FIX - e2e test for pydantic v2 ( #16576 )
...
* FIX - e2e test for pydantic v2
* FIX - e2e test for pydantic v2
* FIX - e2e test for pydantic v2
2024-06-08 10:07:58 +00:00
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 ( #16551 )
...
* Fix - e2e tests for pydantic V2
* add correct default
* add correct default
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* fix apis
* format
2024-06-06 19:36:17 -07:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 ( #16480 )
...
* pydantic v2
* pydanticv2
* fix parser
* fix annotated
* fix model dumping
* mysql ingestion
* clean root models
* clean root models
* bump airflow
* bump airflow
* bump airflow
* optionals
* optionals
* optionals
* jdk
* airflow migrate
* fab provider
* fab provider
* fab provider
* some more fixes
* fixing tests and imports
* model_dump and model_validate
* model_dump and model_validate
* model_dump and model_validate
* union
* pylint
* pylint
* integration tests
* fix CostAnalysisReportData
* integration tests
* tests
* missing defaults
* missing defaults
2024-06-05 21:18:37 +02:00
Onkar Ravgan
e9ec4367ee
Fixed quicksight conn ( #16537 )
2024-06-05 08:54:41 +00:00
Suman Maharana
cdca199ec8
Fix #16370 - Added view/DDL definition rule ( #16486 )
...
* added rule includeDDL
* code optimize
* review changes
2024-06-05 11:11:04 +05:30
Teddy
656da03b14
fix: None type is not iterable ( #16496 )
2024-05-31 10:17:28 +00:00
Mayur Singal
22be0edd79
MINOR: Ignore Cluster Information from columns ( #16495 )
2024-05-31 15:41:12 +05:30
Ayush Shah
a98f6b8dd0
MINOR: Kafka Setup SSL Arg Fix ( #16469 )
2024-05-30 16:03:03 +05:30
Suman Maharana
5ba7769b76
Fix #16425 : Correct Schema Definition for Databricks ( #16441 )
...
* Fix Incorrect Schema Definition for Databricks
* Catalog name is showing correctly
2024-05-30 12:29:12 +05:30
Suman Maharana
0677c4794f
Only Parse view query ( #16470 )
2024-05-30 11:55:33 +05:30
Imri Paran
c8dd9971b4
added usage query to logs in case of error ( #16455 )
2024-05-29 17:14:03 +00:00
Antoine Balliet
0abd3ca5fe
fix: ingestion for dbt > 1.8.0 resource_type is not an enum ( #16415 )
...
* fix: resource_type is not an enum
* feat: add log to display finis
* improve readability
* use getattr to be compatible
* format
2024-05-28 17:00:04 +02:00
Imri Paran
a4c516d2c7
Fixes 16305: Added Test Case for Matching Enum ( #16362 )
...
* Added Test Case for Matching Enum
1. Implemented the test case using the `matchEnum` parameter.
2. Added integration tests.
3. Added migrations.
* fix tests
* fixed tests
* format
* fixed tests
* clear search cache before running ingestion
* format
* changed scopt of aws fixture
* moved migrations to 1.5.0
2024-05-28 09:30:30 +02:00
gpby
d909a3141e
Teradata Connector ( #16373 )
...
* [WIP] add teradata connector
* [WIP] add teradata ingestion
* [WIP] add teradata connector
* [WIP] add teradata connector
* [WIP] add teradata connector
* [WIP] add teradata connector
* [WIP] add teradata connector
* [WIP] add teradata connector
* Reformat code
* Remove unused databaseName property
2024-05-28 06:40:22 +02:00
