3502 Commits

Author SHA1 Message Date
Teddy
354f879866
fix: db_service not passed to metric instantiating (#17504) 2024-08-20 07:57:03 -07:00
Teddy
4ed2035a50
fix: remove root when accessing fqn of entity reference (#17491) 2024-08-20 15:17:16 +02:00
Ayush Shah
9880f06b2c
Fixes #17489: Allow non numeric numbers to be sent via Json, Replace NaN value… (#17490)
* fix: Allow non numeric numbers to be sent via Json, Replace NaN values with None in SQAProfilerInterface

Replace NaN values with None in the SQAProfilerInterface class to maintain database parity. NaN values will be cast to null in OpenMetadata. This change ensures that data handling processes account for this conversion.

* fix: histogram overflow error

* test: Add Unit Test for Null and Null Ratio Metric

* chore: Address comments

* chore: Address comments

* fix: checkstyle and message

* fix: failing tests as null count works as expected
2024-08-20 16:33:55 +05:30
Imri Paran
a59eb2a3cd
fix: pin numpy version (#17487) 2024-08-20 10:19:05 +00:00
Imri Paran
5da7bb049c
MINOR: fix table profiler on empty tables in trino (#17471)
* fix(profiler): trino

coalesce row count to 0 if result is null. this value gets returned for empty tables

* fixed test_metadata.py
2024-08-20 08:42:10 +00:00
IceS2
48b43900b6
Install db2 dependency on amd64 architectures (#17495) 2024-08-20 09:24:38 +02:00
Imri Paran
2722eadc33
fix: gcs (#17486)
1. update docs using gcp credentials in path.
2. updated example `clientIt` in docs
3. fixed client to work with implicit project
4. fixed workflow to warn about missing buckets
2024-08-19 23:14:46 -07:00
Imri Paran
31c2ec8c57
MINOR: fix qlikcloud test connection (#17459)
* fix: qlikcloud test connection

* patch test_connection for qlik cloud unit tests
2024-08-19 23:14:09 -07:00
IceS2
ddd8c41864
Fix DB2 Schema Trailing Whitespaces (#17475) 2024-08-19 23:13:14 -07:00
Imri Paran
7508848376
fix(dq): data types for unique columns (#17431)
1. remove json and array from supported data types of unique column test.
2. migrations.
3. tests.
2024-08-19 14:28:42 +02:00
Mayur Singal
8acf6d3e94
MINOR: Make Include ddl disabled by default (#17450)
* MINOR: Make Include ddl disabled by default

* make schema def failure warning debug

* Add missing condition

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
2024-08-19 14:07:18 +02:00
Pere Miquel Brull
b175e40e99
MINOR - Clean DEBUG logs (#17464)
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
2024-08-19 12:20:29 +02:00
Imri Paran
4c08f82e4e
Fixes 17413: Fix one sided tests for columnValueLengthsToBeBetween and columnValuesToBeBetween (#17423)
* mysql integration tests

* fix(data-quality): accept between with no bounds

add between filters only when the bounds are defined. if they are not (ie: resolve to 'inf' values), do not add any filters

* format

* consolidated ingestion_config

* format

* fixed handling of date and time columns

* fixed tests
2024-08-19 09:09:35 +02:00
Onkar Ravgan
bbb3256c0d
Match correct file names for the dbt artifacts (#17445) 2024-08-18 11:40:37 +02:00
Suman Maharana
de3a82eeb6
Minor: Kill active/idle connections after test connections (#17411)
* Minor: Kill active/idle connections after test connections

* fixed idle conn for multi db

* added exception handling
2024-08-14 15:42:42 +02:00
Ayush Shah
8ad6c95fe4
Fixes #17367: PipelineStatus Timestamp None not allowed (#17422)
* fix(ingestion): Change Timestamp None to Current Time noting pending pipeline

* fix(ingestion): Address comments around PipelineStatus timestamp

* fix(ingestion): Improve timestamps handling for tasks and pipeline status
2024-08-13 15:39:29 +02:00
Suman Maharana
feab12422b
MINOR: Fix Datetime Conversion issue in usage/lineage (#17380)
* MINOR: Fix Datetime Conversion issue in usage/lineage

* Undo mssql specific fixes

* fixed datetime conversion in mssql

* fixed datetime conversion in oracle
2024-08-13 14:04:50 +02:00
IceS2
5e32c2aa78
Install DB2 odbc driver on x86_64 architectures and update docs (#17425) 2024-08-13 13:16:19 +02:00
Onkar Ravgan
1bc0ca7155
MINOR: Added support to process multiple dbt run_results.json for a single dbt project (#17412)
* Added dbt multiple run_results

* correct to suffix

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-08-13 13:19:56 +05:30
harshsoni2024
a67a3241e7
sql query correct wildcard syntax (#17405) 2024-08-13 11:39:05 +05:30
Suman Maharana
1a2fd24c10
Fixes: DBT Cloud lineage not showing (#17395) 2024-08-12 23:48:40 +05:30
Imri Paran
3069a63cb4
remove pandas import for null_ratio (#17401) 2024-08-12 17:20:11 +02:00
Antoine Balliet
34dc79b5fe
feat: DBT - allow to use email to match users as fallback with ES search (#17349)
* feat: allow to use email to match dbt model owner

* improve description

* revert add option to parse from email

* feat: implement email matching in case name was not found

* feat: use email to find user as fallback at connector level
2024-08-12 14:54:54 +02:00
harshsoni2024
c84b77859e
MINOR: databricks describe table query update (#17396)
* databricks describe table query update

* remove extra args

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2024-08-12 17:35:04 +05:30
Mayur Singal
6a31c579f6
MINOR: Add Unity Catalog Lineage Dialect (#17398) 2024-08-12 17:01:37 +05:30
Ayush Shah
af14267e09
Fixes #17319: ArrayDataType issue resolved, Fix Queries + Add DB Name to the queries (#17379)
* fixes arrayDataType must be not null, adding db name to queries as it fails

* Fix Pydantic Issue

* Partial: Add Unity Catalog Topology Test

* Fix lint

* Fix Tests, Fix UnityCatalog Array Column issue

* Fix Tests

* Address comments, add logger to the exception
2024-08-12 09:59:03 +02:00
Ayush Shah
83e2b68a25
Fixes #17377: HIERARCHYID & GEOGRAPHY pyodbc error (#17378)
* fixes: HIERARCHYID & GEOGRAPHY pyodbc error

- Add support for "HIERARCHYID" column type in column_type_parser.py
- Register AzureSQLSampler for "AzureSQLConnection" in sampler_factory.py

* Revert SQA_UTILS
2024-08-11 17:29:43 +02:00
harshsoni2024
0548342239
Fix #16958: column parser data type fix (#17154) 2024-08-11 00:24:46 +05:30
Pere Miquel Brull
a098c20c7c
MINOR - LKML sample data (#17359) 2024-08-10 18:01:00 +02:00
harshsoni2024
264be13b66
databricks foreign table issue: skip table metadata for foreign tables (#17343)
* skip get_columns for foreign catalog tables

* get table type before executing column metadata

* remove duplicate query, pycheckstyle fix

* skip fk table instead of reaching till column metadata

* add debug log
2024-08-10 21:08:49 +05:30
Onkar Ravgan
f33bfe78d6
fixed dbt cloud conn (#17376) 2024-08-10 16:58:55 +05:30
IceS2
e5a7cff5a5
Updated oracle to use DBA_ tables (#17274)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2024-08-10 16:58:21 +05:30
Yung-Chun
453c20f53e
fix type hint (#17354) 2024-08-10 10:48:42 +05:30
harshsoni2024
1b04f1fb37
Fix #16573: get table owners for databaricks & unitycatalog tables (#17282) 2024-08-10 10:45:22 +05:30
IceS2
e52c4af9ee
Fix Databricks TableQuery timestamps to str (#17362) 2024-08-09 17:32:32 +02:00
IceS2
322794ecc2
MINOR: Fix usage datetime format for mssql (#17341)
* Fix usage datetime format for mssql

* Add Integration Test to check that the Usage workflow runs without error

* Fix checkstyle
2024-08-08 16:31:31 +02:00
IceS2
67634f9695
Add a check to see if file_client is present (#17330) 2024-08-07 17:34:41 +02:00
Imri Paran
62c33e026c
MINOR: handle "ignored exception" error message when running table diff (#17322)
* fix: table diff

implemented a safe iterator to handle the sneaky `KeyError`

* changed method to safe_table_diff_iterator

* format

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-08-07 17:21:17 +02:00
sam-mccarty-mavenclinic
11a49ef08d
Fixes 16010: Support Looker ingestion from Gitlab (#17284)
* introduce gitlab option to lookml ingestion

* fix reader and disable test

* fix copy paste in test case

* fix file read and keyset pagination for tree

* fix credentials to include gitlab credentials

* uncomment arguments for unused credentials to fix validation error

* fix credentials test

* fix credentials test

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-07 06:53:44 +02:00
Pere Miquel Brull
4cddee89ad
FIX - Ingestion Airflow Image constraints (#17296)
* FIX - Ingestion constraints

* FIX - Ingestion constraints

* FIX - Ingestion constraints
2024-08-06 06:44:27 +02:00
Suman Maharana
056affbe0b
Fix mysql e2e test (#17265) 2024-08-02 20:59:33 +05:30
Teddy
1349f165b3
[GEN-1069] - Fix dbt issues in ingestion (#17247)
* fix: dbt errors when creating test case

* fix: use fqn builder
2024-08-02 12:29:48 +02:00
Onkar Ravgan
7272281cf0
fixed tableau owner and e2e counts (#17266) 2024-08-02 15:17:09 +05:30
Teddy
2a854f90e1
fix: DBX profiler struct support + nested colum retrieval for profiler (#17267) 2024-08-02 11:34:10 +02:00
Mohit Yadav
f07daae196
[RbacDomain] RBAC with Domain Restrictions (#16586)
* Add domain field to entity jsons

* add subdomains ui

* Add Domain To create Request

* Typo fix

* commit progress

* allow multiple domains

* - add userDomains and teamDomains

* - Restrict Api Access

* allow multiple domain support

* - Add Listing Restricition
- Add Search Restrictions

* - Field Fix

* - Inherit Team Domains

* remove glossary filter from ui

* - Fix Tag and User Filter

* - Id -> FQN in create

* - Missing domain on indexes

* add domain in creation of team, user and glossary

* fix merge conflicts

* fix feedbacks

* fix tests

* add domain playwright

* minor changes

* fix domain fetching calls

* Add Thread Listing with EntityDomain

* Fixes for Team Join

* Mkae list mutable

* Add missing import

* Fix Team Test

* User Test case fix

* Fix User Domain

* Minor bug fixes related to sub domains

* Fix data retaining in add sub domain modal after closing

* fix NavBar unit test

* Make Domain Access Work Via Policies

* Fix errors introduced after conflict resolutions

* update render function in SubDomainsTable.component.tsx to accept an array of owners instead of a single owner

* fix java checkstyle

* fix py-test

* format

* Fix the domain e2e owner field type

* Fix Failing Tests

* fixed pytest list index

* update userDomains and teamDomains to domains

* Entity Domain to Domain for thread and changeEvent

* FIx fields

* @Ovveride

* FIx

* Remove Duplicate Field

* Remove Duplicate Field

* Fix Failing Test

* fix domain related changes

* fix subdomains loader

* Fix Owners Order

* Enforce Domain Permissions

* fix domain cleanup

* Fix Test

* rever owner change

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2024-08-02 10:16:14 +05:30
Mayur Singal
27af481433
MINOR: Fix lineage PATCH API (#17264) 2024-08-01 17:28:22 +05:30
Suman Maharana
71bd67a503
Fix DBT e2e (#17260) 2024-08-01 15:34:17 +05:30
harshsoni2024
407a4496e6
Fix #16434: SQL server date format issue (#17239) 2024-08-01 14:18:45 +05:30
Onkar Ravgan
fe7922c13c
MINOR: [SAP ERP Connector] Added column dtype displaynames and precision/scale values (#17240) 2024-08-01 12:49:34 +05:30
Shin-ichi Hashiba
bb6800d2ec
add ingest postgres schema description (#16946) 2024-07-31 13:15:49 +05:30