1073 Commits

Author SHA1 Message Date
Imri Paran
4c08f82e4e
Fixes 17413: Fix one sided tests for columnValueLengthsToBeBetween and columnValuesToBeBetween (#17423)
* mysql integration tests

* fix(data-quality): accept between with no bounds

add between filters only when the bounds are defined. if they are not (ie: resolve to 'inf' values), do not add any filters

* format

* consolidated ingestion_config

* format

* fixed handling of date and time columns

* fixed tests
2024-08-19 09:09:35 +02:00
Onkar Ravgan
1bc0ca7155
MINOR: Added support to process multiple dbt run_results.json for a single dbt project (#17412)
* Added dbt multiple run_results

* correct to suffix

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-08-13 13:19:56 +05:30
Imri Paran
3069a63cb4
remove pandas import for null_ratio (#17401) 2024-08-12 17:20:11 +02:00
Ayush Shah
af14267e09
Fixes #17319: ArrayDataType issue resolved, Fix Queries + Add DB Name to the queries (#17379)
* fixes arrayDataType must be not null, adding db name to queries as it fails

* Fix Pydantic Issue

* Partial: Add Unity Catalog Topology Test

* Fix lint

* Fix Tests, Fix UnityCatalog Array Column issue

* Fix Tests

* Address comments, add logger to the exception
2024-08-12 09:59:03 +02:00
harshsoni2024
0548342239
Fix #16958: column parser data type fix (#17154) 2024-08-11 00:24:46 +05:30
IceS2
322794ecc2
MINOR: Fix usage datetime format for mssql (#17341)
* Fix usage datetime format for mssql

* Add Integration Test to check that the Usage workflow runs without error

* Fix checkstyle
2024-08-08 16:31:31 +02:00
sam-mccarty-mavenclinic
11a49ef08d
Fixes 16010: Support Looker ingestion from Gitlab (#17284)
* introduce gitlab option to lookml ingestion

* fix reader and disable test

* fix copy paste in test case

* fix file read and keyset pagination for tree

* fix credentials to include gitlab credentials

* uncomment arguments for unused credentials to fix validation error

* fix credentials test

* fix credentials test

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-07 06:53:44 +02:00
Suman Maharana
056affbe0b
Fix mysql e2e test (#17265) 2024-08-02 20:59:33 +05:30
Onkar Ravgan
7272281cf0
fixed tableau owner and e2e counts (#17266) 2024-08-02 15:17:09 +05:30
Mohit Yadav
f07daae196
[RbacDomain] RBAC with Domain Restrictions (#16586)
* Add domain field to entity jsons

* add subdomains ui

* Add Domain To create Request

* Typo fix

* commit progress

* allow multiple domains

* - add userDomains and teamDomains

* - Restrict Api Access

* allow multiple domain support

* - Add Listing Restricition
- Add Search Restrictions

* - Field Fix

* - Inherit Team Domains

* remove glossary filter from ui

* - Fix Tag and User Filter

* - Id -> FQN in create

* - Missing domain on indexes

* add domain in creation of team, user and glossary

* fix merge conflicts

* fix feedbacks

* fix tests

* add domain playwright

* minor changes

* fix domain fetching calls

* Add Thread Listing with EntityDomain

* Fixes for Team Join

* Mkae list mutable

* Add missing import

* Fix Team Test

* User Test case fix

* Fix User Domain

* Minor bug fixes related to sub domains

* Fix data retaining in add sub domain modal after closing

* fix NavBar unit test

* Make Domain Access Work Via Policies

* Fix errors introduced after conflict resolutions

* update render function in SubDomainsTable.component.tsx to accept an array of owners instead of a single owner

* fix java checkstyle

* fix py-test

* format

* Fix the domain e2e owner field type

* Fix Failing Tests

* fixed pytest list index

* update userDomains and teamDomains to domains

* Entity Domain to Domain for thread and changeEvent

* FIx fields

* @Ovveride

* FIx

* Remove Duplicate Field

* Remove Duplicate Field

* Fix Failing Test

* fix domain related changes

* fix subdomains loader

* Fix Owners Order

* Enforce Domain Permissions

* fix domain cleanup

* Fix Test

* rever owner change

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2024-08-02 10:16:14 +05:30
Suman Maharana
71bd67a503
Fix DBT e2e (#17260) 2024-08-01 15:34:17 +05:30
harshsoni2024
407a4496e6
Fix #16434: SQL server date format issue (#17239) 2024-08-01 14:18:45 +05:30
Onkar Ravgan
fe7922c13c
MINOR: [SAP ERP Connector] Added column dtype displaynames and precision/scale values (#17240) 2024-08-01 12:49:34 +05:30
Mayur Singal
840a102887
Fix #17195: Support automated unstructured files ingestion & tags (#17196) 2024-07-31 00:05:58 +05:30
Mayur Singal
ef67474310
Data Insights Refactor (#16532)
---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-07-30 17:38:43 +05:30
Sriharsha Chintalapani
fe107aa3cb
Issue #17012: Multi User/Team Ownership (#17013)
* Add multiple owners

* Multi Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 1

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 2

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 3

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 4

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 5

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 6

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 7

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 8

* Add Migrations for Owner Thread

* update ingestion for multi owner

* fix pytests

* fixed checkstyle

* Add Alert Name to Publishers (#17108)

* Add Alert Name to Publishers

* Fix Test

* Add Bound to Setuptools (#17105)

* Minor: fixed testSummaryGraph issue (#17115)

* feat: updated multi pipeline ui as per new mock (#17106)

* feat: updated multi pipeline ui as per new mock

* translation sync

* fixed failing unit test

* fixed playwright test

* fixed viewService click issue

* sorted pipeline based on test case length

* Added domo federated dataset support (#17061)

* fix usernames (#17122)

* Doc: Updated Doris & Redshift Docs (#17123)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Fix #12677: Added Synapse Connector - docs and side docs (#17041)

* Fix #17098: Fixed case sensitive partition column name in Bigquery (#17104)

* Fixed case sensitive partiion col name bigquery

* update test

* #13876: change placement of comment and close button in task approval workflow (#17044)

* change placment of comment and close button in task approval workflow

* minor change

* playwright test for the close and comment function

* supported ref in activityFeedEditor

* fix playwright test

* added playwright test for data steward

* fix the test for the data streward user

* fix the close button not showing if task has no suggestions and icon fixes

* fix sonar issue

* change glossary and add suggestion button to dropdown button

* fix the glossary failure due to button change

* icon change for add tag and description

* fix glossary cypress failure due to button chnages

* changes as per comments

* MINOR: docs links fix (#17125)

* alation link fix

* dbt yaml config source link fix

* bigquery doc fix

* Explore tree feedbacks (#17078)

* fix explore design

* update switcher icon

* show menu when search query exists

* fix selection of active service

* fix type error

* fix tests

* fix tests

* fix tests

* MINOR: Databricks view TableType fix (#17124)

* Minor: fixed AUT test (#17128)

* Fix #16692: Override Lineage Support for View & Dashboard Lineage (#17064)

* #17065: fix the tags not rendering in selector after selection in edit tags task (#17107)

* fix the tags not rendering in selector after selection in edit tags taks

* added playwright test

* minor changes

* minor fix

* fix the tags not updating in edit and accept tag

* fix explore type changes for collate (#17131)

* MINOR: changed log level to debug (#17126)

* changed log level to debug

* fixed type

* changed type to optional

* Get feed and count data of soft deleted user (#17135)

* Doc: Adding OIDC Docs (#17139)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Doc: Updating Profiler Workflow Docs URL (#17140)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* fix playwright and cypress (#17138)

* Minor: fixed edit modal issue for sql test case (#17132)

* Minor: fixed edit modal issue for sql test case

* fixed test

* Minor: Added whats new content for 1.4.6 release (#17148)

* MINOR [GEN-799]: add option to disable manual trigger using scheduleType (#17031)

* fix: raise for triggering system app

* added scheduleType ScheduledOrManual

* minor: remove "service" field from required properties in createAPIEndpoint schema (#17147)

* initial commit multi ownership

* update glossary and other entities

* update owners

* fix version pages

* fix tests

* Update entity_extension to move owner to array (#17200)

* fix tests

* fix api page errors

* fix owner label design

* locales

* fix owners in elastic search source

* fix types

* fix tests

* fix tests

* Updated CustomMetric owner to entityReferenceList. (#17211)

* Fix owners field in search mappings

* fix search aggregates

* fix inherited label

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 9

* Fix QUeries

* Fix Mysql Queries

* Typo

* fix tests

* fix tests

* fix tests

* fix advanced search constants

* fix service ingestion tests

* fix tests

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: k.nakagaki <141020064+nakaken-churadata@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: harshsoni2024 <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Imri Paran <imri.paran@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
2024-07-30 08:06:39 +02:00
Ayush Shah
34c572334e
Fixes #14945: FiveTran Improvements (#17169) 2024-07-30 10:16:47 +05:30
IceS2
c522f14178
MINOR: Refactor output_handlers to a WorkflowOutputHandler class (#17149)
* Refactor output_handlers to a WorkflowOutputHandler class

* Add old methods as deprecated to avoid breaking changes

* Extract WorkflowInitErrorHandler from workflow_output_handler

* Fix static checks

* Fix tests

* Fix tests

* Update code based on comments from PR

* Update comment
2024-07-29 09:20:34 +02:00
Onkar Ravgan
b6745d7cf1
MINOR: Implemented SAPERP Connector feedback (#17137)
* Implemented saperp feedback

* Fixed pytest
2024-07-29 10:31:20 +05:30
Imri Paran
a681c5184b
tests(unit): use absolute paths (#17193)
use absolute paths when loading files in unit tests
2024-07-26 20:15:34 +05:30
Suman Maharana
80aa9fd80f
Fix: bigquery e2e test fix (#17192) 2024-07-26 16:14:31 +05:30
Imri Paran
cb57941b5f
MINOR: copy data to sql server container instead of using volumes (#17191)
* copy the data files to the containers instead of using volume binds

* format
2024-07-26 10:12:04 +00:00
Suman Maharana
77529d4fa0
Minor: Fix tableau e2e changed filter count (#17180) 2024-07-25 21:01:04 +05:30
Onkar Ravgan
79348a2259
MINOR: Added Alation Sink connector (#16982) 2024-07-24 12:08:07 +05:30
Mayur Singal
5140203311
Fix #16692: Override Lineage Support for View & Dashboard Lineage (#17064) 2024-07-22 20:42:38 +05:30
Onkar Ravgan
b19b7f59a5
Fix #17098: Fixed case sensitive partition column name in Bigquery (#17104)
* Fixed case sensitive partiion col name bigquery

* update test
2024-07-22 12:36:41 +05:30
Ayush Shah
5be5a05390
Fixes #17051: Dynamic import for Profiler Interface (#17073) 2024-07-19 17:33:19 +05:30
Imri Paran
d59b83f9d1
MINOR[GEN-978]: Fix empty test suites (#16975)
* tests: refactor

refactor tests and consolidate common functionality in integrations.conftest

this enables writing tests more concisely.
demonstrated with postgres and mssql.
will migrate more

* format

* removed helpers

* changed scope of fictures

* changed scope of fixtures

* added profiler test for mssql

* fixed import in data_quality test

* json safe serialization

* format

* set MARS_Connection

* fix(data-quality): empty test suite

do not raise for empty test suite

* format

* dont need to check length in _get_test_cases_from_test_suite

* fix

* added warning if no test cases are found
2024-07-19 12:12:34 +02:00
IceS2
d145341026
MINOR: Fix Oracle E2E Tests (#17084)
* Fix Oracle E2E Tests

* Fix Checkstyle

* Add link to issue in the reason
2024-07-19 10:32:22 +02:00
IceS2
aa3d54aede
Fix Vertica E2E Counts (#17071) 2024-07-18 14:54:15 +02:00
harshsoni2024
a117b0224d
MINOR: query log path as dir/file (#17048)
* fetch query log path as dir/file

* add test for multiple files log fetch

* non-csv log file test
2024-07-17 15:23:41 +05:30
Imri Paran
0fee79b200
MINOR: fix sample data issue with Pydantic v2 and refactor python integration tests (#16943)
* tests: refactor

refactor tests and consolidate common functionality in integrations.conftest

this enables writing tests more concisely.
demonstrated with postgres and mssql.
will migrate more

* format

* removed helpers

* changed scope of fictures

* changed scope of fixtures

* added profiler test for mssql

* fixed import in data_quality test

* json safe serialization

* format

* set MARS_Connection

* use SerializableTableData instead of TableData

* deleted file test_postgres.py

* fixed tests

* added more test cases

* format

* changed name test_models.py

* removed the logic for serializing table data

* wip

* changed mapping in common type map

* changed mapping in common type map

* reverted TableData imports

* reverted TableData imports

* reverted TableData imports
2024-07-17 08:11:34 +02:00
Antoine Balliet
e67ba6b14c
feat: BigQuery ingestion allow specify projectId and fix primary key constrains retrieval (#16956) 2024-07-17 11:21:17 +05:30
harshsoni2024
52dc5b551e
MINOR: fix column name in databricks ingestion fail (#17019)
* add backticks in column_name

* add test for column name with .
2024-07-17 10:25:12 +05:30
IceS2
f845663317
Update test_cli_metabase.py (#17026) 2024-07-16 10:05:27 +02:00
Ayush Shah
eeda6d24ae
Fixes #16697: Modify the Query to avoid Numeric Data Overflow (#16920) 2024-07-12 17:23:53 +05:30
Pere Miquel Brull
2aef457785
FIX #16481 - Truncate ingestion pipeline status (#16997)
* FIX #16481 - Truncate ingestion pipeline status

* FIX #16481 - Truncate ingestion pipeline status

* FIX #16481 - Truncate ingestion pipeline status
2024-07-12 09:44:21 +02:00
Imri Paran
98d0e10783
chore: timedelta to string (#16985)
implemented utils.time_utils..timedelta_to_string
2024-07-11 14:23:07 +02:00
Mayur Singal
4eadcfdc5d
Fix #16590: Allow only team groups to be owner (#16995) 2024-07-11 14:17:13 +05:30
Matt Chamberlin
d757aa9d77
Fixes 16652: add GCS storage service (#16917)
* FEAT-16652: add GCS storage service

* reformat

* update connection tests

* fix tests

* relax google-cloud-storage version constraint

* fix GCP config in tests

---------

Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-07-10 14:03:28 +02:00
Ayush Shah
421d191bae
Fixes 16562: Modify HiveCompiler to compile column names properly (#16954)
* Modify HiveCompiler to compile column names properly
2024-07-09 12:59:23 +05:30
Onkar Ravgan
49876b9cd6
fixed e2e for mysql and quicksight (#16959) 2024-07-08 20:50:02 +05:30
Mayur Singal
afafb4af92
MINOR: Add support for s3 unstructured files (#16936) 2024-07-08 15:24:39 +05:30
Imri Paran
d08af1f86d
MINOR: Fix data diff with threshold (#16926)
* fix: table-diff

passed threshold and diff count in wrong order. test was not covering this due to how the parameters were configured.
2024-07-05 07:51:24 +02:00
Pere Miquel Brull
7e98ece3e5
MINOR - Pydantic V2 warnings and better exception msg (#16916) 2024-07-04 14:54:41 +02:00
Onkar Ravgan
80efc7075f
Fix #15163: Added SAP ERP Connector 2024-07-04 10:57:46 +05:30
Imri Paran
9b5ce3560c
MINOR: Pydantic equal assert util (#16918)
* tests: pydantic object assertion

added util for comparing pydantic objects

* fixed test_data_diff.py
2024-07-04 09:59:46 +05:30
Onkar Ravgan
00d74d1776
Fix #15721: Added Override flag to force update of Description, Tags and Owner from Source System (#16815) 2024-07-03 11:48:06 +05:30
harshsoni2024
da37626794
superset lineage fix (#16820)
* fix column list for lineage in db connection

* fix for column yield in datamodel

* superset add test for datamodel of db
2024-07-03 10:55:47 +05:30
Imri Paran
2c9aeebcb8
MINOR: add column diff for table diff test case (#16809)
* feat(table-diff): added column validation

added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report.

* fixed unit tests

* format

* - resolve column types more robustly
- changed test result metric to include "rows" or "columns"
2024-07-02 10:36:03 +00:00