3385 Commits

Author SHA1 Message Date
Onkar Ravgan
d409c339e4
Fix 10689: Added Powerbi Datamodels (#11923) 2023-06-14 17:00:08 +05:30
Onkar Ravgan
81ec95ab27
dbt test value fixed (#11948) 2023-06-13 10:54:32 +05:30
07Himank
62af9bb633
fixed issue for lineage description (#11500)
* fixed issue for lineage description

* fixed issue while ingesting

* fixed issue while ingesting

* added test case for Lingeage with description

* addressing comments .. enhancement

* addressing comments .. enhancement

* modified py test case and removed description from addLineage as we are not using it.

* add support for topic entity and description in lineage details

* fix pylint & test

* pytest fix

* fix column lineage null issue

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-06-12 11:17:32 +05:30
Pere Miquel Brull
095f15064c
Fix #11752 - Multiple inlets in AirflowLineageOperator (#11945)
* Fix multiple inlets in AirflowLineageOperator

* Update dag file
2023-06-12 07:01:19 +02:00
Mayur Singal
05dc42bdb8
Fix #11808: Handle lineage for single db sources in superset (#11933) 2023-06-09 12:43:06 +05:30
Onkar Ravgan
caabe89f9c
Centralize tags ingestion logic (#11880) 2023-06-09 10:45:53 +05:30
Onkar Ravgan
66aa28a3ee
Added check for parent nodes (#11911) 2023-06-08 10:50:11 +05:30
Teddy
8dfa640a82
fix: check for empty description strings (#11914) 2023-06-07 12:49:29 +02:00
Mayur Singal
9e6e00caf9
Fix #9282: Support Databricks Unity Catalog (#11896) 2023-06-07 11:50:31 +05:30
Sriharsha Chintalapani
2b2602b76b
Fix column resize to limit to 128 chars (#11889)
* Fix column resize to limit to 128 chars

* Fix pyformat issue

* replace column_name with col_name

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-07 07:24:08 +02:00
Teddy
b0bfbad9da
fix: disable sys. metrics for snowflake (#11900) 2023-06-06 20:58:12 +00:00
Pere Miquel Brull
b68d063694
Update Test Connection Definition FQN (#11861)
* Update Test Connection Definition FQN

* chore: use object literal for string concatenation

* Fix test

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Fix test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-06 15:22:32 +02:00
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ (#11893)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting

* feat: fixed  update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results (#11881)" (#11892)
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results (#11881)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting
2023-06-06 09:45:49 +02:00
Ayush Shah
236141d9df
Fixes 11825 - Add dataset description on schema level - bigquery (#11878) 2023-06-06 07:28:47 +00:00
Ayush Shah
65f370e4aa
Rename GCS to GCP (#11812) 2023-06-06 11:57:00 +05:30
Mayur Singal
d867a3ab40
remove mac arch from dockerfile (#11876) 2023-06-05 19:10:22 +05:30
Akash Jain
ce9a10dd32
fix: CI Snyk Reports (#11874) 2023-06-05 10:43:52 +05:30
Onkar Ravgan
e21f38c5fd
handled null upstreamcol values (#11835) 2023-06-02 16:55:28 +05:30
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL (#11847)
* feat: Add threshold and strategy logic on the custom SQL object test

* feat: ran python linting

* feat: added safety checks for custom sql query

* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
c98a15ca19
Fixes #11705 - Update ingestion and backend to match new DQ flow (#11836)
* feat: refactor ingestion flow logic

* feat: ran python linting

* feat: update tests to match new workflow

* feat: ran python linting

* feat: update sample data test suite name

* feat: Added backend logic to support logical and executable test suites

* feat: clean up java and json code

* feat: added sample data for logical and executable test suites

* feat: remove executable from CreateTestSuite

* feat: ran python and java linting

* feat: added README info for data quality structure

* skipping cypress to keep main green

* fixed typescript type issue

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-06-01 23:19:13 -07:00
Pere Miquel Brull
fdeea71671
Fix Looker explore git link & Add BitBucket reader (#11837)
* Add looker test connection step

* Add looker test connection step

* Update Credentials

* Fix explore link and add bitbucket reader

* Format

* Fix test

* Fix spline linting

* Fix import
2023-06-02 07:19:32 +02:00
Mayur Singal
b7a1066f5b
Fix #11781: Fix atlas lineage issue (#11832) 2023-06-01 12:02:43 -07:00
Pere Miquel Brull
0a8b28ab7e
Fix redshift e2e with new lineage info (#11843)
* Fix redshift e2e

* Format

* Format
2023-06-01 09:44:27 +02:00
Ayush Shah
8e57b165a6
Fix LineTooLong (#11833)
* Fix LineTooLong

* Add Timeout ( fix lint )
2023-06-01 09:40:18 +05:30
Pere Miquel Brull
3966238703
Remove e2e system metrics test (#11838) 2023-05-31 18:02:17 +02:00
Pere Miquel Brull
11c07ee8ab
Fix #11516 - SAP Hana Connector (#11777)
* SAP Hana skeleton

* Add SAP Hana Connector

* Fix ingestion and docs

* Prep SAP Hana Profiler

* Linting

* Update index.md

* Revert: Update index.md

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-31 16:00:31 +02:00
Mayur Singal
ae386d2e2b
Add Table Sample Data During Sample Data ingestion (#11773) 2023-05-31 12:23:45 +00:00
Mayur Singal
9e437fceec
Fix redshift view lineage parsing issue (#11702)
* Fix redshift view lineage parsing issue

* address review comments
2023-05-31 12:10:39 +00:00
Mayur Singal
b57bbf833f
Fix #11572: Glue Support Partition Columns & Use Pydantic Models (#11776) 2023-05-31 12:03:34 +00:00
Pere Miquel Brull
158250353b
Fix hive e2e tests (#11815)
* Fix hive e2e tests

* Fix hive e2e tests
2023-05-30 17:15:55 +02:00
Pere Miquel Brull
a00fc7fef3
Increase wait time for system metrics e2e test (#11804) 2023-05-30 10:45:14 +02:00
Pere Miquel Brull
7ff00d399e
Install ibm-db-sa if not ARM (#11798) 2023-05-30 10:36:48 +05:30
Pere Miquel Brull
e02fc231da
Improve logging for test connection workflow (#11794) 2023-05-30 10:36:10 +05:30
Chirag Madlani
7adc291364
fix(ui): circular deps for entityReference.json (#11760)
* fix(ui): circular deps for entityReference.json

* Fix circular Dependency python

* Cap Delta Spark version

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-26 18:02:21 +05:30
Pere Miquel Brull
5bd1b09402
Add new Looker test connection steps (#11749) 2023-05-24 18:01:08 +02:00
Mayur Singal
c44bff8fd3
Fix ValueError: too many values to unpack for col lineage (#11723) 2023-05-24 11:30:44 +05:30
Onkar Ravgan
3fbddc2a03
upgrade kafka dep (#11721) 2023-05-23 09:59:12 -07:00
Sriharsha Chintalapani
6509a3670a
Fix #11664: Refactor patch_mixin to use jsonpatch lib (#11696)
* Fix #11664: Refactor patch_mixin to use jsonpatch lib

* Migrate to jsonpatch

* Fix nested cols

* Format

* Update patch_description

* Table constraints

* tag

* owner

* column tag

* column desc

* Format

* Format

* Fix log

* Update dbt patch

* Update column fqn

* Fix test

* Fix tests

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-05-23 15:47:11 +02:00
Onkar Ravgan
7589b7eeef
fixed kafka basicaauth bug (#11729) 2023-05-23 19:05:01 +05:30
Onkar Ravgan
efb37fa7af
Fixed tableau e2e (#11716) 2023-05-23 15:36:08 +05:30
Pere Miquel Brull
1370325762
Fix typing_extension verson (#11719)
* fix extension

* fix extension
2023-05-23 10:09:31 +02:00
Onkar Ravgan
f677408f20
Fixes 11603: Tableau Data Sources as Data Models (#11688)
* added initial changes

* Added lineage details

* fixed pylint

* Fixed name

* Fixed empty lists
2023-05-22 18:07:02 +02:00
Teddy
9f2a02c718
fix: increase sleep delay for system tables (#11704) 2023-05-22 13:43:50 +00:00
Teddy
8c50d1af52
Fixes #4565 - Fetch Metrics from System tables (#11645)
* feat: fetch metrics from system tables

* feat: add permission doc for fetching metrics from system tables

* feat: fix E2E tests to reflect full table row count after table metric update

* feat: ran linting

* feat: fix doc string engine name + function typing

* feat: ran python linting
2023-05-22 09:04:18 +02:00
Teddy
ddbc7fe14d
Fixes #11570 - Add support for BQ Multi-project Profiler (#11692)
* fix: extracted profiler object from workflow and implemented factory to allow service base logic

* fix: ran python linting

* fix: renamed `base` to `base_profiler_source`

* fix: add logic to set correct database for BQ multi project ID connections

* fix: ran python linting
2023-05-20 14:22:53 -07:00
Mayur Singal
8ef98076ae
Fix view lineage not generated for oracle (#11648) 2023-05-19 10:47:55 -07:00
Pere Miquel Brull
0eb2201f94
Restructure NER Scanner internals (#11690)
* Simplify col name scanner

* Restructure NER Scanner internals
2023-05-19 18:21:01 +02:00
Mayur Singal
fafbfdaeab
Fix postgres view lineage parsing issue (#11689) 2023-05-19 17:47:50 +02:00