2535 Commits

Author SHA1 Message Date
Imri Paran
2e9efe5b82
MINOR: fix: use importer as module (#18793)
* fix: use importer as module

to allow monkeypatching in collate

* Update i_validator_builder.py
2024-11-26 16:42:18 +01:00
Imri Paran
3e7808ba78
fixed either annotation (#18778)
* fixed either annotation

* added back Entity

* using `typing_extensions.Annotated` for compatibility with python3.8

* removed bound T for Either
2024-11-26 16:40:32 +01:00
Keshav Mohta
10a7f4ea6c
Fixes #15543: Implemented ElasticSearch Index Template Ingestion (#18686) 2024-11-26 14:42:43 +05:30
Imri Paran
d1b8b9a966
handle import error in param_setter_factory.py (#18745) 2024-11-26 07:21:38 +01:00
Suman Maharana
fe5778361c
Minor: fix dbt ingest failed runs (#18779)
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-26 11:38:31 +05:30
Suman Maharana
f40138a56e
Minor: Fix Powerbi refresh token on expire (#18774)
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-25 23:19:30 +05:30
Imri Paran
d6470b7800
MINOR: fix(data-diff): get added columns (#18694)
* fix(data-diff): get added columns

- use both columns to calculate schema diff

* fix tests
2024-11-25 15:53:50 +01:00
Mayur Singal
4b9e214078
MINOR: Fix StackTraceError validation error - Superset (#18740) 2024-11-25 15:01:59 +05:30
Imri Paran
ee7d043035
[GEN-2109] feat(mongo): added ssl support (#18731)
* feat(mongo): added ssl support

Added SSL support for MongoDB using the SSL manager.

Attached a video demo.

- [Example repository for setting up mongodb with SSL](https://github.com/sushi30/mongodb-docker-ssl-example)
- [MongoDB TLS documentation](https://www.mongodb.com/docs/manual/tutorial/configure-ssl/)

* fixed test_doris.py
2024-11-22 08:54:13 -08:00
Suman Maharana
0a374e3872
Minor: Fix Athena location path (#18750) 2024-11-22 22:05:05 +05:30
Suman Maharana
b220bdb891
Fix: mstr removed dependency issues (#18732)
* Fix: mstr removed dependency issues

* fix session still active error

* py_format

* fix tests

* Addressed Comments

* Addressed Comments

* addressed comments

* Addressed comments

* Add constants

* Fix pytests
2024-11-22 21:19:21 +05:30
Keshav Mohta
10bd051039
Fix #18441: kafka connect password as secret value (#18742) 2024-11-22 19:07:08 +05:30
Ayush Shah
17ffdf9850
fix: modify fqn to allow quotes with dots (#18719) 2024-11-22 09:33:50 +05:30
Pere Miquel Brull
2ea442030c
MINOR - Fix import (#18721) 2024-11-21 13:12:38 +01:00
Imri Paran
0169aad418
[GEN-970] feat(data-quality): support multiple runtime parameter types (#18588)
* feat(data-quality): support multiple runtime parameter types

- changed the runtime parameters setter factory to return sets
- add the runtime parameters based on the name of the runtime of the runtime parameter

**NOTE** requires changes on collate side

* empty set for default case
2024-11-21 08:07:33 +01:00
Pere Miquel Brull
6623ab17dc
MINOR - Fallback for testsuite imports (#18710) 2024-11-20 17:31:48 +01:00
Akash Verma
43bbee9075
fix #18278 "Pipeline ingested from Airflow are now not limited to 100" (#18689) 2024-11-19 18:47:55 +05:30
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow (#18610) 2024-11-19 08:10:45 +01:00
Ayush Shah
6f1df37ba1
Fixes GEN-1260: Add Validators while creating table to escape special characters (#18456) 2024-11-18 15:02:57 +05:30
Sriharsha Chintalapani
88c8fb48f3
Add Edit glossary terms, Edit Tier , Edit Tags as separate permissions (#18331)
* Add EditGlossaryTerms Permission

* Fix #18330: Add EDIT_GLOSSARY_TERM permission and enforce EDIT_TIER permisson

* add edit glossary term permission check in UI

* revert EDIT_GLOSSARY_TERMS operation

* Add EDIT_GLOSSARY_TERMS to common operations

* Add EDIT_TIER to common operations

* add default empty array for tags field, as patch calls can run into issues

* Fix tests

* Fix tests

* added glossary terms

* fix conflicts

* fix permission check for data model

* Add EditGlossaryTerms to DataConsumerPolicy

* Add EditGlossaryTerms,EditTier to DataConsumerPolicy

* fix tests

* Fix migrations for EditTier,EditGlossaryTerms

* add edit tier permission to data consumer

* Fix tests

* fix pytests

* missing test_dbt.py

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 10:50:15 -08:00
mgorsk1
3d2dfeb583
feat: use native trino client authentication classes (#16196)
---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 12:54:42 +05:30
Imri Paran
bde6ee4125
MINOR: Data diff sample fix (#18632)
* fix(data-diff): sampling configuration

handle the sampling condition separately for the 2 tables allowing to apply sampling on columns with mismatching cases

* format
2024-11-15 08:22:13 +01:00
Mayur Singal
75d417d267
MINOR: Fix user search - exclude bots (#18645) 2024-11-14 21:25:21 +05:30
Akash Verma
fa30be0589
fix #17726 Databricks schema name with hyphen issue (#18598) 2024-11-14 20:02:20 +05:30
Keshav Mohta
968f892b48
Fix #16110 - Add support to ingest tags airflow (#18626) 2024-11-14 17:09:00 +05:30
Ayush Shah
6fa03ee66a
Fixes GEN-1994: Remove View Lineage from Metadata Ingestion flow (#18558) 2024-11-13 00:08:55 +05:30
Teddy
45d27a377d
GEN 1184 - Added Workflow Classification and Metric LevelConfig (#18572) 2024-11-11 15:59:42 +01:00
Imri Paran
a6d97b67a8
MINOR: fix system profile return types (#18470)
* fix(redshift-system): redshift return type

* fixed bigquery profiler

* fixed snowflake profiler

* job id action does not support matrix. using plain action summary.

* reverted gha change
2024-11-11 10:49:42 +01:00
Suman Maharana
fc79d60d83
Fixes: Added Sigma Column Level Lineage and Datamodels (#18571) 2024-11-11 14:42:57 +05:30
Imri Paran
cdaa5c10af
[GEN-1996] feat(data-quality): use sampling config in data diff (#18532)
* feat(data-quality): use sampling config in data diff

- get the table profiling config
- use hashing to sample deterministically the same ids from each table
- use dirty-equals to assert results of stochastic processes

* - reverted missing md5
- added missing database service type

* - use a custom substr sql function

* fixed nounce

* added failure for mssql with sampling because it requires a larger change in the data-diff library

* fixed unit tests

* updated range for sampling
2024-11-11 10:07:23 +01:00
Mayur Singal
efed932d97
Mask SQL Queries in Usage & Lineage Workflow (#18565) 2024-11-11 11:44:47 +05:30
Mayur Singal
b02c64931e
MINOR: Fix table not found error (#18560) 2024-11-09 20:33:32 +05:30
Suman Maharana
da039b197f
Add: Azure Data factory Connector (#18543)
* Added Azure Data factory Connector

* Added Lineage data factory

* removed not required files

* removed not required files

* Removed datafactory ui changes from oss

* resolve merge conflicts

* resolve merge conflicts

* added python requirements
2024-11-08 07:38:45 +01:00
Imri Paran
b92b950060
Fix 18434: feat(statistics-profiler): use statistics tables to profile trino tables (#18433)
* feat(statistics-profiler): use statistics tables to profile trino tables

- implemented the collaborative root class
- added the "useStatistics" profiler parameter
- added the "supportsStatistics" database connection property
- implemented the ProfilerWithStatistics and StoredStatisticsSource to add this functionality to specific profilers
- implemented TrinoStoredStatisticsSource for specific trino statistics logic

* added ABC to terminal classes in collaborative root

* fixed docstring for TestSuiteInterface

* reverted unintended changes

* typo
2024-11-07 18:37:31 +01:00
Imri Paran
729a06b5f0
fix: use enum.Enum instead of sqlalchemy enum (#18464) 2024-11-07 11:42:03 +01:00
Mayur Singal
8d40d8ea77
MINOR: Fix Materialized View Lineage (#18539) 2024-11-07 09:21:54 +01:00
Teddy
d579008c99
GEN 1683 - Add Column Value to be At Expected Location Test (#18524)
* feat: added column value to be in expected location test

* fix: renamed value -> values

* doc: added 1.6 documentatio entry

* style: ran python linting

* fix: move data packaging to pyproject.yaml

* fix: add init file back for data package

* fix: failing test case
2024-11-06 11:17:13 +01:00
Mayur Singal
5660a751e3
GEN-2000: Add Support for PowerBI Report Server (#18513) 2024-11-06 14:55:05 +05:30
Suman Maharana
426ad2000b
Fix #17778 : Databricks query run optimisation (#18467)
* Fix : Databricks query run  optimization

* Fixed dialect error

* fix get columns

* py format

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-06 10:10:01 +05:30
IceS2
dccba20101
Return s3 endpoint as str() instead of Url (#18521) 2024-11-05 17:39:50 +00:00
Katarzyna Kałek
47c75fe6a7
Enhanced Glue ingestion with external table features (#18511)
* added fileFormat, locationPath and external table lineage to Glue ingestion

* Improve Lineage Label

---------

Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-05 21:48:20 +05:30
Teddy
9a685d5f19
fix: pass row and result computation for inSet test (#18466) 2024-10-31 08:15:18 +00:00
Nicola Coretti
7ebc62dca7
feat: Add support for exasol datasource (#17166)
* Add flake.nix

* Add lockfile for flake

* Update nix environment and document usage

* Add schema for exasol connector

* Add Exasol definitions to databaseService

* Fix error in exasol connector schema

* Add additional connection options/settings to exasol connector

* Add exasol-connector to ui

* Add depdencies for exasol-connector

* Update notes

* Update ingestion code

* Add Basic Documentation for Exasol Connector

* Update flake file

* Add developer notes

* Add python script which can be used as entry point for debugging in ide

* Add config file which can be used for debugging (manual execution)

* Update debug script

* Update developer notes

* Remove old developer notes

* Add .venv to gitignore

* Update dev notes

* Update development notes

* Update ExasolSource

* Establish basic connection to Exasol DB from connector

* Update exasol connector connection settings

* Add service_spec for exasol plugin

* Remove development files

* Remove unused module

* Applied code formatter

* Update exasol dependency constraint(s)

* Add unit test for exasol connection url(s)

* Fixed test expectations for exasol connection url test(s)

* Adjust the test query for the Exasol connection test
2024-10-31 08:11:30 +01:00
Imri Paran
016a840b2f
MINOR Fix snowflake profiler by using case-insensitive strings (#18438)
* use snowflake system metrics computer instead of source

* reverted pylint

* use case-insensitive strings equality for snowflake filters
2024-10-29 18:33:36 +01:00
Suman Maharana
67a9e63439
Minor: Fixed dbtcloud test connection and improved docs (#18408) 2024-10-29 14:39:52 +05:30
Onkar Ravgan
4a0c8406e9
[ER Diagrams] Add ER diagram APIs and sample data (#18021)
* Add ER diag APIs and sample data

* fix pylint

* formatting fixes2

* fixed es client return

* fixed os client return

* supported TableDetailPage tabs as classBase for supporting collate only tabs

* Added schema Apis

* change the base class to .ts and move the component in the util files

* beautify function arguments

* Added optimizations

* Ingestion changes

* svg dimension change

* supported class base tab in databaseSchema

* supported classBase action button in schema table name column

* added further keys data for constraint modal

* fix sonar issue

* remove old method to override edit action on column and shifted to DisplayNameModal for fields

* supported table right panel component to further extends on collate side

* minor fix around duplicate constraint

* added support to update table constraints and column constraints in the UI

* code optimization and minor fixes

* review comments and multi col fix

* added queryFilter option in NodeSuggestion and tableConstrainst to fetch and use only in service tables

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-10-28 20:26:19 +05:30
Mayur Singal
9d91325af8
Lineage-1: Move view lineage processing to lineage workflow (#18220)
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-10-28 18:18:22 +05:30
Nicola Coretti
24de281026
Fix docstring in the Doris metadata module (#18421) 2024-10-28 17:58:04 +05:30
Mayur Singal
4083838056
MINOR: Couchbase Secondary Index Fix (#18398) 2024-10-24 20:34:55 +05:30
harshsoni2024
1a8bba6058
GEN-1911: Quicksight lineage source fix (#18348) 2024-10-24 11:41:37 +05:30