1118 Commits

Author SHA1 Message Date
mgorsk1
0e260ce36a
feat: use dataset sql for superset lineage (#19617)
* 🎉 Init

* add fixes around row data type and backport fixes around array type

* refactor

* narrow down column lineage

* narrow down column lineage

* fix insert into

* review remarks + linter

* fix tests

* fix style check

* bring back compatibility with older python versions

* add unit tests

* fix tests

* linting

* expand tests

* make compatible with python 3.8

* checkstyle fix

* fix schema alias

* refactor - move get_dashboard_data_model_column_fqn to superset.utils as it's superset specific and won't work with other dashboard systems integrations
2025-02-12 07:14:31 -08:00
Mayur Singal
45efd878cb
Fix #19634: use system.query.history table for lineage - databricks (#19744) 2025-02-12 13:29:07 +05:30
Pere Miquel Brull
e4fa16e574
FIX - Profiler Source Include Views Filter (#19746)
* FIX - Profiler Source Include Views Filter

* FIX - Profiler Source Include Views Filter
2025-02-12 08:35:47 +01:00
Teddy
e1b3e08317
MINOR BQ sampler type missing (#19696)
* fix: missing entity type in bq sampler

* fix: failing tests
2025-02-11 10:46:34 -08:00
sonika-shah
c0eb7d08de
GEN -19588 Sort Enum type Custom Property Values (#19637)
* GEN -19588 Sort Enum type Custom Property Values

* fix py-tests

* use streams for sorting
2025-02-11 14:29:01 +05:30
Suman Maharana
0d73ee7e36
Fixes #19171 - Looker parse derived view references query (#19687)
* Looker parse derived view references query

* Addressed Comments

* Linting

* Added Tests
2025-02-10 02:50:51 +05:30
harshsoni2024
f4fd9b6855
MINOR: vertica e2e fix (#19708) 2025-02-07 17:33:04 +05:30
Teddy
28bd01c471
MINOR: Remove default 100 when profileSample is None (#19672)
* fix: remove default 100% percent

* fix: use get_dataset

* fix: orm_profiler tests
2025-02-05 19:14:31 +01:00
Ethan
48700ae9ea
Fixes #18075: Dockerfile lint warning (#18077)
* fix docker warning

* for running actions

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2025-02-04 15:28:36 +05:30
Teddy
ef131d7e20
MINOR: Wrong attribute name in SampleConfig model (#19641)
* fix: wrong attribute name in SampleConfig model

* fix: test attribute

* fix: failing tests

* fix: trino filter error + adjust test to take into account null value

* fix: mssql and azuresql tablesample on views
2025-02-04 10:40:40 +01:00
Suman Maharana
3e3c702942
Fix - switch to collate-dbt-artifacts-parser (#19647)
* Switch to collate-dbt-artifacts-parser
2025-02-04 11:57:39 +05:30
Mayur Singal
208c40be09
Fix #19633: Fix databricks schema not found (#19646) 2025-02-04 11:42:11 +05:30
mgorsk1
474f77b46b
feat: use table_id instead of table_name for fetching Superset dataset columns (#19618) 2025-02-03 10:30:42 +05:30
harshsoni2024
69d0e3ce88
powerbi e2e count fix (#19626) 2025-02-03 10:24:59 +05:30
Akash Verma
837e6dd418
Metabase: Ingest orphan charts inside a dashbaord named Default (#19419) 2025-01-28 16:53:19 +05:30
Mayur Singal
d2dc7bd038
Fix #19489: Optimise multithreading for lineage (#19524) 2025-01-27 18:15:58 +05:30
harshsoni2024
b1d481f2f1
issue-16744: salesforce column description with toggle api (#19527) 2025-01-27 16:54:35 +05:30
Imri Paran
41b1ec081d
tests(e2e): increase CI for sampling test (#19519)
based on experiment in https://gist.github.com/sushi30/3083e96c9081371fa55e55b0847b96d2
2025-01-27 09:31:43 +00:00
Imri Paran
e003a5f9ed
tests(e2e): data quality (#19513)
* tests(e2e): data quality

- use the data quality pipeline new schema with a list of service configurations

* format
2025-01-24 14:01:54 +00:00
Imri Paran
5fc20c1ab2
tests: fix sample row count for bigquery (#19511)
- changed method name to expected_sample_count
2025-01-24 10:44:21 +01:00
olof-nn
fd2575d244
ISSUE-19454: Fixes broken looker lineage (#19456)
* ISSUE-19454: Fixes the broken lineage in looker when backticks enclosed table refs

* refactor

* use isort

* Update ingestion/src/metadata/ingestion/source/dashboard/looker/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-01-22 17:56:01 +05:30
Keshav Mohta
7bea4f957f
Feature: Docker Host Retry (#19127) 2025-01-14 19:48:10 +05:30
Akash Verma
072711a563
Added condition for datatype=array without type consistency (#19312) 2025-01-14 14:05:42 +05:30
agriev
dcebc41e3f
Adds percona server for postgresql support (#19322)
* percona server for postgresql support

The only meaningful difference is version string in percona server for postgresql. So commit propose universal and safe way to detect server version by integer string, not complicated parsing of unformatted string.

* updated tests with get_server_version_num

commented outdated tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-13 17:51:40 -08:00
Suman Maharana
89dae42407
Fix mysql e2e count (#19315) 2025-01-10 21:41:38 +05:30
Keshav Mohta
4bd8994a00
Feature: Cassandra SSL (#19226) 2025-01-10 11:25:43 +05:30
Akash Verma
9ecc8a8afe
Added integration testcontainer test for mongodb (#19282) 2025-01-10 10:10:11 +05:30
Akash Verma
4cad5762ad
Added lineage in cli e2e (#19216) 2025-01-09 14:59:49 +05:30
harshsoni2024
63aa484fec
MINOR: powerbi e2e fix (#19240) 2025-01-09 13:49:59 +05:30
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Akash Verma
b2898f7007
Cockroach enhancement (#19108) 2025-01-07 18:51:59 +05:30
Suman Maharana
00be51f299
Fixes #17747: dbt update owners (#19144)
* Fixes 17747: dbt update owners

* update messages

* addressed comments

* py_format

* py_format

* Added tests
2025-01-07 16:37:30 +05:30
Pere Miquel Brull
9dc56c3bb0
TEST - Add bots search RBAC validation (#19159)
* TEST - Add bots search RBAC validation

* format

* format
2025-01-02 15:32:03 +01:00
Akash Verma
39dcb5baef
Feature : Cockroach db connector (#18961) 2025-01-02 13:07:55 +05:30
IceS2
3d83f41b82
Remove the sample_data from assert in profiler since it was moved to the classification workflow (#19153) 2024-12-20 10:48:56 +01:00
IceS2
efbdf21098
Fix e2e tests (#19130)
* Fix E2E

* Fix E2E

* Fix MySQL count

* Tentative Fix

* Tentative Fix

* Revert PowerBI changes

* Revert PowerBI changes
2024-12-19 10:39:30 +01:00
Mayur Singal
a49aab7111
MINOR: User search should only look in name & displayName (#19121)
* MINOR: User search should only look in name & displayname

* py_format

* pyformat

---------

Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
2024-12-18 16:44:54 +05:30
Akash Verma
69557e8716
fixes: #15742 Oracle stored package feature (#18852) 2024-12-16 19:35:20 +05:30
Teddy
79b2888bb5
fix: azuresql sampler logic (#19034) 2024-12-13 07:35:04 +01:00
Keshav Mohta
cde3a7dd1e
Feature: Cassandra Connector (#18943) 2024-12-12 15:12:55 +05:30
Imri Paran
16875853a0
test(data-dff): fix flaky test (#18898)
use 99.5 CI for data diff sampling
2024-12-06 18:55:27 +05:30
Imri Paran
e30571cf4e
[GEN-2187] fix(data-diff): added MD5 handling for bigquery (#18904)
* fix(data-diff): added nd5 handling for bigquery

- added MD5 handling for bigquery
- use URL instead of Engine because it requires less steps and less prone to failure

* added e2e test for data diff with sampling in bigquery
2024-12-06 14:21:33 +01:00
Teddy
610322ffed
MINOR - MSSQL timestamp type profiler fix (#18935)
* fix: mssql timestamp processing

* fix: min/max test type on datetime column

* style: fix python format
2024-12-06 08:03:42 +01:00
Teddy
03bd8e9dc4
FEAT: added TABLESAMPLE for MSSQL (#18926)
* feat: added TABLESAMPLE for sqlserver

* fix: class name

* test: added test to generated sample query
2024-12-05 14:17:39 +01:00
Ayush Shah
8664c8df75
Fixes GEN-2199: Allow Fivetran filtering of pipelines using name instead of id (#18929) 2024-12-05 10:55:11 +05:30
Pere Miquel Brull
7aacfe032c
MINOR - FQN encoding in ometa_api, TestSuite pipeline creation & serialization of test case results (#18877)
* DOCS - Update ES config

* MINOR - Add missing FQN encoding & force types

* MINOR - Add missing FQN encoding & force types

* format

* fix tests
2024-12-02 17:17:21 +01:00
Mayur Singal
9b9509f4b9
MINOR: Mysql Lineage Support Main (#18780)
* MINOR: Mysql Lineage Support Main

* fix test

* fix test

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-11-29 20:48:42 +05:30
Teddy
ac2f6d7132
MINOR - Fix sqa table reference (#18839)
* fix: sqa table reference

* style: ran python linting

* fix: added raw dataset to query runner

* fix: get table and schema name from orm object

* fix: get table level config for table tests
2024-11-28 18:49:11 +01:00
mgorsk1
da176767a8
feat: add dbt freshness check test (#18730)
* add dbt freshness check

* docs

* run linting

* add test case param definition

* fix test case param definition

* add config for dbt http, fix linting

* refactor (only create freshness test definition when user executed one)

* fix dbt files class

* fix dbt files class 2

* fix dbt objects class

* fix linting

* fix pylint

* fix linting once and for all

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-11-28 18:30:11 +01:00
harshsoni2024
cb33f274fc
Connector: rename microstrategy connector (#18604) 2024-11-28 18:50:42 +05:30