1181 Commits

Author SHA1 Message Date
harshsoni2024
aedbe8be2d
fix pbi, vertica, metabase tests (#20190) 2025-03-11 16:40:56 +01:00
Pere Miquel Brull
2e7c9a0875
FIX #19765 - Improve Column Name Scanner (#20136) 2025-03-07 14:32:59 +01:00
harshsoni2024
40a9c67875
Day 1 - Dashboard service lineage without db_service_name (#19911) 2025-03-07 11:16:58 +05:30
Akash Verma
af9d7c9fa3
Fixes #15696: Added Powerbi filter query (#19912)
* Powerbi filter query

* error handling

* Added test and minor changes

* Checkstyle

* resolved comments

---------

Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
2025-03-06 18:24:57 +05:30
Suman Maharana
137f86c00d
Day1 - Pipeline Usage (#19960) 2025-03-06 18:19:42 +05:30
Suman Maharana
0cd35015d7
#1049 Day 1 Dashboard Usage - Tableau, Looker (#19796) 2025-03-06 18:15:28 +05:30
Pere Miquel Brull
55d7e50441
MINOR - Add and remove data products Actions in Automator (#19948)
* MINOR - Add and remove Data Product assets in Automator config

* MINOR - Add and remove Data Product assets in Automator config

* domain mixin

* build ref

* build ref

* create types

* fix tests

* fix conflicts

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-03-05 07:11:17 +01:00
Sriharsha Chintalapani
799e49e391
Search: improve relevancy for plural/singular words, partial matches,… (#20000)
* Search: improve relevancy for plural/singular words, partial matches, exact matches

* apply to all indexes

* Fix other query patterns

* Revert changes of database and databaseSchema fields in TableIndex.getFields() and table index mapping

* add missing boost query builder in es

* fix ci

* add max_ngram_diff setting in di-assets index

* fix TestCaseResourceTest mvn test failure

---------

Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-02-27 16:47:08 +01:00
Pere Miquel Brull
6b7a9fe76c
MINOR - Update sampler tablenames (#19976)
* MINOR - Update sampler tablenames

* MINOR - Update sampler tablenames

* MINOR - Update sampler tablenames
2025-02-26 14:08:14 +01:00
Imri Paran
97fad806a2
Fixes 19755: Publish app config with status (#19754)
* feat(app): add config to status

add config to the reported status of the ingestion pipeline

* added separate pipeline service client call for external apps

* fix masking of pydantic model

* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py

* fix: execution time

* fix: mask secrets in dump json

* fix: for python3.8

* fix: for python3.8

* fix: use mask_secrets=False when dumping a model for create

* format

* fix: update mask_secrets=False for workflow configurations

* fix: use context directly when using model_dump_json

* fix: default behavior when dumping json

* format

* fixed tests
2025-02-25 16:51:49 +00:00
Sriharsha Chintalapani
a924064c09
Fix #17723: Generate Incremental Change Events even when consolidation of events applied (#19550)
* Fix #17723: Generate Incremental Change Events even when consolidation of events applied

* Fix #17723: Generate Incremental Change Events even when consolidation of events applied

* fix tests

* Fix tests

* clean policy tests

* update search methods to use incrementalChangeDescription part-1

* Fix the version page playwrights

* update search methods to use incrementalChangeDescription part-2

* introduce new field incrementalChangeDescription for search part-3

* fix mvn endpoint test

* fix followers and page search test

* fix following of assets

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2025-02-20 10:23:08 +05:30
Pere Miquel Brull
91b62fdc32
FIX #19798 - Shortening SQA __tablename__ to avoid hitting errors in … (#19809)
* FIX #19798 - Shortening SQA __tablename__ to avoid hitting errors in postgres

* fix tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-02-17 09:37:06 +01:00
Mayur Singal
365fe7daa0
Minor: Fix mysql & snowflake lineage e2e (#19767) 2025-02-13 08:44:01 +05:30
mgorsk1
0e260ce36a
feat: use dataset sql for superset lineage (#19617)
* 🎉 Init

* add fixes around row data type and backport fixes around array type

* refactor

* narrow down column lineage

* narrow down column lineage

* fix insert into

* review remarks + linter

* fix tests

* fix style check

* bring back compatibility with older python versions

* add unit tests

* fix tests

* linting

* expand tests

* make compatible with python 3.8

* checkstyle fix

* fix schema alias

* refactor - move get_dashboard_data_model_column_fqn to superset.utils as it's superset specific and won't work with other dashboard systems integrations
2025-02-12 07:14:31 -08:00
Mayur Singal
45efd878cb
Fix #19634: use system.query.history table for lineage - databricks (#19744) 2025-02-12 13:29:07 +05:30
Pere Miquel Brull
e4fa16e574
FIX - Profiler Source Include Views Filter (#19746)
* FIX - Profiler Source Include Views Filter

* FIX - Profiler Source Include Views Filter
2025-02-12 08:35:47 +01:00
Teddy
e1b3e08317
MINOR BQ sampler type missing (#19696)
* fix: missing entity type in bq sampler

* fix: failing tests
2025-02-11 10:46:34 -08:00
sonika-shah
c0eb7d08de
GEN -19588 Sort Enum type Custom Property Values (#19637)
* GEN -19588 Sort Enum type Custom Property Values

* fix py-tests

* use streams for sorting
2025-02-11 14:29:01 +05:30
Suman Maharana
0d73ee7e36
Fixes #19171 - Looker parse derived view references query (#19687)
* Looker parse derived view references query

* Addressed Comments

* Linting

* Added Tests
2025-02-10 02:50:51 +05:30
harshsoni2024
f4fd9b6855
MINOR: vertica e2e fix (#19708) 2025-02-07 17:33:04 +05:30
Teddy
28bd01c471
MINOR: Remove default 100 when profileSample is None (#19672)
* fix: remove default 100% percent

* fix: use get_dataset

* fix: orm_profiler tests
2025-02-05 19:14:31 +01:00
Ethan
48700ae9ea
Fixes #18075: Dockerfile lint warning (#18077)
* fix docker warning

* for running actions

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2025-02-04 15:28:36 +05:30
Teddy
ef131d7e20
MINOR: Wrong attribute name in SampleConfig model (#19641)
* fix: wrong attribute name in SampleConfig model

* fix: test attribute

* fix: failing tests

* fix: trino filter error + adjust test to take into account null value

* fix: mssql and azuresql tablesample on views
2025-02-04 10:40:40 +01:00
Suman Maharana
3e3c702942
Fix - switch to collate-dbt-artifacts-parser (#19647)
* Switch to collate-dbt-artifacts-parser
2025-02-04 11:57:39 +05:30
Mayur Singal
208c40be09
Fix #19633: Fix databricks schema not found (#19646) 2025-02-04 11:42:11 +05:30
mgorsk1
474f77b46b
feat: use table_id instead of table_name for fetching Superset dataset columns (#19618) 2025-02-03 10:30:42 +05:30
harshsoni2024
69d0e3ce88
powerbi e2e count fix (#19626) 2025-02-03 10:24:59 +05:30
Akash Verma
837e6dd418
Metabase: Ingest orphan charts inside a dashbaord named Default (#19419) 2025-01-28 16:53:19 +05:30
Mayur Singal
d2dc7bd038
Fix #19489: Optimise multithreading for lineage (#19524) 2025-01-27 18:15:58 +05:30
harshsoni2024
b1d481f2f1
issue-16744: salesforce column description with toggle api (#19527) 2025-01-27 16:54:35 +05:30
Imri Paran
41b1ec081d
tests(e2e): increase CI for sampling test (#19519)
based on experiment in https://gist.github.com/sushi30/3083e96c9081371fa55e55b0847b96d2
2025-01-27 09:31:43 +00:00
Imri Paran
e003a5f9ed
tests(e2e): data quality (#19513)
* tests(e2e): data quality

- use the data quality pipeline new schema with a list of service configurations

* format
2025-01-24 14:01:54 +00:00
Imri Paran
5fc20c1ab2
tests: fix sample row count for bigquery (#19511)
- changed method name to expected_sample_count
2025-01-24 10:44:21 +01:00
olof-nn
fd2575d244
ISSUE-19454: Fixes broken looker lineage (#19456)
* ISSUE-19454: Fixes the broken lineage in looker when backticks enclosed table refs

* refactor

* use isort

* Update ingestion/src/metadata/ingestion/source/dashboard/looker/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-01-22 17:56:01 +05:30
Keshav Mohta
7bea4f957f
Feature: Docker Host Retry (#19127) 2025-01-14 19:48:10 +05:30
Akash Verma
072711a563
Added condition for datatype=array without type consistency (#19312) 2025-01-14 14:05:42 +05:30
agriev
dcebc41e3f
Adds percona server for postgresql support (#19322)
* percona server for postgresql support

The only meaningful difference is version string in percona server for postgresql. So commit propose universal and safe way to detect server version by integer string, not complicated parsing of unformatted string.

* updated tests with get_server_version_num

commented outdated tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-13 17:51:40 -08:00
Suman Maharana
89dae42407
Fix mysql e2e count (#19315) 2025-01-10 21:41:38 +05:30
Keshav Mohta
4bd8994a00
Feature: Cassandra SSL (#19226) 2025-01-10 11:25:43 +05:30
Akash Verma
9ecc8a8afe
Added integration testcontainer test for mongodb (#19282) 2025-01-10 10:10:11 +05:30
Akash Verma
4cad5762ad
Added lineage in cli e2e (#19216) 2025-01-09 14:59:49 +05:30
harshsoni2024
63aa484fec
MINOR: powerbi e2e fix (#19240) 2025-01-09 13:49:59 +05:30
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Akash Verma
b2898f7007
Cockroach enhancement (#19108) 2025-01-07 18:51:59 +05:30
Suman Maharana
00be51f299
Fixes #17747: dbt update owners (#19144)
* Fixes 17747: dbt update owners

* update messages

* addressed comments

* py_format

* py_format

* Added tests
2025-01-07 16:37:30 +05:30
Pere Miquel Brull
9dc56c3bb0
TEST - Add bots search RBAC validation (#19159)
* TEST - Add bots search RBAC validation

* format

* format
2025-01-02 15:32:03 +01:00
Akash Verma
39dcb5baef
Feature : Cockroach db connector (#18961) 2025-01-02 13:07:55 +05:30
IceS2
3d83f41b82
Remove the sample_data from assert in profiler since it was moved to the classification workflow (#19153) 2024-12-20 10:48:56 +01:00
IceS2
efbdf21098
Fix e2e tests (#19130)
* Fix E2E

* Fix E2E

* Fix MySQL count

* Tentative Fix

* Tentative Fix

* Revert PowerBI changes

* Revert PowerBI changes
2024-12-19 10:39:30 +01:00
Mayur Singal
a49aab7111
MINOR: User search should only look in name & displayName (#19121)
* MINOR: User search should only look in name & displayname

* py_format

* pyformat

---------

Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
2024-12-18 16:44:54 +05:30