1145 Commits

Author SHA1 Message Date
Mayur Singal
e6b7b89f86
Fix #20236: Handle Sample Data with non-utf8 characters (#20380) 2025-03-27 14:20:26 +05:30
Ayush Shah
7a3990f350
Fixes 19119: Enhance TableCustomSQLQueryValidator to support threshold operation (#20307) 2025-03-27 13:11:56 +05:30
Ayush Shah
653c878497
MINOR: Transform Reserved keywords like quotes to OM compatible (#20459) 2025-03-27 13:02:07 +05:30
Ayush Shah
60974e4ea1
Revert "Fixes #17660: Oracle handle quotes for lowercase columns in workflow agents (#20309)" (#20364) 2025-03-20 21:02:58 +05:30
Mayur Singal
fb3ba391ff
MINOR: Fix failing pytest (#20332) 2025-03-19 12:35:37 +05:30
Sriharsha Chintalapani
706cebd97a
Opensearch connector (#19698)
* Fix #19667: OpenSearch Connector

* Fix #19667: OpenSearch Connector

* do not ingest any system level indexes

* fix pyformat

* Add AWS auth

* Use common schema and fix ssl config in client

* Add openseach connector docs and update schema

* Remove api key auth type and complete docs checklist

* Remove unnecessary httpx dependency and pyformat

* Add compatible version of httpx for elasticsearch

* Fix pylint fails and py-tests validation error

---------

Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
2025-03-18 18:45:25 +05:30
Ayush Shah
20ab64d1f1
Fixes #17660: Oracle handle quotes for lowercase columns in workflow agents (#20309) 2025-03-18 15:48:58 +05:30
fuzmish
7fa3e53403
Fix: Pass raw value of extraHeaders to ClientConfig (#19989) 2025-03-18 13:55:51 +05:30
harshsoni2024
dba37820d7
MINOR: e2e fixes (#20301) 2025-03-17 21:00:26 +05:30
Akash Verma
cf7a442e32
Fixes #19891 : Added measures in powerbi (#19990) 2025-03-17 14:43:22 +05:30
Mayur Singal
d30fd90096
Minor: Query Cost Table Aggregation Endpoint (#20270) 2025-03-17 11:33:50 +05:30
Mayur Singal
581ab6ce71
MINOR: Fix pytests - usage count (#20247) 2025-03-14 09:07:40 +01:00
harshsoni2024
9bf1ce53ec
MINOR: fix-e2e-tests (#20233) 2025-03-13 20:32:06 +05:30
harshsoni2024
826279608f
issue-19892: parse powerbi table source (#20141) 2025-03-12 12:59:29 +05:30
harshsoni2024
aedbe8be2d
fix pbi, vertica, metabase tests (#20190) 2025-03-11 16:40:56 +01:00
Pere Miquel Brull
2e7c9a0875
FIX #19765 - Improve Column Name Scanner (#20136) 2025-03-07 14:32:59 +01:00
harshsoni2024
40a9c67875
Day 1 - Dashboard service lineage without db_service_name (#19911) 2025-03-07 11:16:58 +05:30
Akash Verma
af9d7c9fa3
Fixes #15696: Added Powerbi filter query (#19912)
* Powerbi filter query

* error handling

* Added test and minor changes

* Checkstyle

* resolved comments

---------

Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
2025-03-06 18:24:57 +05:30
Suman Maharana
137f86c00d
Day1 - Pipeline Usage (#19960) 2025-03-06 18:19:42 +05:30
Suman Maharana
0cd35015d7
#1049 Day 1 Dashboard Usage - Tableau, Looker (#19796) 2025-03-06 18:15:28 +05:30
Pere Miquel Brull
55d7e50441
MINOR - Add and remove data products Actions in Automator (#19948)
* MINOR - Add and remove Data Product assets in Automator config

* MINOR - Add and remove Data Product assets in Automator config

* domain mixin

* build ref

* build ref

* create types

* fix tests

* fix conflicts

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-03-05 07:11:17 +01:00
Sriharsha Chintalapani
799e49e391
Search: improve relevancy for plural/singular words, partial matches,… (#20000)
* Search: improve relevancy for plural/singular words, partial matches, exact matches

* apply to all indexes

* Fix other query patterns

* Revert changes of database and databaseSchema fields in TableIndex.getFields() and table index mapping

* add missing boost query builder in es

* fix ci

* add max_ngram_diff setting in di-assets index

* fix TestCaseResourceTest mvn test failure

---------

Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-02-27 16:47:08 +01:00
Pere Miquel Brull
6b7a9fe76c
MINOR - Update sampler tablenames (#19976)
* MINOR - Update sampler tablenames

* MINOR - Update sampler tablenames

* MINOR - Update sampler tablenames
2025-02-26 14:08:14 +01:00
Imri Paran
97fad806a2
Fixes 19755: Publish app config with status (#19754)
* feat(app): add config to status

add config to the reported status of the ingestion pipeline

* added separate pipeline service client call for external apps

* fix masking of pydantic model

* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py

* fix: execution time

* fix: mask secrets in dump json

* fix: for python3.8

* fix: for python3.8

* fix: use mask_secrets=False when dumping a model for create

* format

* fix: update mask_secrets=False for workflow configurations

* fix: use context directly when using model_dump_json

* fix: default behavior when dumping json

* format

* fixed tests
2025-02-25 16:51:49 +00:00
Sriharsha Chintalapani
a924064c09
Fix #17723: Generate Incremental Change Events even when consolidation of events applied (#19550)
* Fix #17723: Generate Incremental Change Events even when consolidation of events applied

* Fix #17723: Generate Incremental Change Events even when consolidation of events applied

* fix tests

* Fix tests

* clean policy tests

* update search methods to use incrementalChangeDescription part-1

* Fix the version page playwrights

* update search methods to use incrementalChangeDescription part-2

* introduce new field incrementalChangeDescription for search part-3

* fix mvn endpoint test

* fix followers and page search test

* fix following of assets

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2025-02-20 10:23:08 +05:30
Pere Miquel Brull
91b62fdc32
FIX #19798 - Shortening SQA __tablename__ to avoid hitting errors in … (#19809)
* FIX #19798 - Shortening SQA __tablename__ to avoid hitting errors in postgres

* fix tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-02-17 09:37:06 +01:00
Mayur Singal
365fe7daa0
Minor: Fix mysql & snowflake lineage e2e (#19767) 2025-02-13 08:44:01 +05:30
mgorsk1
0e260ce36a
feat: use dataset sql for superset lineage (#19617)
* 🎉 Init

* add fixes around row data type and backport fixes around array type

* refactor

* narrow down column lineage

* narrow down column lineage

* fix insert into

* review remarks + linter

* fix tests

* fix style check

* bring back compatibility with older python versions

* add unit tests

* fix tests

* linting

* expand tests

* make compatible with python 3.8

* checkstyle fix

* fix schema alias

* refactor - move get_dashboard_data_model_column_fqn to superset.utils as it's superset specific and won't work with other dashboard systems integrations
2025-02-12 07:14:31 -08:00
Mayur Singal
45efd878cb
Fix #19634: use system.query.history table for lineage - databricks (#19744) 2025-02-12 13:29:07 +05:30
Pere Miquel Brull
e4fa16e574
FIX - Profiler Source Include Views Filter (#19746)
* FIX - Profiler Source Include Views Filter

* FIX - Profiler Source Include Views Filter
2025-02-12 08:35:47 +01:00
Teddy
e1b3e08317
MINOR BQ sampler type missing (#19696)
* fix: missing entity type in bq sampler

* fix: failing tests
2025-02-11 10:46:34 -08:00
sonika-shah
c0eb7d08de
GEN -19588 Sort Enum type Custom Property Values (#19637)
* GEN -19588 Sort Enum type Custom Property Values

* fix py-tests

* use streams for sorting
2025-02-11 14:29:01 +05:30
Suman Maharana
0d73ee7e36
Fixes #19171 - Looker parse derived view references query (#19687)
* Looker parse derived view references query

* Addressed Comments

* Linting

* Added Tests
2025-02-10 02:50:51 +05:30
harshsoni2024
f4fd9b6855
MINOR: vertica e2e fix (#19708) 2025-02-07 17:33:04 +05:30
Teddy
28bd01c471
MINOR: Remove default 100 when profileSample is None (#19672)
* fix: remove default 100% percent

* fix: use get_dataset

* fix: orm_profiler tests
2025-02-05 19:14:31 +01:00
Ethan
48700ae9ea
Fixes #18075: Dockerfile lint warning (#18077)
* fix docker warning

* for running actions

---------

Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
2025-02-04 15:28:36 +05:30
Teddy
ef131d7e20
MINOR: Wrong attribute name in SampleConfig model (#19641)
* fix: wrong attribute name in SampleConfig model

* fix: test attribute

* fix: failing tests

* fix: trino filter error + adjust test to take into account null value

* fix: mssql and azuresql tablesample on views
2025-02-04 10:40:40 +01:00
Suman Maharana
3e3c702942
Fix - switch to collate-dbt-artifacts-parser (#19647)
* Switch to collate-dbt-artifacts-parser
2025-02-04 11:57:39 +05:30
Mayur Singal
208c40be09
Fix #19633: Fix databricks schema not found (#19646) 2025-02-04 11:42:11 +05:30
mgorsk1
474f77b46b
feat: use table_id instead of table_name for fetching Superset dataset columns (#19618) 2025-02-03 10:30:42 +05:30
harshsoni2024
69d0e3ce88
powerbi e2e count fix (#19626) 2025-02-03 10:24:59 +05:30
Akash Verma
837e6dd418
Metabase: Ingest orphan charts inside a dashbaord named Default (#19419) 2025-01-28 16:53:19 +05:30
Mayur Singal
d2dc7bd038
Fix #19489: Optimise multithreading for lineage (#19524) 2025-01-27 18:15:58 +05:30
harshsoni2024
b1d481f2f1
issue-16744: salesforce column description with toggle api (#19527) 2025-01-27 16:54:35 +05:30
Imri Paran
41b1ec081d
tests(e2e): increase CI for sampling test (#19519)
based on experiment in https://gist.github.com/sushi30/3083e96c9081371fa55e55b0847b96d2
2025-01-27 09:31:43 +00:00
Imri Paran
e003a5f9ed
tests(e2e): data quality (#19513)
* tests(e2e): data quality

- use the data quality pipeline new schema with a list of service configurations

* format
2025-01-24 14:01:54 +00:00
Imri Paran
5fc20c1ab2
tests: fix sample row count for bigquery (#19511)
- changed method name to expected_sample_count
2025-01-24 10:44:21 +01:00
olof-nn
fd2575d244
ISSUE-19454: Fixes broken looker lineage (#19456)
* ISSUE-19454: Fixes the broken lineage in looker when backticks enclosed table refs

* refactor

* use isort

* Update ingestion/src/metadata/ingestion/source/dashboard/looker/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-01-22 17:56:01 +05:30
Keshav Mohta
7bea4f957f
Feature: Docker Host Retry (#19127) 2025-01-14 19:48:10 +05:30
Akash Verma
072711a563
Added condition for datatype=array without type consistency (#19312) 2025-01-14 14:05:42 +05:30