723 Commits

Author SHA1 Message Date
Teddy
209793f315
MINOR - Add support for GX 1.4 (#20934)
* fix: add support for GX 0.18.22 and GX 1.4.x

* fix: add  support for GX 0.18.22 and GX 1.4.x

* style: ran python linting

* fix: skip test if GX version is not installed
2025-04-24 11:55:04 +02:00
chrisrayrayne
b14f83940a
Fixes Issue 20189: REST connector checks updated (#20736) 2025-04-15 10:24:57 +05:30
Mayur Singal
ee5d8eee8b
Revert "MINOR: Implement Column Validation in Lineage (#20544)" (#20658) 2025-04-07 17:13:35 +05:30
Keshav Mohta
0796c6274b
Fixes: Databricks httpPath Required (#20611)
* fix: made databricks httpPath required and added a migration file for the same

* fix: added sql migration in postDataMigration file and fix databricks tests

* fix: added httpPath in test_source_connection.py and test_source_parsing.py files

* fix: added httpPath in test_databricks_lineage.py

* fix: table name in postgres migration
2025-04-07 13:33:55 +05:30
harshsoni2024
7953f98097
issue-20546: REST connector enhancements (#20634) 2025-04-07 10:22:45 +05:30
Suman Maharana
5275975d31
Fix: dbt cloud latest run execution (#20573)
* Fix: dbt cloud latest run execution

* update latest run id

* set default to 100
2025-04-03 11:13:17 +05:30
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
Mayur Singal
7991715135
MINOR: Implement Column Validation in Lineage (#20544) 2025-04-02 17:40:40 +05:30
harshsoni2024
f267d4ef01
issue-20519: Support PowerBI Owners ingestion (#20525) 2025-04-02 16:11:27 +05:30
Mohit Tilala
06ab82170b
Fixes #19534: Snowflake stream ingestion support (#20278) 2025-04-01 13:02:37 +05:30
Mohit Tilala
7ad97afa62
Fixes #19690: Add QlikCloud dashboard filter by space name type (#20315) 2025-04-01 13:00:50 +05:30
Ayush Shah
653c878497
MINOR: Transform Reserved keywords like quotes to OM compatible (#20459) 2025-03-27 13:02:07 +05:30
Sriharsha Chintalapani
706cebd97a
Opensearch connector (#19698)
* Fix #19667: OpenSearch Connector

* Fix #19667: OpenSearch Connector

* do not ingest any system level indexes

* fix pyformat

* Add AWS auth

* Use common schema and fix ssl config in client

* Add openseach connector docs and update schema

* Remove api key auth type and complete docs checklist

* Remove unnecessary httpx dependency and pyformat

* Add compatible version of httpx for elasticsearch

* Fix pylint fails and py-tests validation error

---------

Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
2025-03-18 18:45:25 +05:30
Akash Verma
cf7a442e32
Fixes #19891 : Added measures in powerbi (#19990) 2025-03-17 14:43:22 +05:30
Mayur Singal
d30fd90096
Minor: Query Cost Table Aggregation Endpoint (#20270) 2025-03-17 11:33:50 +05:30
Mayur Singal
581ab6ce71
MINOR: Fix pytests - usage count (#20247) 2025-03-14 09:07:40 +01:00
harshsoni2024
826279608f
issue-19892: parse powerbi table source (#20141) 2025-03-12 12:59:29 +05:30
Pere Miquel Brull
2e7c9a0875
FIX #19765 - Improve Column Name Scanner (#20136) 2025-03-07 14:32:59 +01:00
harshsoni2024
40a9c67875
Day 1 - Dashboard service lineage without db_service_name (#19911) 2025-03-07 11:16:58 +05:30
Akash Verma
af9d7c9fa3
Fixes #15696: Added Powerbi filter query (#19912)
* Powerbi filter query

* error handling

* Added test and minor changes

* Checkstyle

* resolved comments

---------

Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
2025-03-06 18:24:57 +05:30
Suman Maharana
137f86c00d
Day1 - Pipeline Usage (#19960) 2025-03-06 18:19:42 +05:30
Suman Maharana
0cd35015d7
#1049 Day 1 Dashboard Usage - Tableau, Looker (#19796) 2025-03-06 18:15:28 +05:30
Pere Miquel Brull
55d7e50441
MINOR - Add and remove data products Actions in Automator (#19948)
* MINOR - Add and remove Data Product assets in Automator config

* MINOR - Add and remove Data Product assets in Automator config

* domain mixin

* build ref

* build ref

* create types

* fix tests

* fix conflicts

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-03-05 07:11:17 +01:00
Pere Miquel Brull
6b7a9fe76c
MINOR - Update sampler tablenames (#19976)
* MINOR - Update sampler tablenames

* MINOR - Update sampler tablenames

* MINOR - Update sampler tablenames
2025-02-26 14:08:14 +01:00
Imri Paran
97fad806a2
Fixes 19755: Publish app config with status (#19754)
* feat(app): add config to status

add config to the reported status of the ingestion pipeline

* added separate pipeline service client call for external apps

* fix masking of pydantic model

* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py

* fix: execution time

* fix: mask secrets in dump json

* fix: for python3.8

* fix: for python3.8

* fix: use mask_secrets=False when dumping a model for create

* format

* fix: update mask_secrets=False for workflow configurations

* fix: use context directly when using model_dump_json

* fix: default behavior when dumping json

* format

* fixed tests
2025-02-25 16:51:49 +00:00
Pere Miquel Brull
91b62fdc32
FIX #19798 - Shortening SQA __tablename__ to avoid hitting errors in … (#19809)
* FIX #19798 - Shortening SQA __tablename__ to avoid hitting errors in postgres

* fix tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-02-17 09:37:06 +01:00
mgorsk1
0e260ce36a
feat: use dataset sql for superset lineage (#19617)
* 🎉 Init

* add fixes around row data type and backport fixes around array type

* refactor

* narrow down column lineage

* narrow down column lineage

* fix insert into

* review remarks + linter

* fix tests

* fix style check

* bring back compatibility with older python versions

* add unit tests

* fix tests

* linting

* expand tests

* make compatible with python 3.8

* checkstyle fix

* fix schema alias

* refactor - move get_dashboard_data_model_column_fqn to superset.utils as it's superset specific and won't work with other dashboard systems integrations
2025-02-12 07:14:31 -08:00
Mayur Singal
45efd878cb
Fix #19634: use system.query.history table for lineage - databricks (#19744) 2025-02-12 13:29:07 +05:30
Pere Miquel Brull
e4fa16e574
FIX - Profiler Source Include Views Filter (#19746)
* FIX - Profiler Source Include Views Filter

* FIX - Profiler Source Include Views Filter
2025-02-12 08:35:47 +01:00
Teddy
e1b3e08317
MINOR BQ sampler type missing (#19696)
* fix: missing entity type in bq sampler

* fix: failing tests
2025-02-11 10:46:34 -08:00
Suman Maharana
0d73ee7e36
Fixes #19171 - Looker parse derived view references query (#19687)
* Looker parse derived view references query

* Addressed Comments

* Linting

* Added Tests
2025-02-10 02:50:51 +05:30
Teddy
ef131d7e20
MINOR: Wrong attribute name in SampleConfig model (#19641)
* fix: wrong attribute name in SampleConfig model

* fix: test attribute

* fix: failing tests

* fix: trino filter error + adjust test to take into account null value

* fix: mssql and azuresql tablesample on views
2025-02-04 10:40:40 +01:00
Suman Maharana
3e3c702942
Fix - switch to collate-dbt-artifacts-parser (#19647)
* Switch to collate-dbt-artifacts-parser
2025-02-04 11:57:39 +05:30
Mayur Singal
208c40be09
Fix #19633: Fix databricks schema not found (#19646) 2025-02-04 11:42:11 +05:30
mgorsk1
474f77b46b
feat: use table_id instead of table_name for fetching Superset dataset columns (#19618) 2025-02-03 10:30:42 +05:30
Akash Verma
837e6dd418
Metabase: Ingest orphan charts inside a dashbaord named Default (#19419) 2025-01-28 16:53:19 +05:30
Mayur Singal
d2dc7bd038
Fix #19489: Optimise multithreading for lineage (#19524) 2025-01-27 18:15:58 +05:30
harshsoni2024
b1d481f2f1
issue-16744: salesforce column description with toggle api (#19527) 2025-01-27 16:54:35 +05:30
olof-nn
fd2575d244
ISSUE-19454: Fixes broken looker lineage (#19456)
* ISSUE-19454: Fixes the broken lineage in looker when backticks enclosed table refs

* refactor

* use isort

* Update ingestion/src/metadata/ingestion/source/dashboard/looker/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-01-22 17:56:01 +05:30
Keshav Mohta
7bea4f957f
Feature: Docker Host Retry (#19127) 2025-01-14 19:48:10 +05:30
Akash Verma
072711a563
Added condition for datatype=array without type consistency (#19312) 2025-01-14 14:05:42 +05:30
agriev
dcebc41e3f
Adds percona server for postgresql support (#19322)
* percona server for postgresql support

The only meaningful difference is version string in percona server for postgresql. So commit propose universal and safe way to detect server version by integer string, not complicated parsing of unformatted string.

* updated tests with get_server_version_num

commented outdated tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-13 17:51:40 -08:00
Keshav Mohta
4bd8994a00
Feature: Cassandra SSL (#19226) 2025-01-10 11:25:43 +05:30
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Akash Verma
b2898f7007
Cockroach enhancement (#19108) 2025-01-07 18:51:59 +05:30
Suman Maharana
00be51f299
Fixes #17747: dbt update owners (#19144)
* Fixes 17747: dbt update owners

* update messages

* addressed comments

* py_format

* py_format

* Added tests
2025-01-07 16:37:30 +05:30
Akash Verma
39dcb5baef
Feature : Cockroach db connector (#18961) 2025-01-02 13:07:55 +05:30
Akash Verma
69557e8716
fixes: #15742 Oracle stored package feature (#18852) 2024-12-16 19:35:20 +05:30
Teddy
79b2888bb5
fix: azuresql sampler logic (#19034) 2024-12-13 07:35:04 +01:00
Keshav Mohta
cde3a7dd1e
Feature: Cassandra Connector (#18943) 2024-12-12 15:12:55 +05:30