Pere Menal-Ferrer
6ea630d7ef
DevEx: Ingestion development improvement (focus on unit testing) ( #21362 )
...
* Fix test_amundsen: missing None
* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings
* Refactor referencedByQueries validation to use field_validator as per deprecation warning
* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning
* Move superset test to integration test as they are using testcontainers
* Add install_dev_env target to Makefile for development dependencies
* Add test-unit as extra in setup.py
* Skip failing IT test. Requires further investigation.
2025-05-26 10:38:17 +02:00
mgorsk1
474f77b46b
feat: use table_id instead of table_name for fetching Superset dataset columns ( #19618 )
2025-02-03 10:30:42 +05:30
Katarzyna Kałek
47c75fe6a7
Enhanced Glue ingestion with external table features ( #18511 )
...
* added fileFormat, locationPath and external table lineage to Glue ingestion
* Improve Lineage Label
---------
Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-05 21:48:20 +05:30
Onkar Ravgan
e6705f25b3
fixed dbt tag name ( #18273 )
2024-10-15 16:43:03 +05:30
Suman Maharana
bc6f4824ea
Added DBT tests with versionless and fixed v7 parsing ( #18028 )
2024-09-27 19:53:27 +05:30
Onkar Ravgan
80efc7075f
Fix #15163 : Added SAP ERP Connector
2024-07-04 10:57:46 +05:30
Suman Maharana
2477bbc9ca
Fixes #16574 : Fix Databricks pipeline task status ingestion ( #16802 )
...
* Fixed Databricks pipeline task issue
* updated test cases
* addressed review
2024-06-27 12:07:04 +05:30
Suman Maharana
f425d9bf73
Fixes #16591 : DBT Glossary Term linking Independent from Tag rule ( #16745 )
2024-06-21 17:36:25 +05:30
harshsoni2024
68e036418c
Fix #15719 : Improve unit test to increase coverage. ( #15905 )
...
* issue-15719: unit test for superset db source
* issue-15719: use testcontainers for superset_api client test
* issue-15719: superset-api yield data changes
* fix failed test cases due to testcontainer version
* issue-15719: postgres container version fix
* issue-15719: setup & teardown with testcontainers
* issue-15719: remove more patch code
2024-04-29 08:00:39 +02:00
Suman Maharana
16eaf925e9
FIX #13553 Added option to exclude drafts: superset ingestion ( #15770 )
...
* Added option to exclude drafts: superset ingestion
* Updated supserset yaml docs
* Added tests for exlcude draft dashboards
* Added tests for exlcude draft dashboards
* Formatted queries.py
2024-04-03 17:07:02 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration ( #15317 )
...
* 🎉 Init OpenLineage connector
Co-authored-by: dechoma <dominik.choma@gmail.com>
* MLH - make linter happy
* review fixes
* 🐛 Fix path for ol event in tests
* 🐛 Fix path for ol event in tests
* Update ingestion/setup.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* review fixes 2
* linter
* review
* review
* make linter happy
* fix test_yield_pipeline_lineage_details test
* make linter happy
* fix tests
* fix tests 2
---------
Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
Shiyang Xiao
9f5a70bd71
MINOR - update docs & added unit test for SAS Connector ( #14743 )
...
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-23 14:55:29 -08:00
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support ( #12159 )
...
* Add PGSpider service connector
* add sqlalchemy-pgspider to setup.py
* Delete unnecessary code and move files.
* Change to work as a Postgres connector sub-module
* Rebuild PGSpiderLineage class into a set of funcs
---------
Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Mayur Singal
05dc42bdb8
Fix #11808 : Handle lineage for single db sources in superset ( #11933 )
2023-06-09 12:43:06 +05:30
Mayur Singal
e9992a52a8
Fix #1604 : Add Spline Pipeline Connector ( #11562 )
...
* Fix #1604 : Add Spline Connector
* Add tests & grammer validation
* Spline UI Changes & Docs
* fix pipeline workflow doc
* chore: use common field for dbService name
* chore: use const for beta services
* chore: add service icon
* Update ingestion/src/metadata/ingestion/source/pipeline/spline/metadata.py
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-05-19 14:46:32 +05:30
Onkar Ravgan
ca623d0693
dbt performance improvement ( #10366 )
...
* dbt performance improvement
* reduced es calls
* Added unit tests
2023-03-08 06:37:34 +01:00
Onkar Ravgan
8191f92438
Added dbt oracle fix ( #10282 )
...
* Added dbt oracle fix
* Addressed review comments
* Added unit tests
2023-02-23 19:17:54 +05:30
Mayur Singal
42a8622981
Fix #10215 : Fix Query Log Usage ( #10232 )
2023-02-21 17:41:22 +05:30
Mayur Singal
0bea02c202
Superset Unit Tests ( #10078 )
2023-02-09 19:45:24 +05:30
Mayur Singal
392107bc4a
Datalake Avro & Json Lines Support ( #10129 )
2023-02-08 17:31:25 +00:00
NiharDoshi99
012f96a7a2
Domo issue ( #10080 )
...
* Domo issue
* added pydantic models and changes as per comment
* Fix: added ownership details
* fixed pytest
2023-02-07 17:16:26 +05:30
NiharDoshi99
029dbe892e
Fix: added test case for atlas ( #9678 )
...
* Fix: added test case for atlas
* Fix: resolved conflict
* Fix: changing back neo4j to old version
* Fix: changing back neo4j to old version
* Fix: changes as per comment
* Fix: changes as per comment
* Fix: python checkstyle
2023-01-13 16:07:29 +05:30
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector ( #9554 )
...
* feat: update databricks_client
* feat: create pipelineconnection & connection
* test: add unit test
* feat: add examples file
* fix: remove unused imports
* style: fix python style
* fix: code review adjustments
* fix: remove unneeded schema ref
* fix: format requirements
* fix: change context calls
* test: add pipeline_status test
* format: pylint format change
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
michizhou
da06d183db
feat: add QuickSight connector ( #8442 )
2022-11-08 19:54:49 +05:30
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud ( #8387 )
...
* Fixed: dagster to connect to cloud
* Fixed: changed test_cases
* Fixed: lineage between the assets and jobs
* Changes as per comments
* Fix python checkstyle
* Fix as per comments
* Fix added migration scripts
* minor fixes
2022-11-03 18:19:50 +01:00
NiharDoshi99
84a89d7181
Added DOMO Pipeline ( #8133 )
2022-10-18 17:52:28 +05:30
Milan Bariya
e7a3770240
Add: databricks usage and lineage added ( #7386 )
2022-10-14 07:06:28 +05:30
NiharDoshi99
0e44f7d35d
Fixed: Intergrated DOMO Connector APIs ( #7989 )
2022-10-13 10:55:48 +05:30
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API ( #7579 )
2022-09-23 17:46:40 +05:30
Pere Miquel Brull
6cebf37b6c
Fix #6906 - Add Nifi connector ( #6956 )
...
* Add Nifi connector
* Mock token
* Format
* pass verify param
2022-08-26 21:50:01 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added ( #6513 )
...
* dagster-connector-added
* code-formatted
* dagster-comments-removed
* dagster-comments-removed
* dagster-unittest-added
* code-formatted-pr-comments-resolved
* improved-code-quality
* increased-py-test-time-out
* improved-dagster-tests
* improved-dagster-tests
* reverted-pytests-timeout
* dagster-pytest-improved
* added-schema-in-omdcore
* updated-schema-in-omd-core
* ui-schema-updated-omd-core
* updated-dagster-schema
* dagter-schema-test-updated
2022-08-13 15:18:14 +05:30
Onkar Ravgan
7fdab21a72
Glue Topology Refactor ( #6231 )
...
* rebase with main
* refactored glue for topology
* added glue unit tests
* resolved review comments
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-28 15:04:43 +02:00
Mayur Singal
c6efe8c142
Fix #6158 : Fivetran Connector Added ( #6329 )
...
* Fivetran Connector Added
* Fivetran unit test added
* Fixed Unit Tests
* Added Fivetran Docs & Added Fivetran in Pipeline Listing
2022-07-27 20:50:56 +05:30
Mayur Singal
75ccc803ad
Fix #5410 : Topology for Pipeline Services ( #6085 )
...
Fix #5410 : Topology for Pipeline Services (#6085 )
2022-07-22 13:59:14 +02:00
Mayur Singal
5a6e7d743e
ISSUE-545: Added tests for QueryProcessor ( #2510 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-04 08:20:20 -08:00
