Onkar Ravgan
e6705f25b3
fixed dbt tag name ( #18273 )
2024-10-15 16:43:03 +05:30
Suman Maharana
bc6f4824ea
Added DBT tests with versionless and fixed v7 parsing ( #18028 )
2024-09-27 19:53:27 +05:30
Pere Miquel Brull
bcb29b46da
MINOR - Implement SAP Hana Lineage ( #17615 )
...
* MINOR - SAP Hana Lineage
* skeleton
* parser
* lineage
* manage formulas
* add cvs
* add cvs
* better typing
* enum
* handle cvs
* saphana docs
2024-08-30 07:42:43 +02:00
harshsoni2024
a117b0224d
MINOR: query log path as dir/file ( #17048 )
...
* fetch query log path as dir/file
* add test for multiple files log fetch
* non-csv log file test
2024-07-17 15:23:41 +05:30
Onkar Ravgan
80efc7075f
Fix #15163 : Added SAP ERP Connector
2024-07-04 10:57:46 +05:30
Suman Maharana
2477bbc9ca
Fixes #16574 : Fix Databricks pipeline task status ingestion ( #16802 )
...
* Fixed Databricks pipeline task issue
* updated test cases
* addressed review
2024-06-27 12:07:04 +05:30
Suman Maharana
f425d9bf73
Fixes #16591 : DBT Glossary Term linking Independent from Tag rule ( #16745 )
2024-06-21 17:36:25 +05:30
Matt Chamberlin
ac6ddbf6c4
MINOR: support JSONL datalake file types ( #16614 )
...
* fix: support JSONL datalake file types
* add jsonl zip file types
* update fileFormat enum in table schema
* add tests
* fix test data ref
* reformat
* fix tests
---------
Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-06-21 09:54:19 +02:00
Onkar Ravgan
87c8254c38
Fix #15454 : Added protobuf parser complex schema support ( #16071 )
...
* Added protobuf parser complex schema support
* Added options keyword in proto testing
2024-04-30 17:59:27 +05:30
harshsoni2024
68e036418c
Fix #15719 : Improve unit test to increase coverage. ( #15905 )
...
* issue-15719: unit test for superset db source
* issue-15719: use testcontainers for superset_api client test
* issue-15719: superset-api yield data changes
* fix failed test cases due to testcontainer version
* issue-15719: postgres container version fix
* issue-15719: setup & teardown with testcontainers
* issue-15719: remove more patch code
2024-04-29 08:00:39 +02:00
Suman Maharana
16eaf925e9
FIX #13553 Added option to exclude drafts: superset ingestion ( #15770 )
...
* Added option to exclude drafts: superset ingestion
* Updated supserset yaml docs
* Added tests for exlcude draft dashboards
* Added tests for exlcude draft dashboards
* Formatted queries.py
2024-04-03 17:07:02 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration ( #15317 )
...
* 🎉 Init OpenLineage connector
Co-authored-by: dechoma <dominik.choma@gmail.com>
* MLH - make linter happy
* review fixes
* 🐛 Fix path for ol event in tests
* 🐛 Fix path for ol event in tests
* Update ingestion/setup.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* review fixes 2
* linter
* review
* review
* make linter happy
* fix test_yield_pipeline_lineage_details test
* make linter happy
* fix tests
* fix tests 2
---------
Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema ( #14956 )
...
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
Shiyang Xiao
9f5a70bd71
MINOR - update docs & added unit test for SAS Connector ( #14743 )
...
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-23 14:55:29 -08:00
Ayush Shah
0d0d820305
MINOR: Add Static Sample Data, Modify Lineage ( #14752 )
2024-01-17 20:07:49 +05:30
Mayur Singal
a7c17c5351
Fix #14653 : Fix Unity Catalog no children struct issue ( #14688 )
2024-01-12 12:59:50 +05:30
Pere Miquel Brull
b250cd8808
Fix #13699 - Add separator for Storage Container manifest ( #13924 )
...
* Fix #13699 - Add separator for Storage Container manifest
* Fix #13906 - Fix add_mlmodel_lineage description field
* Add separator
* Add separator
2023-11-10 10:44:47 +01:00
Ayush Shah
5fea08cd33
Datalake: Add manifest file support, fix profiler metrics, add array and json column type support ( #13017 )
2023-09-13 15:15:49 +05:30
Pere Miquel Brull
e97d4befb1
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 ( #12776 )
2023-08-09 16:07:16 +05:30
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support ( #12159 )
...
* Add PGSpider service connector
* add sqlalchemy-pgspider to setup.py
* Delete unnecessary code and move files.
* Change to work as a Postgres connector sub-module
* Rebuild PGSpiderLineage class into a set of funcs
---------
Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Pere Miquel Brull
b59ff24531
Fix looker view parser ( #12254 )
2023-07-03 07:25:08 +02:00
Pere Miquel Brull
1ecf5607c7
Looker - Fix file extension and blob import ( #12232 )
...
* Fix file extension and blob import
* Fix file extension and blob import
2023-06-29 16:14:17 +02:00
Mayur Singal
05dc42bdb8
Fix #11808 : Handle lineage for single db sources in superset ( #11933 )
2023-06-09 12:43:06 +05:30
Mayur Singal
e9992a52a8
Fix #1604 : Add Spline Pipeline Connector ( #11562 )
...
* Fix #1604 : Add Spline Connector
* Add tests & grammer validation
* Spline UI Changes & Docs
* fix pipeline workflow doc
* chore: use common field for dbService name
* chore: use const for beta services
* chore: add service icon
* Update ingestion/src/metadata/ingestion/source/pipeline/spline/metadata.py
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-05-19 14:46:32 +05:30
Teddy
a7d98dddda
Fixes #9632 - Add Profiler Support for BQ Arrays of Structs ( #11059 )
2023-04-14 19:29:26 +05:30
Pere Miquel Brull
09b283818d
Rel to #10927 - Looker DataModel ( #10945 )
...
* Organise calls
* Prepare skeleton
* Add looker model handling
* Parse files as sql
* Handle labels
* Linting
* Format
* Fix version
* Also check the API for explore lineage
2023-04-11 08:44:00 +02:00
Mayur Singal
bbce9c5aa4
Fix #10489 : Handle unknown data types & store raw data type ( #10563 )
2023-03-23 11:41:29 +05:30
Onkar Ravgan
ca623d0693
dbt performance improvement ( #10366 )
...
* dbt performance improvement
* reduced es calls
* Added unit tests
2023-03-08 06:37:34 +01:00
Onkar Ravgan
8191f92438
Added dbt oracle fix ( #10282 )
...
* Added dbt oracle fix
* Addressed review comments
* Added unit tests
2023-02-23 19:17:54 +05:30
Mayur Singal
42a8622981
Fix #10215 : Fix Query Log Usage ( #10232 )
2023-02-21 17:41:22 +05:30
Mayur Singal
0bea02c202
Superset Unit Tests ( #10078 )
2023-02-09 19:45:24 +05:30
Mayur Singal
392107bc4a
Datalake Avro & Json Lines Support ( #10129 )
2023-02-08 17:31:25 +00:00
NiharDoshi99
012f96a7a2
Domo issue ( #10080 )
...
* Domo issue
* added pydantic models and changes as per comment
* Fix: added ownership details
* fixed pytest
2023-02-07 17:16:26 +05:30
NiharDoshi99
029dbe892e
Fix: added test case for atlas ( #9678 )
...
* Fix: added test case for atlas
* Fix: resolved conflict
* Fix: changing back neo4j to old version
* Fix: changing back neo4j to old version
* Fix: changes as per comment
* Fix: changes as per comment
* Fix: python checkstyle
2023-01-13 16:07:29 +05:30
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector ( #9554 )
...
* feat: update databricks_client
* feat: create pipelineconnection & connection
* test: add unit test
* feat: add examples file
* fix: remove unused imports
* style: fix python style
* fix: code review adjustments
* fix: remove unneeded schema ref
* fix: format requirements
* fix: change context calls
* test: add pipeline_status test
* format: pylint format change
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
michizhou
da06d183db
feat: add QuickSight connector ( #8442 )
2022-11-08 19:54:49 +05:30
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud ( #8387 )
...
* Fixed: dagster to connect to cloud
* Fixed: changed test_cases
* Fixed: lineage between the assets and jobs
* Changes as per comments
* Fix python checkstyle
* Fix as per comments
* Fix added migration scripts
* minor fixes
2022-11-03 18:19:50 +01:00
NiharDoshi99
84a89d7181
Added DOMO Pipeline ( #8133 )
2022-10-18 17:52:28 +05:30
Milan Bariya
e7a3770240
Add: databricks usage and lineage added ( #7386 )
2022-10-14 07:06:28 +05:30
NiharDoshi99
0e44f7d35d
Fixed: Intergrated DOMO Connector APIs ( #7989 )
2022-10-13 10:55:48 +05:30
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API ( #7579 )
2022-09-23 17:46:40 +05:30
Pere Miquel Brull
6cebf37b6c
Fix #6906 - Add Nifi connector ( #6956 )
...
* Add Nifi connector
* Mock token
* Format
* pass verify param
2022-08-26 21:50:01 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added ( #6513 )
...
* dagster-connector-added
* code-formatted
* dagster-comments-removed
* dagster-comments-removed
* dagster-unittest-added
* code-formatted-pr-comments-resolved
* improved-code-quality
* increased-py-test-time-out
* improved-dagster-tests
* improved-dagster-tests
* reverted-pytests-timeout
* dagster-pytest-improved
* added-schema-in-omdcore
* updated-schema-in-omd-core
* ui-schema-updated-omd-core
* updated-dagster-schema
* dagter-schema-test-updated
2022-08-13 15:18:14 +05:30
Onkar Ravgan
7fdab21a72
Glue Topology Refactor ( #6231 )
...
* rebase with main
* refactored glue for topology
* added glue unit tests
* resolved review comments
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-07-28 15:04:43 +02:00
Mayur Singal
c6efe8c142
Fix #6158 : Fivetran Connector Added ( #6329 )
...
* Fivetran Connector Added
* Fivetran unit test added
* Fixed Unit Tests
* Added Fivetran Docs & Added Fivetran in Pipeline Listing
2022-07-27 20:50:56 +05:30
Abhishek Pandey
bef4d412ce
uniontype-coverage-added ( #6348 )
2022-07-26 15:21:34 +02:00
Mayur Singal
75ccc803ad
Fix #5410 : Topology for Pipeline Services ( #6085 )
...
Fix #5410 : Topology for Pipeline Services (#6085 )
2022-07-22 13:59:14 +02:00
Mayur Singal
5a6e7d743e
ISSUE-545: Added tests for QueryProcessor ( #2510 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-04 08:20:20 -08:00
Sriharsha Chintalapani
f390bac375
Fix 2270: Column Type Parser ( #2271 )
...
* Fix 2270: Column Type Parser
* Fix 2270: Column Type Parser
* Added checks to allow arrayDataType and dataTypeDisplay
* Modified - arrayDataType and dataTypeDisplay
* Update sql_source.py
* Update sql_source.py
* file formatted
* Modified according to column_type_parser.py
* modified / refactored / deleted the files
* Added Tests, modified sql_source
* file formatted
* Added missing datatypes
* Added Tests
* Added Tests
* Added Tests - refactored expected output into a json file
* file formatted
* Sample Data Updated
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-01-22 23:32:58 +05:30
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values ( #1830 )
...
* Prepare tests for basic config cases
* Add test cases
* Use os expandvars
* Add missing commas
* Add missing commas
* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
