Mayur Singal
88ab7475e7
MINOR: Restructure dbServiceName field in dashboard and pipeline ( #15548 )
2024-03-15 12:42:47 +05:30
Mayur Singal
b643206bba
Fix #11905 : Automated lineage between external table and container snowflake ( #15537 )
2024-03-15 00:52:41 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration ( #15317 )
...
* 🎉 Init OpenLineage connector
Co-authored-by: dechoma <dominik.choma@gmail.com>
* MLH - make linter happy
* review fixes
* 🐛 Fix path for ol event in tests
* 🐛 Fix path for ol event in tests
* Update ingestion/setup.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* review fixes 2
* linter
* review
* review
* make linter happy
* fix test_yield_pipeline_lineage_details test
* make linter happy
* fix tests
* fix tests 2
---------
Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
IceS2
418e281daa
Fixes 15375: Metabase metadata extraction fix ( #15376 )
2024-02-28 13:23:53 +05:30
Teddy
056e6368d0
Issue #14765 - Preparatory Work ( #15312 )
...
* refactor!: change partition metadata structure for table entities
* refactor!: updated json schema for TypeScript code gen
* chore: migration of partition for table entities
* style: python & java linting
* updated ui side change for table partitioned key
* miner fix
* addressing comments
* fixed ci error
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-02-28 07:11:00 +01:00
Imri Paran
aeb5fbe303
fixes #12591 : add BigTable ( #15122 )
...
* feat(connector): add BigTable
* bigtable work
1. docstrings
2. tests
3. created a Row BaseModel
4. implemented a ClassConverter
* docs moved to separate PR
* format files
* minor cosmetic
- removed TODO
- changed headers' year to 2024 for new files
- fixed typos
* format
* formatting and comments
1. added missing docstrings.
2. abstracted the _find_instance method.
3. aliased the IDs used in the BigTable connection
* added comment regarding private key
* added comments regarding column families
* enclose get_schema_name_list in `try/except/else`
* format
* streamlined get_schema_name_list to include all logic in the try block
2024-02-13 08:28:01 +01:00
NiharDoshi99
2b56e34b19
#14930 bigquery support for pk, fk and column view description ( #15042 )
2024-02-07 16:49:27 +05:30
Onkar Ravgan
edb9c21bfd
Added /view to tableau dashboard url ( #15031 )
2024-02-05 20:18:02 +05:30
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema ( #14956 )
...
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
IceS2
373cafcda2
Fixes #5448 : Implement initial Iceberg Connector using PyIceberg ( #14825 )
...
* Create the iceberg connection schema
* Link the IcebergConnection configuration with the forms on the UI
* Add the pyiceberg dependency on the ingestion package
* Create the get_connection and test_connection functions
* First iteration on the iceberg ingestion logic
* Add A more comprehensive implementation of the Iceberg Source
* Add UnitTests
* Update icebergConnection definition
* Update the iceberg souce code based on new schema
* Updated icebergConnecgtion schema for simplicity and to be able to configure Converters
* Updated setup dependencies to be more flexible
* Updated get_owner_ref logic
* Fix formatting
* Changed the icebergConnection json schema structure to enable the ClassConverters
* Add the IcebergCatalog and IcebergFileSystem ClassConverters
* Refactor the code to take into account the new jsonSchema structure
* Fix formatting
* Add Documentation for the Iceberg Connector
* Fix Menu order for Iceberg
* ui: add Iceberg service icon and constant
* Fix DynamoDb Catalog issue due to how PyIceberg instantes it
* Changed uri title to URI
* Fix ClassConverter for Iceberg
* Fix GetSecretValue for password types
* Fix formatting
* Fix formatting
* Add Iceberg Connector Images for the docs
* Add pylint disable for Hacky super() call
* Add Iceberg.md for the UI docs
* Fix pylint complaint
* Fix pylint complaint
* Fix UnitTests
* fix type error and unit tests
* update pipeline type checks
* Fix Sonar Cloud complaints
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
NiharDoshi99
c1d62186df
MINOR - metadata tag extraction for Databricks ( #14874 )
...
* metadata tag extraction for databaricks
* fix python test
* changes as per comment
* fix python test
* fix python checkstyle
2024-01-26 07:09:24 +01:00
Ayush Shah
1552aeb2de
Fix #13149 : Multiple Project Id for Datalake GCS ( #14846 )
...
* Fix Multiple Project Id for datalake gcs
* Optimize logic
* Fix Tests
* Add Datalake GCS Tests
* Add multiple project id gcs test
2024-01-25 10:52:16 +01:00
Pere Miquel Brull
85e2058979
MINOR - Fix & Organize topology context ( #14838 )
...
* MINOR - Fix & Organize topology context
* Handle missing context charts
2024-01-25 08:22:07 +01:00
Onkar Ravgan
80fff72949
Fix #14794 : Refactored and cleaned owner processing in sources ( #14817 )
...
* refactor owner processing
* Add exception handling and fix pytest
* review comments addressed
* looker tests
* fixed pycheckstyle
2024-01-25 06:46:22 +01:00
Shiyang Xiao
9f5a70bd71
MINOR - update docs & added unit test for SAS Connector ( #14743 )
...
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-23 14:55:29 -08:00
Pere Miquel Brull
337796d612
MINOR - Fix SP topology context & Looker usage context ( #14816 )
...
* MINOR - Fix SP topology context & Looker usage context
* MINOR - Fix SP topology context & Looker usage context
* Fix tests
2024-01-23 07:02:39 +01:00
NiharDoshi99
3f78e072e1
#13429 support for struct data type in hive ( #14785 )
2024-01-19 18:26:53 +05:30
Onkar Ravgan
f2219a10f3
Fixed oracle tests ( #14738 )
2024-01-16 17:39:10 +01:00
Onkar Ravgan
64a4e1afce
Fix 12180, 14158: Added LF tags to Athena ( #14718 )
...
* Added LF tags to athena
* fixed pytests
* Added docs
2024-01-16 14:24:31 +05:30
NiharDoshi99
54d34934c1
#14630 added oracle stored procedures ( #14641 )
2024-01-15 18:28:27 +05:30
Pere Miquel Brull
24643a397a
#14492 - Fix Snowflake SP parsing with empty signature ( #14623 )
2024-01-08 11:16:35 -08:00
Mayur Singal
a789fc86d6
Fix #13053 : Remove Connection URI config MongoDB ( #14584 )
...
* Fix #13053 : Remove Connection URI cofig MongoDB
* pyformat & test fixes
2024-01-05 10:51:12 -08:00
Pere Miquel Brull
f4bbca3f72
MINOR - Clean topology & add tests ( #14527 )
...
* Clean topo
* Format
* Add tests
* Fix tests
* Merge main
2023-12-29 17:00:59 +01:00
Pere Miquel Brull
b84ce33b80
#11799 - Fix Airfow ownership & add pipeline tasks ( #14510 )
...
* Fix airflow owner and add tasks
* Add pipeline tasks ownership
* MINOR - Fix py CI
* Add pipeline tasks ownership
* Add pipeline tasks ownership
* MINOR - Fix py CI
* MINOR - Fix py CI
* Add pipeline tasks ownership
* patch team
* patch team
* Format
2023-12-28 10:25:00 -08:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 ( #14398 )
...
* MINOR - Bump delta for 3.11
* Update flags
* MINOR - Bump delta for 3.11
* Update tests regex
* Update version
* Deprecations
* Format
* Version
* Try delta spark
* Skip delta tests for 3.11
* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Lucas Garcia
fe06b5cbb2
#14235 : adding dialect based on connection type to LineageParser ( #14249 )
...
* Fix #14235 : adding dialect based on connection type to LineageParser
* Fix: formating changes
* Update ingestion/src/metadata/ingestion/source/dashboard/metabase/metadata.py
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* style: fix indentation errors
* Fix pytest
---------
Co-authored-by: LucasGarcia07 <lucas.junqueira@hurb.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-12-08 19:49:59 +05:30
NiharDoshi99
8d925c46a5
#13696 : add support for dot in schema name to fetch tables ( #14246 )
2023-12-08 12:04:28 +05:30
Mayur Singal
389ae79d3c
#14115 : Separate Unity Catalog From Databricks ( #14138 )
2023-12-04 11:22:46 +05:30
Onkar Ravgan
ebb2317cf0
Fix 14040: Part 1 Remove get_by_name calls from topology ( #14098 )
...
* Changed for database
* Added changes for dashboard_service
* Changed for messaging service
* Changed for mlmodel service
* Changed for pipeline service
* Changed for search service
* Changed for objectstore service
* fixed wrong import
* fixed lint
* fixes
* fixed pytests
* fixed domo db pytest
* Fixed review comments
2023-11-27 16:15:47 +05:30
Mayur Singal
9f14ef7fab
Fix #13954 : Fix ParseException for older version of databricks ( #14015 )
2023-11-20 13:14:40 +05:30
Mohit Yadav
10d8ec84fe
Logs added for Search Indexing and Stats issue fixed ( #13956 )
...
* Logs added for Search Indexing and Stats issue fixed
* Fix uninstall error
* Add error handling
* fix lint
* Push Job Level Exception on top
* disable flaky tests
* Fix Logs not visible in Search
---------
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-11-13 23:39:56 +05:30
Mayur Singal
367bac9064
Fix #13787 : Add support for ES data types ( #13916 )
...
* Fix #13787 : Add support for ES data types
* fixed tests
---------
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-11-10 20:14:42 +05:30
Ayush Shah
bfb361dc85
Fix Bigquery lineage Pytests ( #13695 )
2023-10-25 11:15:41 +05:30
Ayush Shah
57cb72c26f
Fix Checkstyle ( #13683 )
2023-10-23 15:51:40 +05:30
Iaroslav Frolikov
420da29841
Fixes #13607 : BigQuery lineage ingestion fails when using GcpCredentialsPath authentication config ( #13608 )
2023-10-23 15:42:06 +05:30
Onkar Ravgan
0f0bccdd45
Converted and fixed pipelinestatus timestamps to milliseconds ( #13670 )
...
* fixed pipelinestatus timestamps in mills
* Added migrations
2023-10-20 09:39:24 -07:00
Pere Miquel Brull
660bf01a5b
Fix Stored Procedures Lineage for multi-db processes ( #13655 )
2023-10-20 09:14:08 +02:00
Teddy
1cbdfb3ae7
Fixes #12601 - column filter for profiler workflow ( #13535 )
...
* fix: sample data ingestion to match entity profiler column setting
* fix: python linting
* fix: updated fn call
* fix: added logic to handle json filed in datalake connector
* fix: handle NA values in parsing
* fix: reverted sampler changes from #13338
* fix: reverted metric changes from #13338
* fix: added datalake profiler ingestion test
* fix: python linting
* fix: removed normalization of json blob in NoSQL db
2023-10-12 14:51:38 +02:00
Ayush Shah
08d7ee6d55
Fixes #13052 : Datalake Nested Columns Sample Data ingestion ( #13338 )
2023-10-08 20:08:51 +05:30
Mayur Singal
0090286924
Fix Bigquery Test connection for multiproject ( #13380 )
...
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 14:50:42 +05:30
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config ( #13310 )
...
* Create ometa client once and pass it around & improve pycln config
* Fix
* Fix
* Fix tests
* Fix maven ci
* Fix tests
* Fix tests
* Fix tests
* Format
* Fix DI
2023-10-04 09:14:03 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs ( #13376 )
...
* Prepare Storage Connector for ADLS & Docs
* Format
* Fix test
2023-10-02 12:15:09 +02:00
Cristian Calugaru
5d8457b597
Fixes ISSUE-10587: global manifest option for storage services ( #12017 )
...
* global manifest option for storage services
* added a no metadata config source option for global manifest s3 services option
* merge fixes
* more merge fixes.
* black stuff
* test fixes
* formatting
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-28 07:55:40 +02:00
Ayush Shah
04760177f6
Fixes 13321: Fix Test Connection timeout ( #13323 )
2023-09-25 15:17:38 +05:30
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts ( #13247 )
...
* Rename sandboxDomain to instanceDomain
* Test Get Charts in DomoDashboard
* Fix schemas
* Fix test
* Fix test
* Rename to Auto Tag PII
* Fix query test
* Fix query test
* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
02094179e6
Fix #12899 : UI to use Tier TAG displayName if provided ( #13232 )
...
* Fix #12899 : UI to use Tier TAG displayName if provided
* fix python test
2023-09-17 14:28:10 -07:00
Pere Miquel Brull
442528267c
Simplify topology & update context management ( #13196 )
2023-09-15 09:44:42 +02:00
Ayush Shah
5fea08cd33
Datalake: Add manifest file support, fix profiler metrics, add array and json column type support ( #13017 )
2023-09-13 15:15:49 +05:30
Pere Miquel Brull
f0995cbddc
Part of #12998 - Prep Stored Procedures Skeleton for Snowflake ( #13121 )
...
* Prep Stored Procedures Skeleton for Snowflake
* Update pylint and add migrations
* Fix test
* Reuse source url computation
2023-09-12 14:25:42 +02:00
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy ( #13061 )
2023-09-08 12:40:48 +05:30
