Mayur Singal
b3caf5b5d1
Fix #20024 : Fix get schema names logic for postgres ( #21036 )
2025-04-29 22:49:39 +05:30
Pere Miquel Brull
d901dd2948
FIX #16284 - Toggle if we want to raise workflow errors ( #20969 )
...
* FIX #16284 - Toggle if we want to raise workflow errors
* schema
* schema
* move prop
* fix
* move prop
* improve error handling
* Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/IngestionPipelineRepository.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/IngestionPipelineRepository.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add the `Raise on Error` option to the ingestion schedule step
* Revert "Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/IngestionPipelineRepository.java"
This reverts commit 985b73513a59695c6bb39ad41c2d273bbf4e5d22.
* Update the tests
* Fix sonar issue
---------
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-04-29 08:19:13 +02:00
Ayush Shah
7dd987a799
feat: Add logging for table processing in DatalakeSource class ( #20968 )
2025-04-28 11:32:26 +05:30
Keshav Mohta
6d64fbf9c5
fix: service connection config not getting updated with multiple project ids after ingestion ( #20975 )
2025-04-25 14:25:50 +00:00
Teddy
63a55437ae
GEN-1412: Implement load test logic ( #19155 )
...
* feat: implemented load test logic
* style: ran python linting
* fix: added locust dependency in test
* fix: skip locust in 3.8 as not supported
* fix: update gcsfs version
* fix: revert gcsfs versionning
* fix: fix gcsf version to 2023.10
* fix: dagster graphql and gx versions
* fix: dagster version to 1.8 for py8 compatibility
* fix: fix clickhouse to 0.2 as 0.3 requires SQA 2+
* fix: revert changes from main
* fix: revert changes compared to main
2025-04-24 16:08:38 +02:00
Keshav Mohta
0488bef060
fix: string type as get system datatype ( #20922 )
2025-04-24 15:40:58 +05:30
Teddy
209793f315
MINOR - Add support for GX 1.4 ( #20934 )
...
* fix: add support for GX 0.18.22 and GX 1.4.x
* fix: add support for GX 0.18.22 and GX 1.4.x
* style: ran python linting
* fix: skip test if GX version is not installed
2025-04-24 11:55:04 +02:00
Teddy
75b7e463be
ISSUE #19175 : Handle pk for snowflake in data diff ( #19734 )
...
* fix: handle pk for snowflake in data diff
* fix: trino failure
2025-04-23 12:15:42 +02:00
harshsoni2024
0d4e9e0e09
MINOR: REST minor fixes ( #20907 )
2025-04-22 15:10:15 +05:30
Mayur Singal
88d8553084
Revert "MINOR: Improve UDF Lineage Processing & Better Logging Time & MultiProcessing ( #20848 )" ( #20872 )
...
This reverts commit 5ea9f22492749867f9ea53465b817f52bd383ca2.
2025-04-17 00:35:56 +05:30
Mayur Singal
5ea9f22492
MINOR: Improve UDF Lineage Processing & Better Logging Time & MultiProcessing ( #20848 )
2025-04-17 00:09:52 +05:30
Sasha Malahov
105ba064a9
MINOR: Kinesis missing nexttoken
2025-04-16 18:57:31 +05:30
Mayur Singal
654529ab7a
MINOR: Suppress Pydantic Warnings ( #20851 )
2025-04-16 16:44:14 +05:30
Keshav Mohta
1a6224824b
Fixes: BQ Multiple Project E2E ( #20797 )
...
* fix: bq e2e lineage and counts
* fix: bigquery multiple project classify
* fix: tests count from 19 to 17
2025-04-15 17:35:22 +05:30
Teddy
1edeb0baf8
MINOR: classification + test workflow for BQ multiproject ( #20779 )
...
* fix: classification + test workflow for BQ multiproject
* fix: deleted e2e test as handled from the UI
* fix: failing test case
2025-04-15 10:37:29 +02:00
Mayur Singal
3e4b8f5293
MINOR: Fix 'lr_sqlparser' referenced before assignment ( #20823 )
2025-04-15 13:00:04 +05:30
Mayur Singal
40ab1814c0
MINOR: Always Include DDL for Views ( #20784 )
2025-04-15 12:59:50 +05:30
Mayur Singal
56d3c335a7
MINOR: Fix pydantic warnings in ingestion ( #20822 )
...
* MINOR: Fix pydantic warnings in ingestion
* pydantic fix
2025-04-15 06:59:05 +02:00
chrisrayrayne
b14f83940a
Fixes Issue 20189: REST connector checks updated ( #20736 )
2025-04-15 10:24:57 +05:30
Suman Maharana
4dd94516e7
Fix : Dashboard OverrideMetadata ( #20793 )
2025-04-14 12:07:02 +05:30
Pere Miquel Brull
6d02ed8b5c
MINOR - Fix py checkstyle ( #20789 )
...
* MINOR - Fix py checkstyle
* dummy commit
2025-04-13 11:58:52 +02:00
Sasha Malahov
6dac3550ba
fix(ingestion): Correct Topic import in Kinesis source ( #20787 )
...
This PR fixes a bug in the Kinesis messaging source where the `Topic` class was incorrectly imported from `metadata.generated.schema.type.schema` instead of the correct entity
definition path `metadata.generated.schema.entity.data.topic`.
**Problem:**
The `yield_topic_sample_data` method used the `type.schema.Topic` definition when calling `fqn.build` and `metadata.get_by_name`. These functions expect the main entity
class.
**Fix:**
Changed the import statement to use `metadata.generated.schema.entity.data.topic.Topic`.
This ensures the correct type definition is used when interacting with the FQN utility and metadata API, preventing potential downstream issues.
2025-04-13 06:38:07 +02:00
Akash Verma
5cbe9badef
Wherescape Connector ( #20500 )
...
* omd side ws connector files
* Removed files
* add beta tag
* update enum name
* rename connection to databaseConnection
* Revert "rename connection to databaseConnection"
This reverts commit 9f1bc74e7aa6c156bedb8eefeb1a5435fcf72319.
* rename from connection to metastore
* rename connection to dbconnection
* UI Generated files
* fix connector UI
* fix connector UI
---------
Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sweta Agarwalla <swetaagarwalla13@gmail.com>
2025-04-11 18:04:58 +05:30
Keshav Mohta
d65a05e865
fix: added metadata in snowflake get_ometa_tag_and_classification ( #20719 )
2025-04-09 18:00:12 +05:30
Keshav Mohta
18d6701429
Fixes: Snowflake Tags Ingestion ( #20710 )
2025-04-09 07:12:22 +02:00
Mayur Singal
b7d43e7ee2
MINOR: Improve threading for lineage ( #20668 )
2025-04-07 18:31:52 +05:30
Mohit Tilala
f7c4cc54f4
Revert "Fixes #20649 : removed memory leak in Redshift ingestion ( #20650 )" ( #20667 )
...
This reverts commit 6c725278e3e7bd5cb5f492108fa27dc8f9487f82.
2025-04-07 18:16:50 +05:30
Mayur Singal
ee5d8eee8b
Revert "MINOR: Implement Column Validation in Lineage ( #20544 )" ( #20658 )
2025-04-07 17:13:35 +05:30
harshsoni2024
7953f98097
issue-20546: REST connector enhancements ( #20634 )
2025-04-07 10:22:45 +05:30
Katarzyna Kałek
6c725278e3
Fixes #20649 : removed memory leak in Redshift ingestion ( #20650 )
...
* closing proxy result in redshift ingestion
* fixed formatting
---------
Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
2025-04-06 23:32:38 +05:30
Suman Maharana
61e500253f
Fix: Improved test suite logging ( #20635 )
...
* Fix: Improved test suite logging
* linting
2025-04-04 16:51:21 +05:30
Imri Paran
f6441ad404
fix: trino data diff paths ( #20457 )
...
requires https://github.com/open-metadata/collate-data-diff/pull/6
2025-04-03 15:48:10 +02:00
Suman Maharana
5275975d31
Fix: dbt cloud latest run execution ( #20573 )
...
* Fix: dbt cloud latest run execution
* update latest run id
* set default to 100
2025-04-03 11:13:17 +05:30
Mayur Singal
7760663b22
MINOR: Change ingestion licence header ( #20549 )
2025-04-03 10:39:47 +05:30
Mayur Singal
7991715135
MINOR: Implement Column Validation in Lineage ( #20544 )
2025-04-02 17:40:40 +05:30
harshsoni2024
f267d4ef01
issue-20519: Support PowerBI Owners ingestion ( #20525 )
2025-04-02 16:11:27 +05:30
Mayur Singal
c16b3df547
MINOR: Fix public schema lieage for postgres ( #20548 )
2025-04-02 15:30:24 +05:30
Abdallah Serghine
2e0822b830
ISSUE-20427: fix tableau ingestion for null upstream table queries ( #20428 )
...
For tableau ingestion, code does not handle properly null upstream custom table queries
and null values for table OM entities.
Co-authored-by: Abdallah Serghine <abdallah.serghine@olx.pl>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-04-02 11:53:30 +02:00
Ayush Shah
eca3770a93
MINOR: Update Playwright integration test workflows to use 'playwright' deps ( #20558 )
2025-04-01 23:42:31 +05:30
Teddy
fbf6377e3f
fix: switch dbx to approx_percentile ( #20554 )
2025-04-01 17:36:43 +02:00
Mohit Tilala
06ab82170b
Fixes #19534 : Snowflake stream ingestion support ( #20278 )
2025-04-01 13:02:37 +05:30
Mohit Tilala
7ad97afa62
Fixes #19690 : Add QlikCloud dashboard filter by space name type ( #20315 )
2025-04-01 13:00:50 +05:30
Pere Miquel Brull
c08273b4ad
MINOR: Allow loading ometa from env ( #20511 )
2025-03-31 12:06:33 +02:00
Mayur Singal
d9fc607768
MINOR: Fix python checkstyle ( #20489 )
2025-03-31 12:02:32 +05:30
Katarzyna Kałek
fdec8c7b6b
deleted empty warning from s3 ingestion ( #20477 )
...
Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
2025-03-27 16:36:51 +01:00
Keshav Mohta
3c17a3025c
Fixes: Ingest System Tags ( #20432 )
2025-03-27 20:00:33 +05:30
Mayur Singal
e6b7b89f86
Fix #20236 : Handle Sample Data with non-utf8 characters ( #20380 )
2025-03-27 14:20:26 +05:30
Ayush Shah
7a3990f350
Fixes 19119: Enhance TableCustomSQLQueryValidator to support threshold operation ( #20307 )
2025-03-27 13:11:56 +05:30
Ayush Shah
653c878497
MINOR: Transform Reserved keywords like quotes to OM compatible ( #20459 )
2025-03-27 13:02:07 +05:30
Mayur Singal
766d0caebc
MINOR: Lineage Improvements ( #20446 )
2025-03-27 11:57:23 +05:30
