3502 Commits

Author SHA1 Message Date
Mayur Singal
dfb7eeaaaa
MINOR: Improve datalake processing table log (#22631) 2025-08-01 15:07:37 +05:30
Nick Acosta
0a395ea487
Fixed typo referring to airbyte in Fivetran file (#22686)
Co-authored-by: Nick Acosta <nick.acosta@getcollate.io>
2025-08-01 14:49:23 +05:30
IceS2
7f8298d49e
Update DataLake and PostgreSQL connection (#22682) 2025-08-01 11:08:43 +02:00
Pere Miquel Brull
9cb146591c
MINOR - Add Data Contract Alerts (#22671)
* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* rules

* rules

* rules

* fix retrieval by entity

* test dc

* test dc

* WIP: Data contract feature

* destructure component to it's own files

* WIP contract tab

* update local

* fix test

* First iteration for multi domain support

* fix inheritance fields

* fix inheritance fields

* fix create interface

* fix few more tests

* fix indexing updates

* fix domain rel

* update domain --> domains

* merge

* fix merge

* fix csv tests and createEntity interface

* Update generated TypeScript types

* Trigger Build

* migrations

* fix tests

* fix tests

* fix tests

* Update generated TypeScript types

* Trigger Build

* handle drive service

* fix pg migration

* fix domains ref after merge and clean python tests

* Update generated TypeScript types

* fix merge domains

* format

* add missing migrations

* Update generated TypeScript types

* tests

* Update generated TypeScript types

* Trigger Build

* tests

* tests

* fix py test

* migrate domain to domains and fix compilation errors

* fix domain assignement

* fix domain spec

* fix py tests

* fix data product creation issue

* fix domain tests

* fix bulk import

* fix tests

* fix tests

* fix query and domain migration

* fix py test

* fix playwrights

* fix getEntitiesWithDisplayName indexing quotes

* fix domain prapogation tests

* execute dc

* execute dc

* fix domain propagation

* Fix patch api

* fix domain schema build edit playwright

* fix test

* fix test

* fix domain selection issue and console errors

* handle semantics validation

* handle ingestion pipelines for DQ

* prepare for getting DQ status

* get last result

* update schema and prep sample data

* test

* handle data contracts tests in SDK

* handle data contracts tests in SDK

* execute test suite

* execute test suite

* add schema validations

* add schema validations

* add schema validations

* add schema validations

* handle data contracts tests in SDK

* add schema validations

* comments

* generate schemas

* add fields

* MINOR - Data Contract Alerts

* MINOR - Data Contract Alerts

* MINOR - Data Contract Alerts

* Fix url

* tests

* minor ui fixes around localization keys

* supporting locaization keys

* fix test with updated response

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2025-08-01 06:44:47 +02:00
MarkSaf17
58078f582c
[WIP] Issue 22627 (#22647)
* [ISSUE-22627] fix(dbt): Support documented owner configuration with backward compatibility

* typo

* added tests

---------

Co-authored-by: marsafronov <marsafronov@ecom.tech>
Co-authored-by: SumanMaharana <sumanmaharana786@gmail.com>
2025-07-31 11:02:18 +00:00
IceS2
f578a81277
MINOR: Update table profile config to add spark configs. Update spark config to add temp path (#22646)
* Update table profile config to add spark configs. Update spark config to add temp path

* Add default null value for sparkTableProfilerConfig

* Fix TableProfileConfig extension DAO query

* Update generated TypeScript types

* Implemented Dependency Injection for the ProfilerProcessorConfig

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-07-31 11:50:30 +02:00
Roman Sheludko
e098635842
fix: Remove redundant set of braces in additional_config (#22640)
After the changes in https://github.com/open-metadata/OpenMetadata/pull/22474 we've started to experience the following error:
2025-07-31 10:22:45 +02:00
Keshav Mohta
f8b588e6f0
Feature: Security Service (#22450)
* feat: security service initial code

* support secrity type for service from ui

* refactor: added missing java files for security service and require json files

* refactor: added doc file, oneOf java changes for ranger

* Merge branch 'main' of https://github.com/open-metadata/OpenMetadata into issue-20145

* update the security services schema to have domains instead of domain

* fix: added security service metadata pipeline

* Make changes for security service UI

* Update missing translation

* update type

* Fix the java checkstyle

* fix: added ingestionRunner in security service json

* Hide security service for OSS

* fix: reverse metadata documentation

* refactor: added securityConnection in test connection

* fix: documentation and ts file changes

* fix: deleted __init__ file and added SSISConnection again in the workflowclassconverter.java

* fix: convert domain to domains

* fix: added new file for basicAuth for ranger and change in rangerConnectionClassConverter.java file

* fix: java checkstyle

* fix: mysql security service sql

* fix: security service index mapping

* fix: moved migrations in new version and fix classConverterFactory

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2025-07-31 06:38:21 +02:00
Pere Miquel Brull
dfe3fd6357
MINOR - Data Contract Validation (#22541) 2025-07-30 23:01:27 +02:00
Ariel Schulz
798a20afd8
Fix test_connection to adhere to OMD standards for DB connections (#22478) 2025-07-31 00:29:03 +05:30
Suman Maharana
3a90b38a26
Fix: Tableau ca cert auth (#22041)
* Fix: Tableau ca cert auth

* py_format

* Added ssl tests

* fix lint errors
2025-07-30 09:38:47 +05:30
harshsoni2024
50428e2e7b
feat-22574: Datalake ingestion fix for larger files (#22575) 2025-07-29 20:40:19 +05:30
Suman Maharana
670dc53b46
Minor: fix tableau handle none entities (#22630)
* Minor: fix tableau handle none entities

* added tests
2025-07-29 13:58:11 +02:00
Ayush Shah
987e3ed9e0
Fixes #22144: Enhance memory management and proper cleanup of resources in profiler (#22438) 2025-07-29 16:22:21 +05:30
Mayur Singal
199e3b981c
Fix #14830: Ignore non current columns for iceberg tables for glue & athena (#22564) 2025-07-29 16:19:09 +05:30
Mayur Singal
cc9506db20
MINOR: Postgres Implement schema fallback (#21858)
* MINOR: Postgres Implement schema fallback

* missing sql_lineage file
2025-07-29 14:45:21 +05:30
Suman Maharana
54dcdc7d82
Fix #20689: Trino Column validation errors for highly complex fields (#22421)
* Fix: Trino Column validation errors for highly complex fields

* addressed copilot comms

* fixed tests

* fixed tests and addressed comms

* missed file
2025-07-28 11:11:44 +05:30
IceS2
bad772db39
FIX #22099: enable 'Column values to be in set' test case for boolean columns (#22491)
* fix(dq): enable ''Column values to be in set'' test case for boolean columns

Add BOOLEAN to supportedDataTypes array in columnValuesToBeInSet.json
to allow boolean column validation with predefined allowed values.

This enables users to enforce strict true/false validation on boolean
columns directly at the column level, resolving issue #22099.

Co-authored-by: IceS2 <IceS2@users.noreply.github.com>

* Add tests to the new feature

* Add migrations and columnValuesToBeNotInSet

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: IceS2 <IceS2@users.noreply.github.com>
2025-07-25 15:17:38 +02:00
Ayush Shah
1e8e38f2ca
MINOR: Custom properties Data types fix (#22342) 2025-07-25 18:39:53 +05:30
Mayur Singal
37b10a102f
MINOR: Improve ometa logging (#22586) 2025-07-25 18:26:44 +05:30
Mayur Singal
b8db86bc4f
MINOR: Fix airflow ingestion for older version (#22581) 2025-07-25 18:22:33 +05:30
Sriharsha Chintalapani
b0586f849f
Fix #22511: k8s secret support for Secrets Manager (#22516)
* Fix #22511: k8s secret support for Secrets Manager

* Update generated TypeScript types

* address comments

* pylint fix

* fix java checkstyle

* improve inCluster description in schema

* fix failing tests

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-07-24 12:40:51 +02:00
Mayur Singal
a94c1bef47
MINOR: Fix mysql pytest version (#22535) 2025-07-24 09:45:37 +05:30
Lukas Biermann
fb601292ad
fixes 16053: implemented timeout for airflow lineage backend (#22474)
* implemented timeout for airflow lineage backend

* Delete CLAUDE.md

* linting

* restored orginal claude code file

* reimplemented the envsioned solution to address @pmbrulls feedback

* removed unsed comment

* catched issue with empty dictionary

* fixed typing error

* adapted documentation to cover new additional optional settings for airflow backend

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-22 19:29:54 +02:00
Chirag Madlani
b098395602
Data contracts support for tables & Multi Domain Migration (#22108)
* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* rules

* rules

* rules

* fix retrieval by entity

* test dc

* test dc

* WIP: Data contract feature

* destructure component to it's own files

* WIP contract tab

* update local

* fix test

* First iteration for multi domain support

* fix inheritance fields

* fix inheritance fields

* fix create interface

* fix few more tests

* fix indexing updates

* fix domain rel

* update domain --> domains

* merge

* fix merge

* fix csv tests and createEntity interface

* Update generated TypeScript types

* Trigger Build

* migrations

* fix tests

* fix tests

* fix tests

* Update generated TypeScript types

* Trigger Build

* handle drive service

* fix pg migration

* fix domains ref after merge and clean python tests

* Update generated TypeScript types

* fix merge domains

* format

* add missing migrations

* Update generated TypeScript types

* tests

* Update generated TypeScript types

* Trigger Build

* tests

* tests

* fix py test

* migrate domain to domains and fix compilation errors

* fix domain assignement

* fix domain spec

* fix py tests

* fix data product creation issue

* fix domain tests

* fix bulk import

* fix tests

* fix tests

* fix query and domain migration

* fix py test

* fix playwrights

* fix getEntitiesWithDisplayName indexing quotes

* fix domain prapogation tests

* fix domain propagation

* Fix patch api

* fix domain schema build edit playwright

* fix test

* fix test

* fix domain selection issue and console errors

* quick fix landing page changes

* fix remaining tests

* fix ui tests

* Fix adding data products

* format

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-22 09:34:50 +02:00
Teddy
9a1fedf2a0
feat: pass GX description on test case creation (#22481) 2025-07-21 17:57:26 +02:00
Mohit Tilala
64ec471e52
Fixes #22363 #22369: Stored procedure temp table processing and lineage filtering with db & schema (#22416)
* Process temp table graph in stored procedure processor and add db/schema filtering on lineage

* Add tests for stored procedure lineage processing

* Fix tests and py_format

* Fix the filters and log stored proc query count info
2025-07-18 12:32:22 +05:30
Teddy
4a264e7a3a
feat: improve test result details for GX integration (#22424) 2025-07-17 15:41:29 +02:00
Suman Maharana
9838278ac4
Add: Schema and Database Mark Deletion (#22088)
* Added Schema and Database Mark Deletion

* removed unnecessary changes

* fixed marked deleted databases

* Added to all db connectors

* Added generated types

* Added tests
2025-07-15 16:26:46 +02:00
Suman Maharana
5b3bb637fb
Fix: Postgres query column name for exec time (#22366) 2025-07-15 12:58:53 +00:00
Suman Maharana
ea63d7e184
remove dbx debug tests (#22264) 2025-07-15 12:56:10 +02:00
Pere Miquel Brull
a86c51e82d
MINOR - Keep presidio logger at ERROR (#22124)
* MINOR - Keep presidio debug at ERROR

* test

---------

Co-authored-by: Keshav Mohta <68001229+keshavmohta09@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-14 11:55:02 +02:00
Ayush Shah
fe2caf7a5d
MINOR: Enhance patch request handling by adding 'skip_on_failure' parameter (#22142)
* Enhance patch request handling by adding 'skip_on_failure' parameter

* Introduced 'skip_on_failure' option in build_patch and OMetaPatchMixin methods to control behavior on patch operation failures.
* Updated documentation to reflect the new parameter and its default value.
* Improved error handling to log warnings instead of raising exceptions when 'skip_on_failure' is set to True.

* fix: add tests for patch request with skip on failure

* refactor: streamline mock patching and improve test readability in patch request tests

* Consolidated import statements for unittest mock.
* Enhanced readability by reducing line breaks and simplifying mock patching syntax.
* Ensured consistent use of commas in function calls for clarity.
* Updated tests to maintain functionality while improving code style.

* fix: improve error handling in patch operations

* Enhanced logging for patch operation failures in both build_patch and OMetaPatchMixin methods.
* Added detailed entity information in warning and error messages to aid in debugging.
* Ensured consistent behavior when 'skip_on_failure' is set, providing clearer feedback on operation outcomes.

* fix: clean up whitespace in patch request error handling

* Removed unnecessary whitespace in the build_patch function to improve code readability.
* Ensured consistent formatting in warning and error messages for better clarity during logging.

* fix: enhance error handling and improve test assertions in patch request

* Updated the condition for checking 'changeDescription' in the _remove_change_description function for better clarity.
* Modified exception handling in tests to raise RuntimeError instead of a generic Exception, providing more specific error feedback.
* Improved assertions in tests to check for the presence of error messages, enhancing the robustness of error handling verification.
* Adjusted test cases to reflect changes in expected patch operation counts and ensure accurate validation of patch operations.

* fix: enhance patch operation with skip_on_failure handling

* Added 'skip_on_failure' parameter to OMetaPatchMixin methods to control behavior on patch failures.
* Improved error handling to log warnings and provide detailed feedback when patch operations are skipped.
* Updated tests to verify the new behavior of skipping failures and improved assertions for clarity.
2025-07-14 12:33:17 +05:30
Sriharsha Chintalapani
58cd7daf7e
Add drive as service (#22159)
* Fix #22157: Add Drive Service Support to OpenMetadata

* Fix #22157: Add Drive Service Support to OpenMetadata

* Update generated TypeScript types

* Fix tests

* Update generated TypeScript types

* Fix tests

* Tests are passing

* Tests are passing

* Update generated TypeScript types

* Address comments

* Update generated TypeScript types

* Add sample data and ingestion for drives, directories, spreadsheets, worksheets, files

* Fix postgres migrations

* Fix postgres tests, optimize queries

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-07-11 10:00:54 -07:00
Mayur Singal
47b20a5f2d
MINOR: Fix databricks default schema issue (#22254) 2025-07-09 11:50:50 -07:00
Himanshu Khairajani
e3ad25abc9
fix: cross-platform path resolution for dbt ingestion (on Windows) (#22230)
* fix: windows os dbt ingestion fix

* py formatting

---------

Co-authored-by: --global <--global>
2025-07-08 19:02:10 -07:00
Mohit Tilala
a6c0261728
Add lineage stored procedure and view filter pattern support (#22223)
* Add lineage stored procedure and view filter pattern support

* Update generated TypeScript types

* Add tests for lineage filter pattern

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-08 16:32:25 -07:00
Mohit Tilala
ebfce5ba7b
Handle quoted entity names in masking queries (#22174)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-08 15:09:00 -07:00
Keshav Mohta
6e40f976e7
Fix #20145: Implemented Prefix For Dashboard Service (#21585)
* feat: implemented microstrategy lineage & dbServicePrefix

* feat: added dbServicePrefixes support in other dashboards

* fix: test_metabase and powerbi extra code remove

* fix: python checkstyle

* refactor: added prefix support for other connectors - superset, tableau, etc

* refactor: added migration for prefix change and fix dbServicePrefixes field description

* refactor: added prefix changes in superset db source

* doc: add prefix in tableau doc

* fix: typescript files and postgres migration for prefix

* fix: moved migration in 1.8.2

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-07-08 18:54:35 +02:00
Ethan
99486a5006
Fixes #18151 : change replaced copy by model_copy (#18153)
* feat: replace copy

* fix native python copy

---------

Co-authored-by: IceS2 <pjt1991@gmail.com>
2025-07-08 16:08:20 +02:00
IceS2
2d469e1b00
parametrize get private key for snowflake (#22214) 2025-07-08 15:47:00 +02:00
Mayur Singal
573e3bfc21
MINOR: Improve Array Sampler for UC & DBX (#22155)
* MINOR: Improve Array Sampler for UC & DBX

* make log debug

* address comments
2025-07-08 14:25:20 +02:00
Mike Kutzma
2cfabf6017
[FIX] 21910: Update amundsen ingest (#22201)
Addresses: #21910

Removed 'Either' that currently wraps the generators being yieded from
in the amundsen `metadata.py` from the `AmundsenSource` class. I think
these are not necessary and causing the issue since the objects being
yielded inside those methods (`self.create_table_entity`, etc) are all
yielding `Either` objects.

Tested on my local instance and this resolved the issue with the service
not being able to ingest.

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-07-08 17:43:55 +05:30
Mayur Singal
2fcf3281d8
MINOR: Fix snowflake map key type error (#22205) 2025-07-08 14:44:05 +05:30
harshsoni2024
718642541c
MINOR: pbi display table name from source (#22191)
* pbi display table name from source

* Update generated TypeScript types

* log correction

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-07-08 07:26:38 +02:00
harshsoni2024
1e2ab2795e
add logs for all api calls (#22179) 2025-07-08 07:25:28 +02:00
Mayur Singal
c2a3027962
MINOR: Fix pytest 3.10 (#22192) 2025-07-08 10:09:00 +05:30
Keshav Mohta
dc179c30d7
Fixes: Unity Catalog Empty Tag (#22061)
* fix: added tag_value check

* fix: checkstyle
2025-07-07 18:39:25 +02:00
IceS2
1260a0600a
MINOR: Update Snowflake Connection (#22167)
* Update Snowflake Connection

* Extracting needed methods
2025-07-07 16:38:22 +02:00
mgorsk1
3f01be6756
feat: derive downstream lineage from DBT exposures (#21992)
* 🎉 Init

* replace TARGET with EXPOSURE

* refactor and document

* add docs

* handle missing type/entity not matching

* linter

* update docs

* refactor for using label for communicating FQN as name field cannot contain special characters other than underscore. Storing dots in the name works for now but there is a deprecation warning and it will fail in the future.

* improve docs

* improve docs

* improve logging

* refactor for usage of meta.open_metadata_fqn

* linting

* update docs

* update docs

* fix docs

* 🎉 Add tests
2025-07-07 16:34:33 +02:00