859 Commits

Author SHA1 Message Date
Suman Maharana
137f86c00d
Day1 - Pipeline Usage (#19960) 2025-03-06 18:19:42 +05:30
Pere Miquel Brull
55d7e50441
MINOR - Add and remove data products Actions in Automator (#19948)
* MINOR - Add and remove Data Product assets in Automator config

* MINOR - Add and remove Data Product assets in Automator config

* domain mixin

* build ref

* build ref

* create types

* fix tests

* fix conflicts

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-03-05 07:11:17 +01:00
Imri Paran
4ada5b7bbf
Fixes 1048: Added changeSummary for tables (#19887)
* feat: add changeContext to changeDescription

* rev: unintended changes

* rev: unintended changes

* feat: implemented change summary field

- implemented ChangeSummarizer which handles change summaries based on tracked fields and list of fields changes.
- added a changeSummary field to table entity which captures change metadata like user, time and source
- added the changeSource as a field to updater. maybe breaks things in collate.

* use additional properties instead of existingJavaType otherwise types dont get created in java.

* added getUpdater(T original, T updated, Operation operation) to not break collate

* updated API docs

* - fixed null access of changeSummary
- added changeSummary to table index

* fix: PR comments

- added changeSummary as nested field in changeDescription
- removed unneeded migrations
- added changeSummary to table search index

* reverted table.json and TestUtils.java

* reverts
2025-03-03 16:35:23 +01:00
Charalampos Georgiadis
1bc8093769
Add displayName field in the createCustomProperty schema (#19943) 2025-03-03 12:24:52 +01:00
Karan Hotchandani
1cbbe9a7c0
fix generated types (#20033)
* fix generated types

* fix path

* fix unit tests
2025-02-28 15:55:40 +00:00
Gyowanny P. Queiroz
fc90a13e08
GEN-2891 - Add ingestion agent relationship to service entities (#19946)
* GEN-2891 - Add ingestion agent relationship to service entities

* Revert relationship storing

* Centralize ingestion entity relationship logic

* Add default implementation

* Change relationship type
2025-02-28 09:35:33 -03:00
Keshav Mohta
7e52310480
Refactor: Reverse Ingestion Pipeline (#19988)
* refact: removed serviceId and added service

* fix: required in reverse ingestion

* fix: types
2025-02-27 15:29:42 +01:00
Chirag Madlani
ab7666b8fe
Feat persona table page (#18550)
* support table details customization page

* update widget preview

* fix schema tab

* fix issue around save layout across tabs

* fix unit tests

* fix tab label with customized one

* misc fixes for persona edit

* update

* update topic with persona changes

* support dashboard data model

* support data assets with customization

* support pipeline with customize page

* fix console errors

* fix tests

* fix sonar cloud issues

* update code for data assets

* remove fqn prop as

* create generic tabs

* update

* support customization for domain page

* update

* fix errors

* fix tests

* update

* fix detail pages

* fix playwright tests

* fix unit tests

* fix unit tests

* update generic tab

* update

* fix tests

* fix tests

* fix data test id for tags and glossary terms

* fix version pages

* fix unit test

* update page enum with api collection

* update custom property to use generic provider

* fix api collection & endpoint

* fix apiCollection and apiEndpoint customization

* fix container test and fix sonar

* fix tests

* fix delete playwright

* support metric entity

* add metric as page type to customization

* fix container unit tests

* fix metric tests

* fix ml model customization

* fix test for mlmodel page

* fix Metric details tests

* fix tests

* fix test for database schema version page

* reset max failure for playwright

* improve icons

* address comments

* update translation

* update as per comments

* update localization

* update function signature

* update file names and fix imports

* address comments

* support tags and glossary to domain

* fix failing tests

* option to show hide task handlers
2025-02-27 15:43:58 +05:30
IceS2
7e64bfa888
MINOR: Update Data Insights Configurations to enable more fine control management (#19884)
* Update Data Insights Configurations to enable more fine control management

* Update Data Insights Configurations to enable more fine control management

* Hide one configuration from DataInsights App
2025-02-26 16:17:18 +05:30
Karan Hotchandani
9568817cf1
Fix #19950: Add Tags/Glossary Term support for Domains & Data Products (#19969)
* Fix #19950: Add Tags/Glossary Term support for Domains & Data Products

* add tags and glossary terms to domain and data product

* fix snakeyaml dependency

* Fix search filtering for domain/dataProduct

* fix showing of domain on tags page

* fix merge conflicts

* addressed comments

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2025-02-26 14:46:58 +05:30
Keshav Mohta
df193bd0c4
Feature: Reverse Metadata Ingestion (#19723) 2025-02-25 18:41:08 +05:30
Sriharsha Chintalapani
a41c14c546
Add cmd line user ops (#19959)
* Add create-user/update-password commands

* Fix #19958: Add create user option to openmetadata-ops.sh
2025-02-24 18:51:40 -08:00
IceS2
2ac8670c98
MINOR: Add day one experience app (#19883)
* Add Day One Experience App JSONSchema

* Add 'uninstall' method to applications, that should run on deleteApp

* Fix JSONSchemas

* Treating the case where we are trying to suspend or resume a workflow already in that state

* Fix JsonSchema
2025-02-24 13:53:23 +01:00
Pere Miquel Brull
9597f2ebe8
MINOR - Collate AI Tier App Config Spec (#19694)
* MINOR - Collate AI Tier App Config Spec

* MINOR - Collate AI Tier App Config Spec

* MINOR - Collate AI Tier App Config Spec

* Unify limits

* Unify limits

* Unify limits

* Unify limits

* bump version

* bump version

* map

* map
2025-02-20 14:39:05 +01:00
Sriharsha Chintalapani
a924064c09
Fix #17723: Generate Incremental Change Events even when consolidation of events applied (#19550)
* Fix #17723: Generate Incremental Change Events even when consolidation of events applied

* Fix #17723: Generate Incremental Change Events even when consolidation of events applied

* fix tests

* Fix tests

* clean policy tests

* update search methods to use incrementalChangeDescription part-1

* Fix the version page playwrights

* update search methods to use incrementalChangeDescription part-2

* introduce new field incrementalChangeDescription for search part-3

* fix mvn endpoint test

* fix followers and page search test

* fix following of assets

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2025-02-20 10:23:08 +05:30
Imri Paran
3b5218b923
MINOR: feat(event-subscriptions): event consumer extension (#19591)
* feat(event-subscriptions): event consumer extension

- implemented DI for event consumers
- added custom config for event consumers
- initialize the event scheduler along with application
- added workflow custom type

* - removed the ConsumerService as it is specific for each consumer implementation
- added a "custom" alert type

* wip

* cleanup

* reverted pom.xml changes

* reverted pom.xml changes

* reverted pom.xml changes

* removed abstract layer AbstractOmAppJobListener

* fixed import AbstractOmAppJobListener -> OmAppJobListener

* fixed constructor

* - add event subscription if it doesnt exist (set operation)
- add deleteEventSubscription

* - use patch mode when updating app.eventSubscriptions

* added docs for DIContainer

* aligned py tests with new automation schema

* added AppRunRecord.properties

* format

* changes after merge

* generated missing ts

* use openmetadata config builder instead of the config

* add json logic fixes for reverse metadata

* ref: reverted changes ro workflows

they will be part of https://github.com/open-metadata/OpenMetadata/pull/19723

* ref: reverted changes to workflows

they will be part of https://github.com/open-metadata/OpenMetadata/pull/19723

* reverted ts changes

* reverted ts changes

* reverted ts changes

* fix: initialization sequence for event subscription scheduler

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2025-02-19 16:23:46 +01:00
IceS2
9d51add739
MINOR: Add run app task (#19841)
* Add Run App Task

* Work on Run App Task

* Rollback task changes

* Fixing type convertion

* Fix Run External App wait

* Fix Run App Task

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2025-02-19 13:49:46 +01:00
IceS2
7d1ab75bc9
Update Governance Workflow jsonschemas (#19779)
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2025-02-18 14:29:45 +01:00
IceS2
46948601db
MINOR: Add create ingestion pipeline task (#19753)
* Use namespaced variables

* Fix test

* Add LOG, remove uneeded catch

* Add Log, Refactor VariableHandler

* Add Validation at Definition time

* Add migration

* Remove comments

* Add Create Ingestion Pipeline Task

* Fix migrations by not setting deployed if workflowHandler is not initialized

* Add RunIngestionPipeline Task

* Add new flowable nodes

* Add Job Lock Time configuration default - 5 mins

* fix checkstyle

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2025-02-13 13:10:17 +01:00
Gyowanny P. Queiroz
2559c79edd
MINOR - Hybrid Ingestion Agent - Link ingestion agents to ingestion pipelines (#19751)
* Hybrid Ingestion Agent - Link ingestion agents to ingestion pipelines

* Store relationship as HAS types

* Java checkstyle

* Trigger CI
2025-02-12 14:31:51 +01:00
Mayur Singal
e873ba0f1b
Fix #19201: Fix connection password masking (#19749) 2025-02-12 16:16:25 +05:30
Mayur Singal
45efd878cb
Fix #19634: use system.query.history table for lineage - databricks (#19744) 2025-02-12 13:29:07 +05:30
Mayur Singal
155571f50a
MINOR: Add support for cluster key information - bigquery (#19727) 2025-02-12 12:03:18 +05:30
Pere Miquel Brull
92c9f0754b
MINOR - Automator - Remove tags by label type (#19681)
* MINOR - Automator - Remove tags by label type

* MINOR - Automator - Remove tags by label type

* MINOR - Automator - Remove tags by label type

* MINOR - Automator - Remove tags by label type

* MINOR - Automator - Remove tags by label type

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2025-02-11 15:22:09 +01:00
IceS2
b120748656
MINOR: Use namespaced variables (#19648)
* Use namespaced variables

* Fix test

* Add LOG, remove uneeded catch

* Add Log, Refactor VariableHandler

* Add Validation at Definition time

* Add migration

* Remove comments

* Fix migrations by not setting deployed if workflowHandler is not initialized
2025-02-11 17:26:32 +05:30
Mayur Singal
29f067c162
Fix #19635: Snowflake allow custom account_usage schema (#19678) 2025-02-11 14:58:44 +05:30
Imri Paran
5f73dd717b
Fix 19499: feat(apps): add app config in run record (#19544)
* feat(apps): add app config in run record

- changed returned schema type of "status" to "AppRunRecord"!
- added a "config" field in the app record
- added a "pending" stage so that "external" app status can be mapped ti internal app status

* added note about schema compatiblity

* added app config for external apps

* show config for every app run

* add tests

* revert open change for now

* Revert "revert open change for now"

This reverts commit e1492a5c92d1967b3d47230a25df4c2c3d399de2.

* update dependency

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
2025-02-07 15:21:58 +01:00
Mayur Singal
636a83514d
Fix #12094: Add support for temp table lineage (#19654) 2025-02-04 11:40:10 +05:30
mohitdeuex
04cab8e71a Revert "- Update lineage model and api"
This reverts commit c5cce2b0c7c1be7ba654bfc2677da0d05215a895.
2025-02-03 12:11:07 +05:30
mohitdeuex
03f492ad1d Revert "- Adds updatedAt and updatedBy"
This reverts commit c6b462f0ab5905fe339c08a05ab819ca940dd06d.
2025-02-03 12:11:07 +05:30
mohitdeuex
c6b462f0ab - Adds updatedAt and updatedBy 2025-02-02 23:13:15 +05:30
mohitdeuex
c5cce2b0c7 - Update lineage model and api 2025-02-02 22:46:28 +05:30
Imri Paran
5482b94a57
feat(schedule-application): enable scheduling application using config file (#19470)
- added schedule to app private config
- reschedule a schedule application job in quartz if it exists.
2025-01-27 13:39:46 +05:30
Shrushti Polekar
d386e9ca35
Fix: Hide Incremental Metadata Extraction config for unsupported database sources (#19468)
* implement toggle for incremental extraction config in unsupported databases

* added supportsIncrementalMetadataExtraction in connection config

* update serviceData type

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2025-01-27 10:48:50 +05:30
Siddhant
473688648b
Fix #18834: Introduce new setting OpenMetadataBaseUrlConfiguration to set the server endpoint (#19366)
* Refactor: Introduce openMetadataBaseUrlConfiguration to set up the server endpoint and remove OpenMetadataUrl from smtpSettings.json

* migrations.

* refactor(cli): add setOpenMetadataUrl command to store OpenMetadata URL

* refactor(cli): add configureEmailSettings command to manage SMTP/Email configuration

* load initialData when config is not present in the db

* add om url settings page

* update locales

* add e2e

* load omBaseUrl with default host and port: localhost:8585 when not present

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-01-27 10:20:04 +05:30
Ayush Shah
e59ddecf28
Feat: Enhance Kafka SSL configuration support with consumerConfigSSL (#19433) 2025-01-27 10:15:48 +05:30
Imri Paran
a6670af500
MINOR: Use interface schemas for governance workflows (#19411)
* feat(apps): support event subscriptions

- added support for apps with event subscriptions.
- added support for custom consumers on event subscriptions.
- use native application methods instead of reflection in ApplicationHandler

* removed runMethodFromApplication reflection and use concrete methods from AbstractNativeApplication

* format

* fix(governance-workflows): defined interface in workflow schema

- defined types for workflow nodes
- fixed updateEdge method in WorkflowDefinitionRepository

* ref(governance-workflows): use explicit types in schema

use explicit interface type in json schemas and use annotations to infer the concrete classes when deserializing

* - implemented CustomSignal trigger
- use JsonTypeInfo and JsonSubTypes to infer types in jsonschema for nodes and triggers
- Implemented JsonLogicFilter task
- implemented placeholder noop task

* - simplified trigger type names
- removed governance workflow root "type"
- added "deployed" field to governance workflow
- applied changes to existing gov-workflows
- migrations

* fixed migrations
2025-01-24 09:08:06 +01:00
Aniket Katkar
9a167c7c3f
Chore: Add the missing generated types and fix json schemas (#19480)
* Add the missing generated types

* Fix the type errors

* fix the JSONs and add respective generated type files
2025-01-23 20:45:31 +05:30
Teddy
cacfabd9ce
ISSUE #1052 - Implement List entities with test suites repo logic (#19461)
* fix: centralized listWithOffset logic

* feat: dq app config + list entities with tests logic

* fix: test case
2025-01-22 08:08:07 +01:00
Siddhant
486230827d
#15600: Retention Application : Delete change_events, activity threads, versions based on admin retention policies (#18913)
* feat(retention-policy): initial setup of the Retention Policy app

* refactor: Update logic to make RetentionPolicyApp a ScheduledOrManual app

* refactor: Update retention policy JSON schema and logic to support event subscription, versions, and activity thread with individual retention periods

* refactor: Rename eventSubscriptionRetentionPeriod to changeEventRetentionPeriod

* fix: Implement activity thread retention policy cleanup

* chore: Remove activity thread and versions retention code; add banner and icon
2025-01-21 18:12:35 +05:30
Imri Paran
3abae45403
MINOR: feat(apps): support event subscriptions (#19338)
* feat(apps): support event subscriptions

- added support for apps with event subscriptions.
- added support for custom consumers on event subscriptions.
- use native application methods instead of reflection in ApplicationHandler

* removed runMethodFromApplication reflection and use concrete methods from AbstractNativeApplication

* format
2025-01-21 07:10:26 +01:00
Gyowanny P. Queiroz
cbbef9e3b9
Move PipelineServiceClient to openmetadata-spec (#19381) 2025-01-15 14:41:47 -03:00
Suman Maharana
4cd5988bc1
Fix #19313 - dbt ingestion picks up wrong service to patch metadata (#19337)
* Fix #19313 - dbt ingestion picks up wrong service to patch metadata

* py format and linting

* Added a flag for searching across databases and docs

* py format
2025-01-15 11:45:32 +05:30
Keshav Mohta
4bd8994a00
Feature: Cassandra SSL (#19226) 2025-01-10 11:25:43 +05:30
Imri Paran
7d05902945
fix: reverted deprecated fields on testSuite (#19284)
Added deprecated fields with a notice. This is to ensure migrations that use them do not break.
2025-01-08 18:34:17 -08:00
Matt Chamberlin
bf8a9e4f46
fix: Add docStore endpoints to Ometa API routes (#19172) 2025-01-08 15:47:18 +05:30
Mayur Singal
e32ad88c81
Fix #19160: Add Depth Support Part-1 (#19275) 2025-01-08 11:18:05 +05:30
sonika-shah
5ccc0c5a2a
Improvement #19065 : Support removing existing enumKeys (for enum type custom property) (#19054)
* Improvement : Support removing existing enumKeys (for enum type custom property)

* initial implementation of background job job runner for enum cleanup

* fix postgres query

* ui: allow existing enum value updates in EditCustomPropertyModal

* create BackgroundJob to use jsonSchema

* feat: add socket event handling for background job part1

* fix(i18n): remove trailing spaces in Galician language JSON file

* refactor to use jsonSchema

* feat(ui): show browser notification for custom property update

* feat(utils): add getCustomPropertyEntityPathname function and tests

* change enumCleanup trigger condition

* feat(i18n): add enum property update message to multiple language JSON files

* enhancement : improve execution flow and better exception handling with test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-07 19:25:59 -08:00
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Akash Verma
b2898f7007
Cockroach enhancement (#19108) 2025-01-07 18:51:59 +05:30