426 Commits

Author SHA1 Message Date
Chirag Madlani
b098395602
Data contracts support for tables & Multi Domain Migration (#22108)
* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* rules

* rules

* rules

* fix retrieval by entity

* test dc

* test dc

* WIP: Data contract feature

* destructure component to it's own files

* WIP contract tab

* update local

* fix test

* First iteration for multi domain support

* fix inheritance fields

* fix inheritance fields

* fix create interface

* fix few more tests

* fix indexing updates

* fix domain rel

* update domain --> domains

* merge

* fix merge

* fix csv tests and createEntity interface

* Update generated TypeScript types

* Trigger Build

* migrations

* fix tests

* fix tests

* fix tests

* Update generated TypeScript types

* Trigger Build

* handle drive service

* fix pg migration

* fix domains ref after merge and clean python tests

* Update generated TypeScript types

* fix merge domains

* format

* add missing migrations

* Update generated TypeScript types

* tests

* Update generated TypeScript types

* Trigger Build

* tests

* tests

* fix py test

* migrate domain to domains and fix compilation errors

* fix domain assignement

* fix domain spec

* fix py tests

* fix data product creation issue

* fix domain tests

* fix bulk import

* fix tests

* fix tests

* fix query and domain migration

* fix py test

* fix playwrights

* fix getEntitiesWithDisplayName indexing quotes

* fix domain prapogation tests

* fix domain propagation

* Fix patch api

* fix domain schema build edit playwright

* fix test

* fix test

* fix domain selection issue and console errors

* quick fix landing page changes

* fix remaining tests

* fix ui tests

* Fix adding data products

* format

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-22 09:34:50 +02:00
sonika-shah
4d473c8204
Fix 1.7.3 pending migrations (#22434)
* Fix 1.7.3 pending migrations

* move to 1.8.7

* move to 1.8.7
2025-07-21 20:58:11 -07:00
Sriharsha Chintalapani
e3e248ef0a
Fix #22440: Performance Optimization for Database Service Cascade Deletion (#22441)
* Fix #22440: Performance Optimization for Database Service Cascade Deletion

* Fix stylecheck

* Fix #22440: Performance Optimization for Database Service Cascade Deletion

* Fix stylecheck

* Fix styling
2025-07-20 07:32:48 -07:00
Sriharsha Chintalapani
1c092ea743
Fix #22403: Smart Reindex (#22404)
* Fix #22403: Smart Reindex

* Update generated TypeScript types

* Docs: Artifacts Removal (#22442)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* Docs: Meta Tags Updation (#22444)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* Fixes #22363 #22369: Stored procedure temp table processing and lineage filtering with db & schema (#22416)

* Process temp table graph in stored procedure processor and add db/schema filtering on lineage

* Add tests for stored procedure lineage processing

* Fix tests and py_format

* Fix the filters and log stored proc query count info

* Playwright Update: Bulk Import (#22436)

Co-authored-by: Satish <satish@Satishs-MacBook-Pro.local>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>

* Docs: Databrick Features Updation (#22448)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* Docs: Meta Updation (#22451)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>

* Docs: 18.4 to 1.8.6 Release Updation (#22455)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* fix the flaky glossary bulk edit failure (#22446)

* Move schema changes to 1.8.7

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Rounak Dhillon <162090200+RounakDhillon@users.noreply.github.com>
Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
Co-authored-by: satish <satishmwaghela@gmail.com>
Co-authored-by: Satish <satish@Satishs-MacBook-Pro.local>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2025-07-18 21:53:43 -07:00
Mohit Yadav
060df56aaf
Fix Entity Relationship Cleanup to not deleted entities without repo… (#22400)
* - Fix Entity Relationship Cleanup to not delted entities without repositories

* - Add Migration from Test Case to Test Case Resolution

* - Typo should be resolution id

* Add Tests and Fix Condition

* Update BatchSize
2025-07-16 14:26:38 -07:00
Pere Miquel Brull
397c41457c
MINOR - Add columns.description in search settings (#22299)
* MINOR - Add columns.description in search settings

* remove from defaults
2025-07-15 09:21:57 +02:00
Sriharsha Chintalapani
58cd7daf7e
Add drive as service (#22159)
* Fix #22157: Add Drive Service Support to OpenMetadata

* Fix #22157: Add Drive Service Support to OpenMetadata

* Update generated TypeScript types

* Fix tests

* Update generated TypeScript types

* Fix tests

* Tests are passing

* Tests are passing

* Update generated TypeScript types

* Address comments

* Update generated TypeScript types

* Add sample data and ingestion for drives, directories, spreadsheets, worksheets, files

* Fix postgres migrations

* Fix postgres tests, optimize queries

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-07-11 10:00:54 -07:00
Keshav Mohta
6e40f976e7
Fix #20145: Implemented Prefix For Dashboard Service (#21585)
* feat: implemented microstrategy lineage & dbServicePrefix

* feat: added dbServicePrefixes support in other dashboards

* fix: test_metabase and powerbi extra code remove

* fix: python checkstyle

* refactor: added prefix support for other connectors - superset, tableau, etc

* refactor: added migration for prefix change and fix dbServicePrefixes field description

* refactor: added prefix changes in superset db source

* doc: add prefix in tableau doc

* fix: typescript files and postgres migration for prefix

* fix: moved migration in 1.8.2

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-07-08 18:54:35 +02:00
Sriharsha Chintalapani
02a5a5fcc2
Bulk load fields for paginations and indexing (#21771)
* Add db max connections to used while auto tune search index app

* Add more logging

* Add batch reading of entities

* Fix bulk loading issues

* Fix bulk loading issues

* Fix bulk loading issues

* Increase the default max concurrent requests

* Remove entitynotfound from errors

* Add higher no.of threads

* Fix KPI, TestCase, Policy repository fixes

* Fix Merge Issues

* Remove column tests

* Add bulk load for Tags

* Fix tests

* Fix tests for TestSuiteRepository

* Improve bulk loading in Table, DatabaseSchema

* Improve bulk loading in all entities

* fix merge error

* Improve indexing for entities

* Improve performance while pagination

* fix tests

* fix tests

* fix tests

* fix markdown test

* Address issues with bulkloading owners,inherited owners in database schema, tables, roles

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-07-07 07:46:42 -07:00
Sriharsha Chintalapani
0162f42a1d
Fix #20621: User Status Tracking in the System (#21911)
* Fix #20621: User Status Tracking in the System

* Fix #20621: User Status Tracking in the System

* Fix #20621: Add Playwright tests

* Update generated TypeScript types

* fix import

* remove unused imports and fix broken ui for settings user page

* revert path  change for PersonaDetailsPage

* fix unit tests

* fix tests

* Add separate api for last login time

* update last login time api

* remove last login time patch

* remove updateLoginTime

---------

Co-authored-by: shrushti2000 <shrushtipolekar@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2025-07-02 14:37:36 -07:00
Sriharsha Chintalapani
db8e2fa548
Improve indexing auto tune based on heap mem, parse custer metrics for new versions of OpenSearch (#21931)
* Improve indexing auto tune based on heap mem, parse custer metrics for new versions of OpenSearch

* Use native bulk processors

* lower the batch size, add composite index for app_runs table

* while updated secrets, if there is entityNotFoundException ignore and move on

* Fix deadlock

* Fix deadlock

* Fix gzip headers

* remove unnecessary code to check compression, batch sizes etc..

* Improved performance

* Improved performance

* fix merge error

* fix styling

* fix tests

* Fix build issues

* improve queue size recommendations

* put max limits on threads

* fix checkstyle
2025-06-30 07:40:25 -07:00
Mohit Yadav
5b6e25999e
Disabled bot creating activity feeds (#21773)
* - Disabled bot creating feeds

* - nit
2025-06-14 19:21:00 +05:30
Mohit Yadav
6124c5573e
fix tests (#21751) 2025-06-12 20:47:41 -07:00
Mohit Yadav
4309f79453
Fix Reset Link Template (#21747)
* Fix Reset Link Template

* 1.7.3
2025-06-13 08:57:18 +05:30
Sriharsha Chintalapani
18bd4689a0
Add Data Contracts Specification and APIs (#21164)
* Add Data Contracts Specification and APIs

* Fix checkstyle

* Add @Repository

* Add @Repository

* Address YAML APIs

* Fix tests

* Fix tests

* Address comments

* Fix tests for checking table/topic fields do exists for a data contract

* migrate to jakarta

* migrate to jakarta

* add semantics

---------

Co-authored-by: Harsha <harsha@Harshas-MacBook-Air.local>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-06-04 06:36:28 +02:00
Pere Miquel Brull
e58802b659
FIX - Automation Workflows should not be updated by the SM & cleanup migration (#21435)
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2025-06-03 12:17:14 +02:00
Imri Paran
baee931b85
MINOR: App changes for collate support (#21068)
* feat: prep for collate support

- Added support for delayed background jobs. This is to delete the support user after a while.
- added ScheduleType.OnlyManual for applications to support only manual triggers
- Added a four hour token expiration
- Allow a null trigger type in job data (not sure what it's for)

* added test for delayed job

* added type checking for enumCleanupArgs in navbar

* - added a support toten type
- added job handler for delete token

* fixed whitelist links

* patch OnlyManual apps

* Update openmetadata-spec/src/main/resources/json/schema/entity/applications/app.json

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* added indexes for runAt

* use NO_JOB_SLEEP_SECONDS in background job test

* Fixed NavBar test

* removed backticls

* fix indexes

* change for support app

* fix tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-05-26 12:02:14 +02:00
Mohit Yadav
1ee36d41c6
Escape ? to causing issues in jdbi binding (#21381) 2025-05-23 17:13:45 +05:30
Mohit Yadav
94ceb7792a
Failing Postgres sql (#21378) 2025-05-23 16:39:56 +05:30
IceS2
6f4204731f
MINOR: Fix app marketplace migrations (#21360)
* Fix app marketplace migrations

* Fix test
2025-05-22 15:13:00 +02:00
IceS2
04bfb7b85d
Improve Workflow Logs and Reduce Concurrency (#21345) 2025-05-21 16:22:28 +02:00
IceS2
b761efbfd3
MINOR: Add missing workflow config status migrations (#21286)
* Add missing workflow config status migrations

* Moved to Java based migrations to redeploy pipeline

* Moved migrations to Java. Updated Listener based on config

* Improved update on the migrations
2025-05-20 08:02:02 +02:00
Imri Paran
07a6530b5b
fix: move autopilot runtime migrations to 1.7.1 (#21267) 2025-05-19 08:54:57 +00:00
Suman Maharana
f81ee52ec4
Chore Ingestion Tableau library change (#21076) 2025-05-15 17:48:39 +05:30
Imri Paran
112dc4be1f
MINOR fix: autopilot marketplace runtime (#21134) 2025-05-13 15:45:50 +05:30
Ayush Shah
49de0119cf
fix: Migrations - add operator parameter to test_definition for custom SQL queries in MySQL and Postgres (#21044) 2025-04-30 14:57:20 +05:30
Mohit Yadav
e0eac43636
Add cleanup apps_extension_time_series (#20857) 2025-04-16 14:54:11 +05:30
Pere Miquel Brull
97b9b75fce
Update openmetadata-ops.sh for collate-spec classpath (#20855)
This fixes an issue we saw in AUTs and sandbox-beta:

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final long java.time.Instant.seconds accessible: module java.base does not "opens java.time" to unnamed module @44075f31
The RCA of this issue is an overriding we are doing on some internal argo lib classes in io.argoproj.workflow.JSON.

This was previously placed in collate-service. In the recent move to collate-spec when preparing the Hybrid Pipeline Service Client, we missed moving this. Then, the classpath was first loading the JSON class from the argo lib, rather than our fixed class inside collate-spec.

This is tricky since running the server from intellij works properly. However, the issue happens when running from openmetadata-server-start.sh, which is the dep used within Docker too.
2025-04-16 10:32:31 +02:00
Teddy
98c6b6e4ef
migration: fix duplicate param key insertion (#20802) 2025-04-15 14:10:51 +02:00
Keshav Mohta
0796c6274b
Fixes: Databricks httpPath Required (#20611)
* fix: made databricks httpPath required and added a migration file for the same

* fix: added sql migration in postDataMigration file and fix databricks tests

* fix: added httpPath in test_source_connection.py and test_source_parsing.py files

* fix: added httpPath in test_databricks_lineage.py

* fix: table name in postgres migration
2025-04-07 13:33:55 +05:30
IceS2
c9182016db
MINOR: Update DayOneWorkflow Agent Names (#20473)
* Update DayOneWorkflow Agent Names

* Migrations for CollateAI DisplayName Update

* Remove wrong migrations
2025-03-28 14:01:42 +01:00
IceS2
deca9f9574
Fix Workflow State not Updating on Exception (#20395) 2025-03-24 09:26:05 -07:00
Teddy
5a3fc2cfa4
feat: add created by for test case entities (#20304)
* feat: add created by for test case entities

* migration: createdBy sql migration
2025-03-24 13:13:01 +01:00
Imri Paran
451506cb08
MINOR: chore: added missing timestamp indexes for time series tables (#20373)
* chore: added index for apps_extension_time_series.timestamp

* - added timestamp index for workflow_instance_time_series
- added timestamp index for workflow_instance_state_time_series
2025-03-24 07:43:07 +01:00
Imri Paran
277e602482
fix: workflow definition migrations (#20357) 2025-03-20 14:28:23 +01:00
IceS2
ce497e020e
MINOR: Expose Workflow Stage Status and Bring OneDayApp to OSS (#20123)
* Expose Workflow Stage Status and Bring OneDayApp to OSS

* Fix Migrations

* Change Bot and add comment

* Fix checkstyle
2025-03-10 09:25:32 +01:00
Mayur Singal
46947277c4
MINOR: Fix cost analysis build failing for postgres (#20122) 2025-03-06 21:30:28 +00:00
Mayur Singal
c1592b54c5
Capture Query Cost (#19991) 2025-03-06 19:49:59 +05:30
IceS2
fd7b00d246
MINOR: Fix Migrations (#19925)
* Fix Migrations

* Fix WorkflowHandler initialization

* Refactor how to fix the migrations by passing the Config object itself

* Fix Migrations

* Remove comments

* Remove comments
2025-02-26 12:53:08 +05:30
Mohit Yadav
cb7942f4b6
Cleanup App data (#19571) 2025-01-28 19:22:33 +05:30
Imri Paran
76c9b5a0ae
fix(migrations): use double question-mark (#19557)
Use double question mark to escape parametrization by jdbi.
https://jdbc.postgresql.org/documentation/query/#using-the-statement-or-preparedstatement-interface
2025-01-28 18:07:23 +05:30
Mohit Yadav
58d5988a16
Add Onboarding SQL (#19525)
* Add Primary Key Migration

* Update queries and conflict on apps data store

* Update queries and conflict on apps data store

* add index on isBot
2025-01-28 11:45:58 +05:30
Imri Paran
5fb7abb820
MINOR: fix syntax in migration (#19540) 2025-01-27 18:03:54 +00:00
Imri Paran
a0fcf67a4e
fix(governance-workflows): fix migrations for 1.7.0 (#19533)
eventBasedEntityWorkflow -> eventBasedEntity
periodicBatchEntityWorkflow -> periodicBatchEntity
2025-01-27 14:29:15 +01:00
Siddhant
473688648b
Fix #18834: Introduce new setting OpenMetadataBaseUrlConfiguration to set the server endpoint (#19366)
* Refactor: Introduce openMetadataBaseUrlConfiguration to set up the server endpoint and remove OpenMetadataUrl from smtpSettings.json

* migrations.

* refactor(cli): add setOpenMetadataUrl command to store OpenMetadata URL

* refactor(cli): add configureEmailSettings command to manage SMTP/Email configuration

* load initialData when config is not present in the db

* add om url settings page

* update locales

* add e2e

* load omBaseUrl with default host and port: localhost:8585 when not present

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-01-27 10:20:04 +05:30
sonika-shah
3267b71217
Fix postgres query added in pr #19411 (#19512) 2025-01-24 16:56:07 +05:30
Imri Paran
a6670af500
MINOR: Use interface schemas for governance workflows (#19411)
* feat(apps): support event subscriptions

- added support for apps with event subscriptions.
- added support for custom consumers on event subscriptions.
- use native application methods instead of reflection in ApplicationHandler

* removed runMethodFromApplication reflection and use concrete methods from AbstractNativeApplication

* format

* fix(governance-workflows): defined interface in workflow schema

- defined types for workflow nodes
- fixed updateEdge method in WorkflowDefinitionRepository

* ref(governance-workflows): use explicit types in schema

use explicit interface type in json schemas and use annotations to infer the concrete classes when deserializing

* - implemented CustomSignal trigger
- use JsonTypeInfo and JsonSubTypes to infer types in jsonschema for nodes and triggers
- Implemented JsonLogicFilter task
- implemented placeholder noop task

* - simplified trigger type names
- removed governance workflow root "type"
- added "deployed" field to governance workflow
- applied changes to existing gov-workflows
- migrations

* fixed migrations
2025-01-24 09:08:06 +01:00
sonika-shah
5ccc0c5a2a
Improvement #19065 : Support removing existing enumKeys (for enum type custom property) (#19054)
* Improvement : Support removing existing enumKeys (for enum type custom property)

* initial implementation of background job job runner for enum cleanup

* fix postgres query

* ui: allow existing enum value updates in EditCustomPropertyModal

* create BackgroundJob to use jsonSchema

* feat: add socket event handling for background job part1

* fix(i18n): remove trailing spaces in Galician language JSON file

* refactor to use jsonSchema

* feat(ui): show browser notification for custom property update

* feat(utils): add getCustomPropertyEntityPathname function and tests

* change enumCleanup trigger condition

* feat(i18n): add enum property update message to multiple language JSON files

* enhancement : improve execution flow and better exception handling with test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-07 19:25:59 -08:00
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Teddy
bd68d959f3
fix: added index on timestamp field for reindex (#19101) 2024-12-16 20:57:42 +01:00