413 Commits

Author SHA1 Message Date
Pere Miquel Brull
8607f10d95
MINOR - Add Tests & fix migrations (#22714) 2025-08-03 15:19:54 +02:00
sonika-shah
d5515ded3b
Add migrations to update widget descriptions (#22707)
* Add migrations to update widget descriptions

* move to postDataMigrationSQLScript.sql

* move to postDataMigrationSQLScript.sql

* move to postDataMigrationSQLScript.sql
2025-08-02 16:55:02 +05:30
sonika-shah
4cf5a410fe
Feed migrations to support multi-domain (#22601)
* WIP : feed migrations to support multi-domain

* fix domain field in ChangeEvent and thread.json , move to maintaining list<uuid>
rather than list<entityRef>

* Update generated TypeScript types

* fix query migrations

* add default for domains field in thread and changeEvent

* add backend test , and allow domain param in createThread

* Update generated TypeScript types

* Apply suggestions from code review

handling null issue in JSON_ARRAY

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix flakiness in test, add pending migration

* fix sql query

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-02 04:59:05 +05:30
Copilot
5057f29efc
Fix alert events ordering to show latest events first (#22309)
* Initial plan

* Initial analysis: identify ordering issue in alert events

Co-authored-by: harshach <38649+harshach@users.noreply.github.com>

* Fix PostgreSQL query ordering in listAllEventsWithStatuses

Co-authored-by: harshach <38649+harshach@users.noreply.github.com>

* Add comprehensive test coverage and performance indexes for event ordering

- Add CollectionDAOEventOrderingTest with 4 comprehensive test methods
- Test chronological ordering, pagination, edge cases, and cross-database consistency
- Add critical timestamp indexes for consumers_dlq and successful_sent_change_events tables
- Indexes prevent table scans on ORDER BY timestamp DESC queries
- Migration 1.9.1 includes both MySQL and PostgreSQL index definitions

Addresses performance concerns and test coverage gaps identified in PR review.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix tests

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2025-07-31 13:31:53 +02:00
Keshav Mohta
f8b588e6f0
Feature: Security Service (#22450)
* feat: security service initial code

* support secrity type for service from ui

* refactor: added missing java files for security service and require json files

* refactor: added doc file, oneOf java changes for ranger

* Merge branch 'main' of https://github.com/open-metadata/OpenMetadata into issue-20145

* update the security services schema to have domains instead of domain

* fix: added security service metadata pipeline

* Make changes for security service UI

* Update missing translation

* update type

* Fix the java checkstyle

* fix: added ingestionRunner in security service json

* Hide security service for OSS

* fix: reverse metadata documentation

* refactor: added securityConnection in test connection

* fix: documentation and ts file changes

* fix: deleted __init__ file and added SSISConnection again in the workflowclassconverter.java

* fix: convert domain to domains

* fix: added new file for basicAuth for ranger and change in rangerConnectionClassConverter.java file

* fix: java checkstyle

* fix: mysql security service sql

* fix: security service index mapping

* fix: moved migrations in new version and fix classConverterFactory

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2025-07-31 06:38:21 +02:00
sonika-shah
79e00ebc94
Add missing migrations for changes related to searchSettings introduced in version 1.8.7. (#22634) 2025-07-29 18:49:19 +05:30
IceS2
bad772db39
FIX #22099: enable 'Column values to be in set' test case for boolean columns (#22491)
* fix(dq): enable ''Column values to be in set'' test case for boolean columns

Add BOOLEAN to supportedDataTypes array in columnValuesToBeInSet.json
to allow boolean column validation with predefined allowed values.

This enables users to enforce strict true/false validation on boolean
columns directly at the column level, resolving issue #22099.

Co-authored-by: IceS2 <IceS2@users.noreply.github.com>

* Add tests to the new feature

* Add migrations and columnValuesToBeNotInSet

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: IceS2 <IceS2@users.noreply.github.com>
2025-07-25 15:17:38 +02:00
Sriharsha Chintalapani
d1bf66052b
Fix #22422: Classification term count showing 0 in UI … (#22470)
* Fix #22422: Classification term count showing 0 in UI                                                                                                                   │
│                                                                                                                                                                                          │
│   - Added bulk fetch methods to ClassificationDAO for efficient term count retrieval                                                                                                     │
│   - Implemented proper batch fetching in ClassificationRepository.setFieldsInBulk                                                                                                        │
│   - Added ConnectionAwareSqlQuery annotations for MySQL and PostgreSQL compatibility                                                                                                     │
│   - Fixed duplicate key error in TagRepository.batchFetchUsageCounts                                                                                                                     │
│   - Added Playwright test to verify classification term counts display correctly

* Fix java checkstyle

* remove unused query

* refactor query :  switch to join-based classification hash matching and add generated column for classificationHash

---------

Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2025-07-24 11:38:49 +05:30
Sriharsha Chintalapani
e3ba945630
Update databaseServiceMetadataPipeline.json (#22518)
* Update databaseServiceMetadataPipeline.json

markDeletedSchemas and markDeletedDatabases should be false by default

* add migrations

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2025-07-23 11:14:18 +02:00
Chirag Madlani
b098395602
Data contracts support for tables & Multi Domain Migration (#22108)
* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* WIP - MINOR - Rule Engine

* rules

* rules

* rules

* fix retrieval by entity

* test dc

* test dc

* WIP: Data contract feature

* destructure component to it's own files

* WIP contract tab

* update local

* fix test

* First iteration for multi domain support

* fix inheritance fields

* fix inheritance fields

* fix create interface

* fix few more tests

* fix indexing updates

* fix domain rel

* update domain --> domains

* merge

* fix merge

* fix csv tests and createEntity interface

* Update generated TypeScript types

* Trigger Build

* migrations

* fix tests

* fix tests

* fix tests

* Update generated TypeScript types

* Trigger Build

* handle drive service

* fix pg migration

* fix domains ref after merge and clean python tests

* Update generated TypeScript types

* fix merge domains

* format

* add missing migrations

* Update generated TypeScript types

* tests

* Update generated TypeScript types

* Trigger Build

* tests

* tests

* fix py test

* migrate domain to domains and fix compilation errors

* fix domain assignement

* fix domain spec

* fix py tests

* fix data product creation issue

* fix domain tests

* fix bulk import

* fix tests

* fix tests

* fix query and domain migration

* fix py test

* fix playwrights

* fix getEntitiesWithDisplayName indexing quotes

* fix domain prapogation tests

* fix domain propagation

* Fix patch api

* fix domain schema build edit playwright

* fix test

* fix test

* fix domain selection issue and console errors

* quick fix landing page changes

* fix remaining tests

* fix ui tests

* Fix adding data products

* format

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-07-22 09:34:50 +02:00
sonika-shah
4d473c8204
Fix 1.7.3 pending migrations (#22434)
* Fix 1.7.3 pending migrations

* move to 1.8.7

* move to 1.8.7
2025-07-21 20:58:11 -07:00
Sriharsha Chintalapani
e3e248ef0a
Fix #22440: Performance Optimization for Database Service Cascade Deletion (#22441)
* Fix #22440: Performance Optimization for Database Service Cascade Deletion

* Fix stylecheck

* Fix #22440: Performance Optimization for Database Service Cascade Deletion

* Fix stylecheck

* Fix styling
2025-07-20 07:32:48 -07:00
Sriharsha Chintalapani
1c092ea743
Fix #22403: Smart Reindex (#22404)
* Fix #22403: Smart Reindex

* Update generated TypeScript types

* Docs: Artifacts Removal (#22442)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* Docs: Meta Tags Updation (#22444)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* Fixes #22363 #22369: Stored procedure temp table processing and lineage filtering with db & schema (#22416)

* Process temp table graph in stored procedure processor and add db/schema filtering on lineage

* Add tests for stored procedure lineage processing

* Fix tests and py_format

* Fix the filters and log stored proc query count info

* Playwright Update: Bulk Import (#22436)

Co-authored-by: Satish <satish@Satishs-MacBook-Pro.local>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>

* Docs: Databrick Features Updation (#22448)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* Docs: Meta Updation (#22451)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>

* Docs: 18.4 to 1.8.6 Release Updation (#22455)

Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>

* fix the flaky glossary bulk edit failure (#22446)

* Move schema changes to 1.8.7

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Rounak Dhillon <162090200+RounakDhillon@users.noreply.github.com>
Co-authored-by: “Rounak <“rounakpreet.d@deuexsolutions.com”>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
Co-authored-by: satish <satishmwaghela@gmail.com>
Co-authored-by: Satish <satish@Satishs-MacBook-Pro.local>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2025-07-18 21:53:43 -07:00
Mohit Yadav
060df56aaf
Fix Entity Relationship Cleanup to not deleted entities without repo… (#22400)
* - Fix Entity Relationship Cleanup to not delted entities without repositories

* - Add Migration from Test Case to Test Case Resolution

* - Typo should be resolution id

* Add Tests and Fix Condition

* Update BatchSize
2025-07-16 14:26:38 -07:00
Pere Miquel Brull
397c41457c
MINOR - Add columns.description in search settings (#22299)
* MINOR - Add columns.description in search settings

* remove from defaults
2025-07-15 09:21:57 +02:00
Sriharsha Chintalapani
58cd7daf7e
Add drive as service (#22159)
* Fix #22157: Add Drive Service Support to OpenMetadata

* Fix #22157: Add Drive Service Support to OpenMetadata

* Update generated TypeScript types

* Fix tests

* Update generated TypeScript types

* Fix tests

* Tests are passing

* Tests are passing

* Update generated TypeScript types

* Address comments

* Update generated TypeScript types

* Add sample data and ingestion for drives, directories, spreadsheets, worksheets, files

* Fix postgres migrations

* Fix postgres tests, optimize queries

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-07-11 10:00:54 -07:00
Keshav Mohta
6e40f976e7
Fix #20145: Implemented Prefix For Dashboard Service (#21585)
* feat: implemented microstrategy lineage & dbServicePrefix

* feat: added dbServicePrefixes support in other dashboards

* fix: test_metabase and powerbi extra code remove

* fix: python checkstyle

* refactor: added prefix support for other connectors - superset, tableau, etc

* refactor: added migration for prefix change and fix dbServicePrefixes field description

* refactor: added prefix changes in superset db source

* doc: add prefix in tableau doc

* fix: typescript files and postgres migration for prefix

* fix: moved migration in 1.8.2

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-07-08 18:54:35 +02:00
Sriharsha Chintalapani
02a5a5fcc2
Bulk load fields for paginations and indexing (#21771)
* Add db max connections to used while auto tune search index app

* Add more logging

* Add batch reading of entities

* Fix bulk loading issues

* Fix bulk loading issues

* Fix bulk loading issues

* Increase the default max concurrent requests

* Remove entitynotfound from errors

* Add higher no.of threads

* Fix KPI, TestCase, Policy repository fixes

* Fix Merge Issues

* Remove column tests

* Add bulk load for Tags

* Fix tests

* Fix tests for TestSuiteRepository

* Improve bulk loading in Table, DatabaseSchema

* Improve bulk loading in all entities

* fix merge error

* Improve indexing for entities

* Improve performance while pagination

* fix tests

* fix tests

* fix tests

* fix markdown test

* Address issues with bulkloading owners,inherited owners in database schema, tables, roles

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2025-07-07 07:46:42 -07:00
Sriharsha Chintalapani
0162f42a1d
Fix #20621: User Status Tracking in the System (#21911)
* Fix #20621: User Status Tracking in the System

* Fix #20621: User Status Tracking in the System

* Fix #20621: Add Playwright tests

* Update generated TypeScript types

* fix import

* remove unused imports and fix broken ui for settings user page

* revert path  change for PersonaDetailsPage

* fix unit tests

* fix tests

* Add separate api for last login time

* update last login time api

* remove last login time patch

* remove updateLoginTime

---------

Co-authored-by: shrushti2000 <shrushtipolekar@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2025-07-02 14:37:36 -07:00
Sriharsha Chintalapani
db8e2fa548
Improve indexing auto tune based on heap mem, parse custer metrics for new versions of OpenSearch (#21931)
* Improve indexing auto tune based on heap mem, parse custer metrics for new versions of OpenSearch

* Use native bulk processors

* lower the batch size, add composite index for app_runs table

* while updated secrets, if there is entityNotFoundException ignore and move on

* Fix deadlock

* Fix deadlock

* Fix gzip headers

* remove unnecessary code to check compression, batch sizes etc..

* Improved performance

* Improved performance

* fix merge error

* fix styling

* fix tests

* Fix build issues

* improve queue size recommendations

* put max limits on threads

* fix checkstyle
2025-06-30 07:40:25 -07:00
Mohit Yadav
5b6e25999e
Disabled bot creating activity feeds (#21773)
* - Disabled bot creating feeds

* - nit
2025-06-14 19:21:00 +05:30
Mohit Yadav
6124c5573e
fix tests (#21751) 2025-06-12 20:47:41 -07:00
Mohit Yadav
4309f79453
Fix Reset Link Template (#21747)
* Fix Reset Link Template

* 1.7.3
2025-06-13 08:57:18 +05:30
Sriharsha Chintalapani
18bd4689a0
Add Data Contracts Specification and APIs (#21164)
* Add Data Contracts Specification and APIs

* Fix checkstyle

* Add @Repository

* Add @Repository

* Address YAML APIs

* Fix tests

* Fix tests

* Address comments

* Fix tests for checking table/topic fields do exists for a data contract

* migrate to jakarta

* migrate to jakarta

* add semantics

---------

Co-authored-by: Harsha <harsha@Harshas-MacBook-Air.local>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-06-04 06:36:28 +02:00
Pere Miquel Brull
e58802b659
FIX - Automation Workflows should not be updated by the SM & cleanup migration (#21435)
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2025-06-03 12:17:14 +02:00
Imri Paran
baee931b85
MINOR: App changes for collate support (#21068)
* feat: prep for collate support

- Added support for delayed background jobs. This is to delete the support user after a while.
- added ScheduleType.OnlyManual for applications to support only manual triggers
- Added a four hour token expiration
- Allow a null trigger type in job data (not sure what it's for)

* added test for delayed job

* added type checking for enumCleanupArgs in navbar

* - added a support toten type
- added job handler for delete token

* fixed whitelist links

* patch OnlyManual apps

* Update openmetadata-spec/src/main/resources/json/schema/entity/applications/app.json

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* added indexes for runAt

* use NO_JOB_SLEEP_SECONDS in background job test

* Fixed NavBar test

* removed backticls

* fix indexes

* change for support app

* fix tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-05-26 12:02:14 +02:00
Mohit Yadav
1ee36d41c6
Escape ? to causing issues in jdbi binding (#21381) 2025-05-23 17:13:45 +05:30
Mohit Yadav
94ceb7792a
Failing Postgres sql (#21378) 2025-05-23 16:39:56 +05:30
IceS2
6f4204731f
MINOR: Fix app marketplace migrations (#21360)
* Fix app marketplace migrations

* Fix test
2025-05-22 15:13:00 +02:00
IceS2
04bfb7b85d
Improve Workflow Logs and Reduce Concurrency (#21345) 2025-05-21 16:22:28 +02:00
IceS2
b761efbfd3
MINOR: Add missing workflow config status migrations (#21286)
* Add missing workflow config status migrations

* Moved to Java based migrations to redeploy pipeline

* Moved migrations to Java. Updated Listener based on config

* Improved update on the migrations
2025-05-20 08:02:02 +02:00
Imri Paran
07a6530b5b
fix: move autopilot runtime migrations to 1.7.1 (#21267) 2025-05-19 08:54:57 +00:00
Suman Maharana
f81ee52ec4
Chore Ingestion Tableau library change (#21076) 2025-05-15 17:48:39 +05:30
Imri Paran
112dc4be1f
MINOR fix: autopilot marketplace runtime (#21134) 2025-05-13 15:45:50 +05:30
Ayush Shah
49de0119cf
fix: Migrations - add operator parameter to test_definition for custom SQL queries in MySQL and Postgres (#21044) 2025-04-30 14:57:20 +05:30
Mohit Yadav
e0eac43636
Add cleanup apps_extension_time_series (#20857) 2025-04-16 14:54:11 +05:30
Teddy
98c6b6e4ef
migration: fix duplicate param key insertion (#20802) 2025-04-15 14:10:51 +02:00
Keshav Mohta
0796c6274b
Fixes: Databricks httpPath Required (#20611)
* fix: made databricks httpPath required and added a migration file for the same

* fix: added sql migration in postDataMigration file and fix databricks tests

* fix: added httpPath in test_source_connection.py and test_source_parsing.py files

* fix: added httpPath in test_databricks_lineage.py

* fix: table name in postgres migration
2025-04-07 13:33:55 +05:30
IceS2
c9182016db
MINOR: Update DayOneWorkflow Agent Names (#20473)
* Update DayOneWorkflow Agent Names

* Migrations for CollateAI DisplayName Update

* Remove wrong migrations
2025-03-28 14:01:42 +01:00
IceS2
deca9f9574
Fix Workflow State not Updating on Exception (#20395) 2025-03-24 09:26:05 -07:00
Teddy
5a3fc2cfa4
feat: add created by for test case entities (#20304)
* feat: add created by for test case entities

* migration: createdBy sql migration
2025-03-24 13:13:01 +01:00
Imri Paran
451506cb08
MINOR: chore: added missing timestamp indexes for time series tables (#20373)
* chore: added index for apps_extension_time_series.timestamp

* - added timestamp index for workflow_instance_time_series
- added timestamp index for workflow_instance_state_time_series
2025-03-24 07:43:07 +01:00
Imri Paran
277e602482
fix: workflow definition migrations (#20357) 2025-03-20 14:28:23 +01:00
IceS2
ce497e020e
MINOR: Expose Workflow Stage Status and Bring OneDayApp to OSS (#20123)
* Expose Workflow Stage Status and Bring OneDayApp to OSS

* Fix Migrations

* Change Bot and add comment

* Fix checkstyle
2025-03-10 09:25:32 +01:00
Mayur Singal
46947277c4
MINOR: Fix cost analysis build failing for postgres (#20122) 2025-03-06 21:30:28 +00:00
Mayur Singal
c1592b54c5
Capture Query Cost (#19991) 2025-03-06 19:49:59 +05:30
IceS2
fd7b00d246
MINOR: Fix Migrations (#19925)
* Fix Migrations

* Fix WorkflowHandler initialization

* Refactor how to fix the migrations by passing the Config object itself

* Fix Migrations

* Remove comments

* Remove comments
2025-02-26 12:53:08 +05:30
Mohit Yadav
cb7942f4b6
Cleanup App data (#19571) 2025-01-28 19:22:33 +05:30
Imri Paran
76c9b5a0ae
fix(migrations): use double question-mark (#19557)
Use double question mark to escape parametrization by jdbi.
https://jdbc.postgresql.org/documentation/query/#using-the-statement-or-preparedstatement-interface
2025-01-28 18:07:23 +05:30
Mohit Yadav
58d5988a16
Add Onboarding SQL (#19525)
* Add Primary Key Migration

* Update queries and conflict on apps data store

* Update queries and conflict on apps data store

* add index on isBot
2025-01-28 11:45:58 +05:30