163 Commits

Author SHA1 Message Date
Adrià Manero
840f7e2a68
Setup NotificationTemplate CRUD (#23293) 2025-09-16 15:21:11 +02:00
Sriharsha Chintalapani
cf7931ee3b
Add logging endpoint into S3 (#22533)
* Add logging endpoint into S3

* Update generated TypeScript types

* Stream Ingestion logs to S3

* Update generated TypeScript types

* Address comments

* Update generated TypeScript types

* create logs mixin, use clients to stream logs

* centralize logs sending into mixin

* use StreamableLogHandlerManager instead global handler

* improve condition

* remove example workflow file

* formatting changes

* fix tests and format

* tests, checkstyle fix

* minor changes

* reformat code

* tests fix

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-09-15 07:22:25 -07:00
Bhanu Agrawal
55d9054d69
fix: resolve critical security vulnerabilities (Netty, Spring, commons-lang) (#23324)
* fix: upgrade Spring Framework to 6.1.15 for security vulnerability

  - Upgrades spring-beans, spring-core, spring-expression from 6.1.14 to 6.1.15
  - Addresses security vulnerability detected by Snyk
  - Patch release with no breaking changes

* fix: resolve critical security vulnerabilities in dependencies
2025-09-15 17:02:59 +05:30
Mohit Yadav
c0d7a574d7
chore(release): Prepare Branch for 1.10.0-SNAPSHOT (#23034)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2025-08-21 21:43:01 +05:30
Sriharsha Chintalapani
a6d544a5d8
RDF Ontology, Json LD, DCAT vocabulary support by mapping OM Schemas to RDF (#22852)
* Support for RDF, SPARQL, SQL-TO-SPARQL

* Tests are working

* Add  RDF relations tests

* improve Knowledge Graph UI, tags , glossary term relations

* Lang translations

* Fix level depth querying

* Add semantic search interfaces , integration into search

* cleanup

* Update generated TypeScript types

* Fix styling

* remove duplicated ttl file

* model generator cleanup

* Update OM - DCAT vocab

* Update DataProduct Schema

* Improve JsonLD Translator

* Update generated TypeScript types

* Fix Tests

* Fix java checkstyle

* Add RDF workflows

* fix unit tests

* fix e2e

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2025-08-17 18:36:26 -07:00
Mohit Yadav
30b5634522
Fix Vulnerabilities (#22813) 2025-08-07 13:59:24 +05:30
Mohit Yadav
b92e9d0e06
chore(release): Prepare Branch for 1.9.0-SNAPSHOT (#22742)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2025-08-04 20:00:25 +05:30
dependabot[bot]
0a2a470c50
Chore(deps): Bump com.nimbusds:nimbus-jose-jwt in /openmetadata-service (#22310)
Bumps [com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.42 to 10.0.2.
- [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/10.0.2..9.42)

---
updated-dependencies:
- dependency-name: com.nimbusds:nimbus-jose-jwt
  dependency-version: 10.0.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2025-07-24 09:11:50 -07:00
Sriharsha Chintalapani
b0586f849f
Fix #22511: k8s secret support for Secrets Manager (#22516)
* Fix #22511: k8s secret support for Secrets Manager

* Update generated TypeScript types

* address comments

* pylint fix

* fix java checkstyle

* improve inCluster description in schema

* fix failing tests

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-07-24 12:40:51 +02:00
Sriharsha Chintalapani
cec7dd31ad
Fix #21662: Optional Redis Write-Through Cache for Database Queries (#21667)
* Fix #21662: Optional Redis Write-Through Cache for Database Queries

* Fix #21662: Optional Redis Write-Through Cache for Database Queries

* Fix #21662: Optional Redis Write-Through Cache for Database Queries

* Add lazy load, add test profile

* Fix stylecheck

* Fix tests, sonar comments

* Fix tests, sonar comments

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2025-07-04 23:21:20 -07:00
sonika-shah
5a124873e1
add name tag in pom.xml of all modules (#22104) 2025-07-02 22:48:11 +05:30
Ajith Prasad
33711766e2
commons-beanutils vulnerability reported in security scan (#21830) 2025-06-19 08:43:19 +05:30
Sriharsha Chintalapani
8adda4955c
Revert "Issues in Search Relevancy (#21841)" (#21853)
This reverts commit f388e570c1dac5b9eee31364870fb66e42715f18.
2025-06-18 16:43:34 -07:00
Mohit Yadav
f388e570c1
Issues in Search Relevancy (#21841)
* Fix explain turned by default, use dfs_query_then_fetch in cases of sharding of search cluster

* Add exact match configs

* Add exact match configs

* Update Logic to build search source builder with exact match priority

* Revert "Update Logic to build search source builder with exact match priority"

This reverts commit 175a2e9c6b67ee90d4b2a35af89bb035e8c45131.

* Revert "Add exact match configs"

This reverts commit 3fd52606610bbb97a676170004cab6d7adc31a0d.

* revert display name change

* make boost mode as sum by defaul

* add more fqnparts for schema and database

* revert DFS_QUERY_THEN_FETCH since sharding wasn the issue

* use fqn split

* Refactor FQN Parts

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-06-18 16:33:46 -07:00
Sriharsha Chintalapani
7d07222635
Fix #21668: Add Auto Tune to conifgure Search Index Application Parameters (#21669)
* Fix #21668: Add Auto Tune to conifgure Search Index Application Parameters

* add auto tune param in app json

* add test

* use virtual threads

* Fix Failing Test

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2025-06-13 09:23:22 -07:00
sonika-shah
d2c9952c9c
Bump jetty.version to 11.0.25 (#21699) 2025-06-12 18:24:42 -07:00
Sriharsha Chintalapani
7e6e23b95a
Fix API pages (#21695)
* Fix API pages

* Fix API pages

* Fix API pages
2025-06-12 12:23:34 +05:30
sonika-shah
ac5d6cc82d
Bump ch.qos.logback:logback-core from 1.4.x to 1.5.13 in /openmetadata-service and platform (#21484)
* Bump ch.qos.logback:logback-core from 1.4.x to 1.5.13 in /openmetadata-service

* match logback-classic.version to logback-core.version

* upgrade dropwizard version to 4.0.14 and logback to 1.5.18

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-06-03 13:57:12 -07:00
Mohit Yadav
17c4ab872d
Revert to jetty connector (#21494) 2025-05-30 20:40:15 +05:30
Mohit Yadav
e90fb0d8fd
Removed apache connector (#21481) 2025-05-30 11:54:34 +05:30
Sriharsha Chintalapani
138d57f890
Issue-19251: Upgrade dropwizard to 4.x and Jetty to 11.x (#19252)
* Issue-19251: Upgrade dropwizard to 4.x and Jetty to 11.x

* keep jdbi3.37.1 version for ConnectionAware functionality

* Code builds

* Merge with main and fix MCP

* Fix to make server run

* Tests fix

* JsonArray causing issue

* Fix WebSockets

* fix json2schema using jakarta instead of javax

* Fix patch tests

* Fix tests

* Register auditing subtypes in OpenmeteadataOperations

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2025-05-27 20:31:59 +05:30
Mohit Yadav
bbc450b2d1
Embedded MCP Server (#21206)
* Mcp Server

* Update Server

* Refactored into multiple files

* Add Tool Dynamic loading

* Updated to use toolName

* add description for tools

* initial create glossary term action

* initial patch entity tool

* Fix Glossary Tool

* Use prepare

* Changed const to default

* Prepare for Collate Tools

* Update HttpServletSseServerTransportProvider.java

* Checkstyle fix

* endpoint changed to messages in new versions

* Add Auth Filter to MCP Request

* description

* clean response

---------

Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2025-05-20 07:23:50 +02:00
Ashish Gupta
73aaa34b75
update the snapshot to 1.8.0 (#20925) 2025-04-24 10:46:36 +05:30
Pere Miquel Brull
34fbe5d64c
Docs - Prepare 1.7 docs and 1.8 snapshot (#20882)
* DOCS - Prepare 1.7 Release and 1.8 SNAPSHOT

* DOCS - Prepare 1.7 Release and 1.8 SNAPSHOT
2025-04-18 12:12:17 +05:30
Mohit Yadav
8615a5db29
Update Dependency of jetty-server (#20871) 2025-04-17 00:24:42 +05:30
Akash Jain
0f6d0523d8
feat: Bump Versions to 1.7.0-SNAPSHOT on Main Branch (#20847)
* feat: Bump Versions to 1.7.0-SNAPSHOT on Main Branch

* fix(script): Add a condition for "-SNAPSHOT" is version update script
2025-04-16 15:21:01 +05:30
Sriharsha Chintalapani
8947c7efbf
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#19603)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-ORGECLIPSEJETTY-8186141

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2025-03-24 15:31:44 -07:00
Sriharsha Chintalapani
62c0ef09e9
Fix #19103: Search Settings with Ranking config (#19962)
* Search Settings with Ranking config

* Fix #19103: Search Settings with Ranking config

* feat: search setting based on entity

* refactor: css and address review comments

* test: add unit tests

* test: e2e test for update search setting

* update searchSettings.json

* Refactor search builder

* Refactor search builder

* Integrate it into SearchClients

* Integrate it into SearchClients

* Integrate it into SearchClients

* Add preview endpoint

* Add preview endpoint

* test: fix and add unit tests

* test: add playwright tests

* style: fix minor styles

* fix: search preview on reload page

* Fix merge conflicts

* refactor: address PR comments

* style: term boost and icons

* fix: route for entity search page

* Fix fqn search, store search settings, add validation

* Fix SearchSettings tests

* Refactor the code to remove duplication; fix aggregation to have only unique field names

* Fix preview calls, common aggregations

* Fix search ranking

* Fix search relevancy

* Remove useNaturalLanguageSearch

* fix: style, delete field and tests

* Add text fields for tags, tier, certification

* Fix java style

* fix: style variables

* fix: localisation and styles

* fix merge conflicts

* Fix failing test due to same aggrgation name coming from the asset configuration and global

* Fix Hierarchy Tests

* Fix Test Case failures

* Add NLQ integration interfaces

* disable NLS by default

* Fix Glossary and Glossary Term Resource Tests

* revert openmetadata

* fix path for json search settings

* fix playwright failure

* Fix failing test

* add tag and search index

---------

Co-authored-by: Pranita <pfulsundar8@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2025-03-18 23:12:23 +05:30
Mohit Yadav
a47d725975
Fixed tomcat-jdbc dependency (#20218) 2025-03-12 23:25:30 +05:30
Sriharsha Chintalapani
eb1dfca8bc
Fix json smart version (#19826) 2025-02-17 07:35:05 -08:00
Akash Jain
9111d98eec
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#19773)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-IONETTY-8707740
- https://snyk.io/vuln/SNYK-JAVA-IONETTY-8707739

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2025-02-16 12:00:17 -08:00
Sriharsha Chintalapani
7e7691df15
Remove unused json-smart dependency (#19701)
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2025-02-10 20:36:10 -08:00
Sriharsha Chintalapani
3678d1c3d6
upgrade logback versions (#19259) 2025-01-07 19:30:36 +05:30
Aniket Katkar
9055bceba0
Fix: Pom after docs removal (#18999) 2024-12-11 13:23:29 +05:30
Mohit Yadav
0bf5a98501
Add Checksum package to pom (#18696) 2024-11-19 18:49:07 +05:30
Imri Paran
12945c62c0
revert aws sdk version from f0b76c45d081fe66bded44c60fa4560a879cb346 (#18695)
Due to failures in migration:

```
2024-11-19 17:16:15.916	picocli.CommandLine$ExecutionException: Error while calling command (public java.lang.Integer org.openmetadata.service.util.OpenMetadataOperations.migrate(boolean)): java.lang.NoClassDefFoundError: software/amazon/awssdk/checksums/SdkChecksum
2024-11-19 17:16:15.641	Updating bot users in case of an update on the JSON schema: [managed-aws]
2024-11-19 17:16:14.903	Updating services in case of an update on the JSON schema: [managed-aws]
```
2024-11-19 18:42:05 +05:30
Sriharsha Chintalapani
f0b76c45d0
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#18651)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-IONETTY-8367012

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-11-18 09:04:18 -08:00
Sriharsha Chintalapani
88c8fb48f3
Add Edit glossary terms, Edit Tier , Edit Tags as separate permissions (#18331)
* Add EditGlossaryTerms Permission

* Fix #18330: Add EDIT_GLOSSARY_TERM permission and enforce EDIT_TIER permisson

* add edit glossary term permission check in UI

* revert EDIT_GLOSSARY_TERMS operation

* Add EDIT_GLOSSARY_TERMS to common operations

* Add EDIT_TIER to common operations

* add default empty array for tags field, as patch calls can run into issues

* Fix tests

* Fix tests

* added glossary terms

* fix conflicts

* fix permission check for data model

* Add EditGlossaryTerms to DataConsumerPolicy

* Add EditGlossaryTerms,EditTier to DataConsumerPolicy

* fix tests

* Fix migrations for EditTier,EditGlossaryTerms

* add edit tier permission to data consumer

* Fix tests

* fix pytests

* missing test_dbt.py

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 10:50:15 -08:00
Sriharsha Chintalapani
783ec62e4c
Add Lineage Export Async endpoint using Search (#18553)
* Add Lineage Export Async endpoint using Search

* Lineage api (#18593)

* add api for lineage export

* update API call for lineage async

* use JsonUtils instead of objectMapper to read lineage search response

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sweta Agarwalla <105535990+sweta1308@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-11-15 09:11:23 -08:00
Chirag Madlani
fda0e542e4
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#18620)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-IONETTY-8367012

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-11-13 20:48:07 -08:00
IceS2
e5441a9969
Initial implementation for Governance Workflows (#17837)
* Initial implementation for Governance Workflows

* Remove file that shouldn't be there

* Initial implementation for the governance workflows

* Refactor JSONSchema for Governance Workflows

* Refactored WorkflowInstanceStatus to keep the stage properly

* Fix Migration for MySQL for the WorkflowInstanceStates change

* Update WorkflowInstanceState to track all the states propertly

* Initial code tro migrate Request type of Tasks

* copy governance schemas into json directory

* update schema names

* Extract WorkflowInstance into its own resource

* Update WorkflowInstance to have the workflow starting variables

* Changed relatedEntity to be an EntityLink

* First iteration on checkEntityAttributes JSONLogic

* Refactored Workflow into TriggerWorkfow and MainWorkflow

* Fixed WorkflowInstance and WorkflowInstanceState

* Fixed WorkflowInstance and WorkflowInstanceState when process is terminated

* Fix UserTask

* json logic changes to workflow

* Fix Patch request. Add exclude filter to EventBasedEntityTrigger

* update query builder

* Fix Migrations

* Fix Migrations

* modify query builder widget

* fix search props

* update widget and use default fields

* Update edges to always have a true/false condition

* made changes to provider for fields

* push progress

* fix grouping of array field

* Remove duplicated migrations

* Fix migrations

* Fix migrations

* Fix Tests and commit some improvements suggestion

* fix unit tests

* Add Certification Workflow

* Fix Tests

* Fix Tests

* fix tests

* add delay in pagination tests

* fix minor config changes for json logic

* change the glossary page

* Remove Table Certification Workflow seed Data

* add json logic and elastic search parsers

* minor schedule interval changes for workflows

* minor fixes to the parsing logic

* certification ui

* fix tests

* fix tests

* add other search fields

* Add Certification to search index

* Update index mapping for certification

* add quick filter for certification

* update locales

* Fix index mapping

* Add missing index updates

* Fix checkstyle

* increase timeout

* skipping user test

* skipping team test

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2024-11-13 08:47:44 +01:00
Siddhant
36025377fa
fix dependency. (#18536) 2024-11-06 14:02:12 +01:00
Sriharsha Chintalapani
b763091bf6
Make Export CSV Async API, websocket to push data back (#18497)
* Make Export CSV Async API, websocket to push data back

* ui: implement csvExportChannel to receive entity export data via websocket

* ui: update flow to show dialogbox with the response message and once data is available download the file

* minor: remove unused code

* ui: show inline banner for success and error message

* remove unused locale keys

* ui: clear existing csvexport job on sucess

* fix failing UserResourceTest

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-11-05 18:32:41 +05:30
Sriharsha Chintalapani
572822e36a
Minor: add versions to maven build plugins (#18443)
Co-authored-by: Suresh Srinivas <srini30005@gmail.com>
2024-10-29 18:54:50 -07:00
Suresh Srinivas
00c38323bd
Update dependencies for release 1.6 (#18444) 2024-10-28 14:52:55 -07:00
Sriharsha Chintalapani
d8f5398efb
Add Azure Token Base Authentication (#18387)
* Add azure token based auth

* Add Azure Token Base Authentication

* Update azure-auth.md

* Update azure-auth.md

* feat: Add `azure-identity-extensions` library for passwordless database connection

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Akash-Jain <15995028+akash-jain-10@users.noreply.github.com>
2024-10-24 11:29:46 +02:00
Sriharsha Chintalapani
85cc240e1e
Minor: Add an API to list all the properties and custom properties for a given entity type (#18248)
* Minor: Add an API to list all the properties and custom properties for a given entity type

* Minor: Add an API to list all the properties and custom properties for a given entity type

* add custom properties

* add custom properties

* add custom properties
2024-10-19 14:14:34 -07:00
Ethan
e708a3242e
feat: update version (#18259) 2024-10-17 16:18:37 -07:00
Sriharsha Chintalapani
2da9c34730
Issue-13782: RBAC for search (#17757)
* Issue-13782: RBAC for search

* Handle complex conditions

* Add optimizations to condition parsing

* Add SPeL based condition evaluation

* Fix styling

* use SpeL Parser AST to compose ES conditions

* Fix test message

* improvements in query generation

* improvements in query generation

* Add more complex queries

* Add more complex codndition tests and evaluation

* Add OpenSearch query builders and tests

* Add handling resources in a rule, add tests for multiple rules and inherited roles
2024-09-16 11:45:21 +05:30
Sriharsha Chintalapani
ea62f34385
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#17841)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-ORGJETBRAINSKOTLIN-2393744

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-09-15 16:31:34 -07:00