939 Commits

Author SHA1 Message Date
Imri Paran
a6670af500
MINOR: Use interface schemas for governance workflows (#19411)
* feat(apps): support event subscriptions

- added support for apps with event subscriptions.
- added support for custom consumers on event subscriptions.
- use native application methods instead of reflection in ApplicationHandler

* removed runMethodFromApplication reflection and use concrete methods from AbstractNativeApplication

* format

* fix(governance-workflows): defined interface in workflow schema

- defined types for workflow nodes
- fixed updateEdge method in WorkflowDefinitionRepository

* ref(governance-workflows): use explicit types in schema

use explicit interface type in json schemas and use annotations to infer the concrete classes when deserializing

* - implemented CustomSignal trigger
- use JsonTypeInfo and JsonSubTypes to infer types in jsonschema for nodes and triggers
- Implemented JsonLogicFilter task
- implemented placeholder noop task

* - simplified trigger type names
- removed governance workflow root "type"
- added "deployed" field to governance workflow
- applied changes to existing gov-workflows
- migrations

* fixed migrations
2025-01-24 09:08:06 +01:00
Aniket Katkar
9a167c7c3f
Chore: Add the missing generated types and fix json schemas (#19480)
* Add the missing generated types

* Fix the type errors

* fix the JSONs and add respective generated type files
2025-01-23 20:45:31 +05:30
Teddy
cacfabd9ce
ISSUE #1052 - Implement List entities with test suites repo logic (#19461)
* fix: centralized listWithOffset logic

* feat: dq app config + list entities with tests logic

* fix: test case
2025-01-22 08:08:07 +01:00
Siddhant
486230827d
#15600: Retention Application : Delete change_events, activity threads, versions based on admin retention policies (#18913)
* feat(retention-policy): initial setup of the Retention Policy app

* refactor: Update logic to make RetentionPolicyApp a ScheduledOrManual app

* refactor: Update retention policy JSON schema and logic to support event subscription, versions, and activity thread with individual retention periods

* refactor: Rename eventSubscriptionRetentionPeriod to changeEventRetentionPeriod

* fix: Implement activity thread retention policy cleanup

* chore: Remove activity thread and versions retention code; add banner and icon
2025-01-21 18:12:35 +05:30
Imri Paran
3abae45403
MINOR: feat(apps): support event subscriptions (#19338)
* feat(apps): support event subscriptions

- added support for apps with event subscriptions.
- added support for custom consumers on event subscriptions.
- use native application methods instead of reflection in ApplicationHandler

* removed runMethodFromApplication reflection and use concrete methods from AbstractNativeApplication

* format
2025-01-21 07:10:26 +01:00
Gyowanny P. Queiroz
cbbef9e3b9
Move PipelineServiceClient to openmetadata-spec (#19381) 2025-01-15 14:41:47 -03:00
Suman Maharana
4cd5988bc1
Fix #19313 - dbt ingestion picks up wrong service to patch metadata (#19337)
* Fix #19313 - dbt ingestion picks up wrong service to patch metadata

* py format and linting

* Added a flag for searching across databases and docs

* py format
2025-01-15 11:45:32 +05:30
Keshav Mohta
4bd8994a00
Feature: Cassandra SSL (#19226) 2025-01-10 11:25:43 +05:30
Imri Paran
7d05902945
fix: reverted deprecated fields on testSuite (#19284)
Added deprecated fields with a notice. This is to ensure migrations that use them do not break.
2025-01-08 18:34:17 -08:00
Matt Chamberlin
bf8a9e4f46
fix: Add docStore endpoints to Ometa API routes (#19172) 2025-01-08 15:47:18 +05:30
Mayur Singal
e32ad88c81
Fix #19160: Add Depth Support Part-1 (#19275) 2025-01-08 11:18:05 +05:30
sonika-shah
5ccc0c5a2a
Improvement #19065 : Support removing existing enumKeys (for enum type custom property) (#19054)
* Improvement : Support removing existing enumKeys (for enum type custom property)

* initial implementation of background job job runner for enum cleanup

* fix postgres query

* ui: allow existing enum value updates in EditCustomPropertyModal

* create BackgroundJob to use jsonSchema

* feat: add socket event handling for background job part1

* fix(i18n): remove trailing spaces in Galician language JSON file

* refactor to use jsonSchema

* feat(ui): show browser notification for custom property update

* feat(utils): add getCustomPropertyEntityPathname function and tests

* change enumCleanup trigger condition

* feat(i18n): add enum property update message to multiple language JSON files

* enhancement : improve execution flow and better exception handling with test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-07 19:25:59 -08:00
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Akash Verma
b2898f7007
Cockroach enhancement (#19108) 2025-01-07 18:51:59 +05:30
Suman Maharana
00be51f299
Fixes #17747: dbt update owners (#19144)
* Fixes 17747: dbt update owners

* update messages

* addressed comments

* py_format

* py_format

* Added tests
2025-01-07 16:37:30 +05:30
Suman Maharana
5b77c14dc8
Fixes #18816 - Airbyte api endpoint (#19207)
* Fix Airbyte api endpoint

* Fix Airbyte api endpoint
2025-01-07 12:41:56 +05:30
Akash Verma
39dcb5baef
Feature : Cockroach db connector (#18961) 2025-01-02 13:07:55 +05:30
Mohit Yadav
06d71f9450
Fix EntityHierarchy Schema (#19104)
* Fix Entity Schema

* fix additionalProperties in entityHierarchy json

---------

Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2024-12-20 20:13:04 +05:30
Pere Miquel Brull
d084243d0a
MINOR - Add Gov entities for entity ref & Automator Actions (#19090)
* MINOR - Add Gov entities for entity ref

* MINOR - Add Gov entities for entity ref

* add automator configs

* add automator configs
2024-12-20 07:14:52 +01:00
Mohit Yadav
9d934cbcd3
[Code Improv] Added Mappers (#19111)
* Added Mappers and moved logic from controller to mapper class for resources

* Add missed Fqn

* Fix PyTest Failures

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-12-20 11:40:39 +05:30
Akash Verma
69557e8716
fixes: #15742 Oracle stored package feature (#18852) 2024-12-16 19:35:20 +05:30
Mohit Yadav
50ae01e2ce
Add Algorithm option for validation in yaml (#19049)
* Add algorithm option in authentication

* ENtity Repository code remove

* Keep Default Value

* Fix Test

---------

Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
2024-12-15 01:24:03 +05:30
Mohit Yadav
521619ae32
Correct Imports for Class Converter Factory (#19057) 2024-12-15 01:20:28 +05:30
Karan Hotchandani
497b477be3
remove extra assignees from workflow (#19030) 2024-12-14 09:00:19 -08:00
Sriharsha Chintalapani
cbd2c6ad87
Fix #issue-16041: The persona JSON schema is named Team (#19035) 2024-12-12 19:27:53 -08:00
Keshav Mohta
f4ff43c24c
Fixes #17888 - Implemented Cross Database Lineage (#18831) 2024-12-12 15:13:36 +05:30
Keshav Mohta
cde3a7dd1e
Feature: Cassandra Connector (#18943) 2024-12-12 15:12:55 +05:30
IceS2
c24bc870d0
Parametrizing Flowable configurations and adding a way to handle history cleaning (#18990) 2024-12-12 10:01:39 +01:00
Mayur Singal
6326ae3cb2
MINOR: DB2 Support License (#18928) 2024-12-05 09:54:58 +05:30
Sriharsha Chintalapani
0629750e0a
Minor: fix indexType definition in searchIndex spec (#18912)
* Minor: fix indexType definition in searchIndex spec

* Minor: fix indexType definition in searchIndex spec
2024-12-04 08:50:01 -08:00
Pere Miquel Brull
613fd331e0
MINOR - Clean up configs & add auto classification docs (#18907)
* MINOR - Clean up configs & add auto classification docs

* deprecation notice
2024-12-04 09:32:25 +01:00
Sriharsha Chintalapani
bf00c9c12e
Create a relationship to manage ER based on tableConstraints (#18892)
* Create a relationship to manage ER based on tableConstraints

* Add relationship/remove relationship from entity_relationship table whenever table constriants are updated/added , validate table constraints

* remove findRelatedTables queries

* Add Migrations to add constrait relationship

* remove findRelatedTables code

* Fix postgres migration query

* fix pg migration

* fix test

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-12-03 20:47:34 -08:00
Mayur Singal
9b9509f4b9
MINOR: Mysql Lineage Support Main (#18780)
* MINOR: Mysql Lineage Support Main

* fix test

* fix test

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-11-29 20:48:42 +05:30
Siddhant
3453c799f7
FIX: Mask Slack App User & Bot Tokens (#18814)
* fix: mask slack user and bot token.

* refactor: Move the code in SlackApp
2024-11-28 23:59:46 +05:30
mgorsk1
da176767a8
feat: add dbt freshness check test (#18730)
* add dbt freshness check

* docs

* run linting

* add test case param definition

* fix test case param definition

* add config for dbt http, fix linting

* refactor (only create freshness test definition when user executed one)

* fix dbt files class

* fix dbt files class 2

* fix dbt objects class

* fix linting

* fix pylint

* fix linting once and for all

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-11-28 18:30:11 +01:00
harshsoni2024
cb33f274fc
Connector: rename microstrategy connector (#18604) 2024-11-28 18:50:42 +05:30
Suman Maharana
9a21e77e15
Added dbt cloud multi projects and jobs filter (#18801)
* Added dbt cloud multi project and jobs filter

* added tests

* change to array type

* updated yaml config

* added migrations
2024-11-28 16:10:34 +05:30
Aniket Katkar
c101c7cf30
Minor: Auto classification UI (#18785)
* Add the auto classification pipeline option in the service ingestion

* Localization changes for other languages

* Improve the logic for getSupportedPipelineTypes function and add unit tests for newly added logic

* Add playwright tests for the auto classification feature

* Improve the getSupportedPipelineTypes function logic to reduce the cognitive complexity

* update md docs

* Add classificationFilterPattern in the UI schema form order

* fix logs from backend for auto classification

* Changes to view the auto classification logs

* Fix the sonar errors

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-11-27 14:51:59 +05:30
IceS2
8cade39bad
MINOR: Increase visibility on flowable process (#18792)
* Add annotation to turn delete_recursive test into the last one to be executed

* Improve Fetching Entities by only fetching the FQN

* Cut back default batchsize to 500

* implement logging for flowable process

* Implement Error Handling Flow

* Remove testing code

* Checking if User is Reviewer when resolving glossary task
2024-11-26 17:04:59 +01:00
Keshav Mohta
10a7f4ea6c
Fixes #15543: Implemented ElasticSearch Index Template Ingestion (#18686) 2024-11-26 14:42:43 +05:30
Kenil Shah
09db201a57
UI: Add option to stop the running application (#18759)
* Add Stop functionality for Applications

* show kill button by default

* Add option to stop the running application

* fix: show stop button for ruuning & activeerror status

* add localisation files

* added unit tests

* update localisation files

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2024-11-26 11:55:47 +05:30
Mayur Singal
40a70357e8
MINOR: Add UDF support for SP - backend (#18767) 2024-11-25 15:27:53 +05:30
IceS2
b3d765dce1
MINOR: Governance Workflows fixes (#18744)
* Add annotation to turn delete_recursive test into the last one to be executed

* Improve Fetching Entities by only fetching the FQN

* Cut back default batchsize to 500
2024-11-24 11:30:56 -08:00
Imri Paran
ee7d043035
[GEN-2109] feat(mongo): added ssl support (#18731)
* feat(mongo): added ssl support

Added SSL support for MongoDB using the SSL manager.

Attached a video demo.

- [Example repository for setting up mongodb with SSL](https://github.com/sushi30/mongodb-docker-ssl-example)
- [MongoDB TLS documentation](https://www.mongodb.com/docs/manual/tutorial/configure-ssl/)

* fixed test_doris.py
2024-11-22 08:54:13 -08:00
Siddhant
702c34c0af
NoSchedule as ScheduleType for Slack App (#18715)
* fix: NoSchedule as scheduleType for Slack App

* fix: remove callback url from the config.

* no render of schedule for noSchedule type

* fix: Migrations and add ScheduleType.NoSchedule

* fix: Migrations

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
2024-11-21 22:29:09 -08:00
Ayush Shah
17ffdf9850
fix: modify fqn to allow quotes with dots (#18719) 2024-11-22 09:33:50 +05:30
Imri Paran
d089ffa2d0
doc(mongodb): mongodb+srv connection scheme (#18712)
- added information about the mongodb+srv connection scheme
- added link to atlas guide for getting the connection string
2024-11-21 18:42:35 +00:00
Siddhant
dad3c9d7f2
refactor: Slack app improvements (#18684) 2024-11-19 19:08:47 +05:30
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow (#18610) 2024-11-19 08:10:45 +01:00
sonika-shah
853bbc205b
GEN-1983 : Feat - Support Tag Asset Page (#18505)
* Feat :  Support bulk adding tag to Assets

* remove warnings

* add apis for assets remove

* Fix: Add tag page (#18461)

* add tag page which all Assets

* update as per feedbacks

* update as per feedbacks

* add divider in header badge

* remove styling

* update permission and refactoring code

* updated as per comments

* fix sonar cloud issues

* add delete asset functionality

* refactor entityTypeString

* made the top bar fixed to top

* add tests for tag page

* fix check failures issue

* fix tag page check failure

* fix flaky test issue

* add tag page tests

* update the add asset test

* update playwright tests

* add right panel in tag page

* updated as per feedbacks

* remove usage test

* updated as per feedbacks

---------

* Backend: make bulkAssets api async

* Backend: limit bulkAssets api async to Tag Assets page

* Update Tag page Assets API (#18622)

* create branch

* update add asset API

* add websocket on tags page

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>

* add failed case for socket operation

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sweta Agarwalla <105535990+sweta1308@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-11-18 17:57:07 +05:30