379 Commits

Author SHA1 Message Date
sonika-shah
5ccc0c5a2a
Improvement #19065 : Support removing existing enumKeys (for enum type custom property) (#19054)
* Improvement : Support removing existing enumKeys (for enum type custom property)

* initial implementation of background job job runner for enum cleanup

* fix postgres query

* ui: allow existing enum value updates in EditCustomPropertyModal

* create BackgroundJob to use jsonSchema

* feat: add socket event handling for background job part1

* fix(i18n): remove trailing spaces in Galician language JSON file

* refactor to use jsonSchema

* feat(ui): show browser notification for custom property update

* feat(utils): add getCustomPropertyEntityPathname function and tests

* change enumCleanup trigger condition

* feat(i18n): add enum property update message to multiple language JSON files

* enhancement : improve execution flow and better exception handling with test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-07 19:25:59 -08:00
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites (#19221)
* backend

* format & tests

* rename backend

* migrations and ingestion

* format & tests

* format & tests

* tests

* format & tests

* tests

* updated ui side of changes

* addressing comment

* fixed failing unit test

* fix test list

* added e2e test, and fixed existing test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Teddy
bd68d959f3
fix: added index on timestamp field for reindex (#19101) 2024-12-16 20:57:42 +01:00
Mohit Yadav
3578a4b32d
Domain Policy Update to be non-system (#19060) 2024-12-15 01:18:12 +05:30
Siddhant
4985496051
fix: Postgresql migrations for eventSubscriptionOffset. (#18980) 2024-12-09 20:19:39 -08:00
Siddhant
e770ba8e0e
FIX: Combined index on offset and eventTime in changeEvent table (#18922)
* fix: combined index on offset and eventTime in changeEvent table

* fix: correct syntax error in Postgres query.

* fix: correct syntax error in Postgres query.

* fix: correct syntax error in Postgres query.
2024-12-04 15:14:31 -08:00
IceS2
f939e7d3c0
Reduce amount of flowable created tables (#18882)
* Reduce the number of flowable tables on postgresql

* Reduce the amount of Flowable Created Tables
2024-12-04 19:34:33 +01:00
Siddhant
28f551d20a
GEN-Activity Feed Filtering (#18873)
* fix: Update configuration for activity feed.

* fix: Migrations

* remove the customProperty and testSuite feed beautification case from the feed

* fix: Remove custom properties from the source

* fix: add metric in the matchAnySource.

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-12-04 16:56:21 +05:30
Pere Miquel Brull
613fd331e0
MINOR - Clean up configs & add auto classification docs (#18907)
* MINOR - Clean up configs & add auto classification docs

* deprecation notice
2024-12-04 09:32:25 +01:00
harshsoni2024
cb33f274fc
Connector: rename microstrategy connector (#18604) 2024-11-28 18:50:42 +05:30
Suman Maharana
9a21e77e15
Added dbt cloud multi projects and jobs filter (#18801)
* Added dbt cloud multi project and jobs filter

* added tests

* change to array type

* updated yaml config

* added migrations
2024-11-28 16:10:34 +05:30
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow (#18610) 2024-11-19 08:10:45 +01:00
Aniket Katkar
bfc12feafc
GEN-1976: Alert improvements (#18448)
* Move the alert components to the components folder from pages folder

* auto generated localization changes and linting changes

* Fix the imports

* Add AlertConfigDetails component

* Add tabs in the alert details page

* auto generated localization changes

* Apply permissions for alert and add owner update functionality

* Change the alert update method from PUT to PATCH

* Add the recent events tab

* auto generated localization changes

* Improve the recent events tab for alert details

* Implement the logic to show alert events count on the alert details page

* auto generated localization and linting changes

* Add the pagination logic for the recent events list on the alert details page

* Fix the logic for previous click in NextPreviousWithOffset

* Localization change for other languages

* Hide pagination when not required

* Add pagination and permissions to API; create eventsRecord API; fix timestamp issue

* fix: remove publisherName from test destination message.

* Fix the recent events tab error

* fix: Resolve PATCH API issue with destination ID creation.

* fix: Delete associated alert metrics upon alert deletion

* feat: Detailed HTTP Response for Destination Test

* Add the test external destination functionality

* localization changes for other languages

* fix: Set status for testing destination in EmailPublisher

* Add advanced configurations for webhook destinations

* Localization changes for other languages

* Fix the destination status not showing
Add the POST and PUT radio button in advanced configs

* Localization changes for other languages

* Update the API to fetch the alert event diagnostic counts

* Improve the AlertConfigDetails component to show the alert configs properly

* fix: Display displayName instead of fullyQualifiedName in templates

* Fix the test destination result not showing the first time

* Fix the test destination status not showing for already existing destinations

* Fix the unit tests

* Fix playwright tests

* refactor: Separate sendMessage and sendTestMessage methods; add statusContext

* fix: Use name or fqn as fallback if displayName is null or empty

* fix: upsert successfulChangeEvent

* eliminate the GovernanceWorkflowChangeEvent option from external destination

* Add code comments

* Fix the unit test

---------

Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
Co-authored-by: Siddhanttimeline <siddhanttimeline9@gmail.com>
2024-11-14 17:39:45 +05:30
IceS2
e5441a9969
Initial implementation for Governance Workflows (#17837)
* Initial implementation for Governance Workflows

* Remove file that shouldn't be there

* Initial implementation for the governance workflows

* Refactor JSONSchema for Governance Workflows

* Refactored WorkflowInstanceStatus to keep the stage properly

* Fix Migration for MySQL for the WorkflowInstanceStates change

* Update WorkflowInstanceState to track all the states propertly

* Initial code tro migrate Request type of Tasks

* copy governance schemas into json directory

* update schema names

* Extract WorkflowInstance into its own resource

* Update WorkflowInstance to have the workflow starting variables

* Changed relatedEntity to be an EntityLink

* First iteration on checkEntityAttributes JSONLogic

* Refactored Workflow into TriggerWorkfow and MainWorkflow

* Fixed WorkflowInstance and WorkflowInstanceState

* Fixed WorkflowInstance and WorkflowInstanceState when process is terminated

* Fix UserTask

* json logic changes to workflow

* Fix Patch request. Add exclude filter to EventBasedEntityTrigger

* update query builder

* Fix Migrations

* Fix Migrations

* modify query builder widget

* fix search props

* update widget and use default fields

* Update edges to always have a true/false condition

* made changes to provider for fields

* push progress

* fix grouping of array field

* Remove duplicated migrations

* Fix migrations

* Fix migrations

* Fix Tests and commit some improvements suggestion

* fix unit tests

* Add Certification Workflow

* Fix Tests

* Fix Tests

* fix tests

* add delay in pagination tests

* fix minor config changes for json logic

* change the glossary page

* Remove Table Certification Workflow seed Data

* add json logic and elastic search parsers

* minor schedule interval changes for workflows

* minor fixes to the parsing logic

* certification ui

* fix tests

* fix tests

* add other search fields

* Add Certification to search index

* Update index mapping for certification

* add quick filter for certification

* update locales

* Fix index mapping

* Add missing index updates

* Fix checkstyle

* increase timeout

* skipping user test

* skipping team test

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2024-11-13 08:47:44 +01:00
Mayur Singal
8e1c839473
MINOR: Summary Card changes require for UI (#18585) 2024-11-13 12:22:47 +05:30
Ayush Shah
6fa03ee66a
Fixes GEN-1994: Remove View Lineage from Metadata Ingestion flow (#18558) 2024-11-13 00:08:55 +05:30
Mohit Yadav
9a5dc61ca7
Fix Search Index Contention (#18605)
* Fix Search Index Contention

* Update searchIndexingAppConfig.json

* Missing Error Logs and Stats

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-11-12 20:36:23 +05:30
Mayur Singal
943b4efb4d
GEN-1418: DI - Allow Support for Multiple Metrics (#18476) 2024-11-11 14:27:46 +05:30
Siddhant
2f39b76c33
Fix Fetching Successfully Sent Change Events For Debuggability API (#18493)
* fix getSuccessfullySentChangeEventsForAlert

* migrations.

* offset-based pagination

* add startingOffset in EventSubscriptionDiagnosticInfo

* fix missing paginationOffset

* Record successfully sent change events in a separate table, clean up associated successful and failed records on alert deletion.

* Remove oldest 30 records if record count reaches 100 for successful_sent_change_events.

* list unprocessed changeEvents based on alert subscription rules, ensuring only relevant unprocessed events are listed.

* refactor: enhance eventSubscriptionDiagnosticInfo

Added new fields to the eventSubscriptionDiagnosticInfo schema to support detailed diagnostics:
- `relevantUnprocessedEventsCount`: Count of relevant unprocessed events based on alert filtering rules.
- `relevantUnprocessedEventsList`: List of relevant unprocessed events based on alert filtering rules.
- `totalUnprocessedEventsCount`: Count of all unprocessed events.
- `totalUnprocessedEventsList`: List of all unprocessed events.

* Refactor: delete 30 oldest records per alert when the count reaches 100 records

* fix: table creation issue in PostgreSQL

* fix tests.

* scheduleCleanupJob

* remove in-line deletion of records.

* fix postgres table structure issue.
2024-11-08 12:47:16 +01:00
Siddhant
4f5bfa8549
fix migration, source column NOT NULL constraint on consumer_dlq (#18485) 2024-10-31 14:10:19 +05:30
Siddhant
f0fc482c46
Fixes #17718 : Notification Debuggability (#18199)
* create EventSubscriptionDiagnosticInfo

* API for failedEvents

* migrations

* add index on source

* list all failedEvents optionally filtered by source

* refactor

* refactor

* getSuccessfullySentChangeEventsForAlert API

* increase defaultValue of limit to 100

* resolve conflicts

* listEvents API with 'failed', 'successful,' and 'unprocessed' query parameters

* fix description

* eventSource as enums

* refactor name.

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-10-30 11:55:51 -07:00
Pere Miquel Brull
ed9b66c5c5
MINOR - Move appName migration to 1.5.9 (#18435) 2024-10-28 16:29:56 +01:00
Pere Miquel Brull
7d01373261
MINOR - Renaming MetaPilot to CollateAI (#18393)
* MINOR - Renaming MetaPilot to CollateAI

* MINOR - Renaming MetaPilot to CollateAI

* MINOR - Renaming MetaPilot to CollateAI

* MINOR - Renaming MetaPilot to CollateAI

* fix

* moving limits migrations for 1.5.9
2024-10-28 09:21:15 +01:00
Imri Paran
d0fb5137af
[GEN-970] added supportsSystemProfile to connector (#18203)
* added supportsSystemProfile to connector

* reverted unintended changes

* added supportsSystemProfile for bigquery
2024-10-22 07:39:18 +02:00
Imri Paran
befa1f72e8
MINOR: add supportsDataDiff flag (#18205)
* feat: add supportsDataDiff flag

enables filtering services based on support of the data diff test case

* fixed mysql migration

* fixed conflicts
2024-10-16 11:07:31 +02:00
Imri Paran
52b547ede1
chore(data-diff): added migrations (#18287)
added migrations for data diff test definition
2024-10-16 08:04:19 +00:00
Siddhant
07b039b8b8
Migrations fixes for #17967 (#18232)
* fix migrations.

* fix logging
2024-10-11 13:51:12 +00:00
Siddhant
c6e12fe703
migrate serviceType (#18226) 2024-10-11 01:47:29 +05:30
Mohit Yadav
81729bf36b
[OnBoardingApp] Add apps data store (#18209)
* fix(ui): pagination issue with search in services list page

* Add Apps Data store

* Revert Services Changes

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-10-10 19:25:11 +05:30
Pere Miquel Brull
e81efc8d98
MINOR - Add App Limits (#18072)
* App Limits - Prep extension

* App Limits - Prep extension

* App Limits - Allow to search by name

* App Limits - Allow to search by name

* fix postgres sql

* comments
2024-10-03 11:00:29 +02:00
Mohit Yadav
f1ae49e1a0
[Search] Indexing Fixes (#18048)
* Fix OpenSearch Content Size Issue
entity content is too long [204857600] for the configured buffer limit [104857600]

* Change Type to Long

* Add Payload Size to take dynamic entry

* Migrations for 1.5.6

* Mark Stale Entries Stopped

* Format checkstyle

* Fix failure

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-09-30 23:39:27 +05:30
Mohit Yadav
afd7887bd8
[Issue-16487] Add Stop for Search Indexing (#17914)
* Add Stop for Search Indexing

* Update Logs and Message

* test for stop func

* remove test
2024-09-25 18:29:32 +05:30
Teddy
a6c2ec747a
Minor fix psql dq migration failures (#17985)
* fix import issue

* fix: psql null testCaseResult migration failure

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-25 12:15:26 +02:00
Teddy
305b8b7202
MINOR - fix Data Quality 1.6 migration (#17966)
* fix import issue

* fix: delete unlinked test case results

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-24 12:55:10 +02:00
Teddy
33c50efdbf
GEN-1192 - Move Test Case to its Own Resource (#17862)
* feat: indexed test case results

* feat: added indexation logic for test case results

* style: ran java linting

* fix: IDE warnigns

* chore: added test case results migration

* style: ran java linting

* fix: postgres migration column json ref

* empty commit to trigger queued

* chore: extracted test case results to its own resource

* chore: fix failing tests

* chore: move testCaseResult state from testSuite and testCase to dynamic field fetched from test case results search index

* chore: clean up test case repository

* style: ran java linting

* chore: removed testCaseResultSummary and testCaseResult state from db

* fix: test failures

* chore: fix index mapping type for result value

* chore: fix test failure
2024-09-18 11:58:59 +02:00
Teddy
aa399dfd68
MINOR: move constraint update from schemaChanges to postDataMigration (#17801)
* fix: move constraint update from schemaChanges to postDataMigration

* fix: move index after column is created
2024-09-12 08:13:41 +02:00
Teddy
1f72a3c9c6
GEN-1213 - logic to index test case result (#17706)
* feat: indexed test case results

* feat: added indexation logic for test case results

* style: ran java linting

* fix: IDE warnigns

* chore: added test case results migration

* style: ran java linting

* fix: postgres migration column json ref

* empty commit to trigger queued

* fix: added index on id field
2024-09-09 07:58:45 +02:00
sonika-shah
cce22a1b5c
Improve count/feed api performance for 1.5 (#17576) 2024-08-23 11:20:34 -07:00
sonika-shah
266920197f
Implement full text index on taskAssigneesIds column in both mysql and postgresql (#17520)
* Implement full text index on taskAssigneesIds column in both mysql and postgresql

* move migrations in 1.5
2024-08-21 12:18:50 -07:00
Onkar Ravgan
f2f8f49cfc
update test def name len (#17522) 2024-08-21 09:27:07 -07:00
mohitdeuex
30a3f32b0f Change Query for Postgres 2024-08-21 17:07:45 +05:30
Mohit Yadav
9309cf94c1
Add Api Service, Collection and Endpoint in reindex app (#17527)
* Add Api Service and Collection and Endpoint

* Keep SearchIndexing as false by Default
2024-08-21 16:26:50 +05:30
Pere Miquel Brull
f7aab6b7a4
MINOR - Fix list API for automation workflows (#17494)
* MINOR - Fix list API for automation workflows

* MINOR - Fix list API for automation workflows
2024-08-20 12:40:21 +02:00
Onkar Ravgan
2a8ee41c38
Fix entity_ext owner update query (#17493) 2024-08-19 16:01:07 -07:00
sonika-shah
424a03f49c
add index on extension_index table (#17496) 2024-08-19 15:18:53 -07:00
Imri Paran
7508848376
fix(dq): data types for unique columns (#17431)
1. remove json and array from supported data types of unique column test.
2. migrations.
3. tests.
2024-08-19 14:28:42 +02:00
Sriharsha Chintalapani
39203a1156
Fix #17352: Remove TablesInitializer and bootstrap_storage.sh (#17353)
* Fix #17352: Remove TablesInitializer and bootstrap_storage.sh

* Fix #17352: Remove TablesInitializer and bootstrap_storage.sh

* Fix #17352: Remove TablesInitializer and bootstrap_storage.sh
2024-08-15 19:27:02 -07:00
Mohit Yadav
6e80d3f405
Fix Migrations (#17364) 2024-08-09 20:58:44 +05:30
Mohit Yadav
9bbc6a1801
Migrate NameHash (#17317) 2024-08-06 18:41:37 +05:30
Mohit Yadav
21120a0130
Add Fqn Migration to lowercase (#17311) 2024-08-06 16:49:23 +05:30