2141 Commits

Author SHA1 Message Date
Ayush Shah
6f1df37ba1
Fixes GEN-1260: Add Validators while creating table to escape special characters (#18456) 2024-11-18 15:02:57 +05:30
IceS2
231025df67
MINOR: Improve PeriodicBatchEntityTrigger Workflows performance (#18664)
* Update default configurations

* Refactor how we call the main Workflow in order to increase performance
2024-11-18 08:17:22 +01:00
Sriharsha Chintalapani
88c8fb48f3
Add Edit glossary terms, Edit Tier , Edit Tags as separate permissions (#18331)
* Add EditGlossaryTerms Permission

* Fix #18330: Add EDIT_GLOSSARY_TERM permission and enforce EDIT_TIER permisson

* add edit glossary term permission check in UI

* revert EDIT_GLOSSARY_TERMS operation

* Add EDIT_GLOSSARY_TERMS to common operations

* Add EDIT_TIER to common operations

* add default empty array for tags field, as patch calls can run into issues

* Fix tests

* Fix tests

* added glossary terms

* fix conflicts

* fix permission check for data model

* Add EditGlossaryTerms to DataConsumerPolicy

* Add EditGlossaryTerms,EditTier to DataConsumerPolicy

* fix tests

* Fix migrations for EditTier,EditGlossaryTerms

* add edit tier permission to data consumer

* Fix tests

* fix pytests

* missing test_dbt.py

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 10:50:15 -08:00
Sriharsha Chintalapani
783ec62e4c
Add Lineage Export Async endpoint using Search (#18553)
* Add Lineage Export Async endpoint using Search

* Lineage api (#18593)

* add api for lineage export

* update API call for lineage async

* use JsonUtils instead of objectMapper to read lineage search response

---------

Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sweta Agarwalla <105535990+sweta1308@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-11-15 09:11:23 -08:00
IceS2
71f93633ac
Update Glossary Approval Workflow name (#18658) 2024-11-15 16:15:01 +01:00
Mohit Yadav
8c6de372f1
Add Latch for consumers (#18655) 2024-11-15 14:15:35 +05:30
Mohit Yadav
560e28df76
Search Index Issues (#18648)
* Search Index Issues

* Log Thread

* Make logs debug

* Remove Current Therad Name
2024-11-14 22:19:35 +05:30
Aniket Katkar
bfc12feafc
GEN-1976: Alert improvements (#18448)
* Move the alert components to the components folder from pages folder

* auto generated localization changes and linting changes

* Fix the imports

* Add AlertConfigDetails component

* Add tabs in the alert details page

* auto generated localization changes

* Apply permissions for alert and add owner update functionality

* Change the alert update method from PUT to PATCH

* Add the recent events tab

* auto generated localization changes

* Improve the recent events tab for alert details

* Implement the logic to show alert events count on the alert details page

* auto generated localization and linting changes

* Add the pagination logic for the recent events list on the alert details page

* Fix the logic for previous click in NextPreviousWithOffset

* Localization change for other languages

* Hide pagination when not required

* Add pagination and permissions to API; create eventsRecord API; fix timestamp issue

* fix: remove publisherName from test destination message.

* Fix the recent events tab error

* fix: Resolve PATCH API issue with destination ID creation.

* fix: Delete associated alert metrics upon alert deletion

* feat: Detailed HTTP Response for Destination Test

* Add the test external destination functionality

* localization changes for other languages

* fix: Set status for testing destination in EmailPublisher

* Add advanced configurations for webhook destinations

* Localization changes for other languages

* Fix the destination status not showing
Add the POST and PUT radio button in advanced configs

* Localization changes for other languages

* Update the API to fetch the alert event diagnostic counts

* Improve the AlertConfigDetails component to show the alert configs properly

* fix: Display displayName instead of fullyQualifiedName in templates

* Fix the test destination result not showing the first time

* Fix the test destination status not showing for already existing destinations

* Fix the unit tests

* Fix playwright tests

* refactor: Separate sendMessage and sendTestMessage methods; add statusContext

* fix: Use name or fqn as fallback if displayName is null or empty

* fix: upsert successfulChangeEvent

* eliminate the GovernanceWorkflowChangeEvent option from external destination

* Add code comments

* Fix the unit test

---------

Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
Co-authored-by: Siddhanttimeline <siddhanttimeline9@gmail.com>
2024-11-14 17:39:45 +05:30
Mayur Singal
14643f2b50
DI - Add metric name in group name (#18634) 2024-11-14 10:50:19 +05:30
Mayur Singal
f345c4914f
GEN-1933: Separate Description Status for table and column (#18614) 2024-11-14 10:31:44 +05:30
Chirag Madlani
fda0e542e4
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#18620)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-IONETTY-8367012

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-11-13 20:48:07 -08:00
Mohit Yadav
e7e4cbf77a
Fix Producer Sync Issue (#18635) 2024-11-14 02:20:49 +05:30
Karan Hotchandani
887ffab7e1
MINOR - fix config for workflows (#18628)
* fix config for workflows

* minor spacing

* Fix certification spec

* Fix checkstyle

---------

Co-authored-by: Pablo Takara <pjt1991@gmail.com>
2024-11-13 18:23:12 +01:00
Mohit Yadav
ee327ce0c4
Fix Unlimited Queue Size Issue (#18633) 2024-11-13 22:02:34 +05:30
IceS2
128e4e8e93
MINOR: Fix App Insights not processing URL Encoded Data Assets (#18613)
* use 'URLDecoder.decode' before getting entity by name

* Update Test to cover this scenario
2024-11-13 16:14:56 +01:00
Mohit Yadav
30ee4fafb5
Update Default Values (#18621) 2024-11-13 13:24:16 +05:30
IceS2
e5441a9969
Initial implementation for Governance Workflows (#17837)
* Initial implementation for Governance Workflows

* Remove file that shouldn't be there

* Initial implementation for the governance workflows

* Refactor JSONSchema for Governance Workflows

* Refactored WorkflowInstanceStatus to keep the stage properly

* Fix Migration for MySQL for the WorkflowInstanceStates change

* Update WorkflowInstanceState to track all the states propertly

* Initial code tro migrate Request type of Tasks

* copy governance schemas into json directory

* update schema names

* Extract WorkflowInstance into its own resource

* Update WorkflowInstance to have the workflow starting variables

* Changed relatedEntity to be an EntityLink

* First iteration on checkEntityAttributes JSONLogic

* Refactored Workflow into TriggerWorkfow and MainWorkflow

* Fixed WorkflowInstance and WorkflowInstanceState

* Fixed WorkflowInstance and WorkflowInstanceState when process is terminated

* Fix UserTask

* json logic changes to workflow

* Fix Patch request. Add exclude filter to EventBasedEntityTrigger

* update query builder

* Fix Migrations

* Fix Migrations

* modify query builder widget

* fix search props

* update widget and use default fields

* Update edges to always have a true/false condition

* made changes to provider for fields

* push progress

* fix grouping of array field

* Remove duplicated migrations

* Fix migrations

* Fix migrations

* Fix Tests and commit some improvements suggestion

* fix unit tests

* Add Certification Workflow

* Fix Tests

* Fix Tests

* fix tests

* add delay in pagination tests

* fix minor config changes for json logic

* change the glossary page

* Remove Table Certification Workflow seed Data

* add json logic and elastic search parsers

* minor schedule interval changes for workflows

* minor fixes to the parsing logic

* certification ui

* fix tests

* fix tests

* add other search fields

* Add Certification to search index

* Update index mapping for certification

* add quick filter for certification

* update locales

* Fix index mapping

* Add missing index updates

* Fix checkstyle

* increase timeout

* skipping user test

* skipping team test

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
2024-11-13 08:47:44 +01:00
Mayur Singal
8e1c839473
MINOR: Summary Card changes require for UI (#18585) 2024-11-13 12:22:47 +05:30
Mohit Yadav
b6812f732f
Fix Version Comparison Logic (#18616) 2024-11-13 02:52:41 +05:30
Sriharsha Chintalapani
b5f4aee676
Search RBAC improvements (#18591)
* Search RBAC: process multiple view policies that can grant access to different set of entities

* Search RBAC: process multiple view policies that can grant access to different set of entities

* Search RBAC: process multiple view policies that can grant access to different set of entities
2024-11-12 09:11:02 -08:00
Mohit Yadav
9a5dc61ca7
Fix Search Index Contention (#18605)
* Fix Search Index Contention

* Update searchIndexingAppConfig.json

* Missing Error Logs and Stats

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-11-12 20:36:23 +05:30
Sriharsha Chintalapani
5589b9665e
Minor: add searchPageHierarchy method (#18578) 2024-11-11 09:04:50 -08:00
Sriharsha Chintalapani
1c05415321
Improve parallelization in SearchIndexApp (#18556)
* Improve parallelization in SearchIndexApp

* Improve parallelization in SearchIndexApp

* add countdown latch

* typo

* Status issues - #1

* Improve search indexing

* cleanup the code

* schemas

* Status Updates Via WebSockets

* Add Updates

* Make Logs Debug

* show duration in table

* add searchIndexApp params to openmetadata-ops.sh reindex commandline

* Fix code style

* Add Parallel Entity Level Reader

* Make code more readable

* Send App Run Record instead of jobData

* Sned App Run Record WebSocket for Data Insights

* ui: use socket connection to get the real-time updates of application data

* Send WebSocket Final Update

* ui: remove failedRecords, totalRecords and successRecordsfrom entities record

* Invert Condition

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2024-11-11 08:13:54 -08:00
Mayur Singal
943b4efb4d
GEN-1418: DI - Allow Support for Multiple Metrics (#18476) 2024-11-11 14:27:46 +05:30
Siddhant
2437d0124e
Improve Templates for Slack, GChat, and MSTeams (#17771)
* Add default ChangeEvent template and Slack destination test template

* fix conflict

* initial ingestionPipeline template

* clean templates.

* clean templates

* teams templates.

* add getBoldWithSpace()

* java check style

* google chat templates.

* Refactor GChat and Teams message for template data handling for DQ_Template_Section and General_Template_Section.

* Refactor slack message for template data handling for DQ_Template_Section and General_Template_Section.

* improvements gChatMessageDecorator

* improvements slackMessageDecorator

* improve MSTeamsMessageDecorator

* Call templates per entityType and refactor code

* Update EmailPublisher to use testEmail template instead of changeEvent template for sendTestMessage

* add parameterValues sections for teams

* Update SlackMessage to have attachments.
Update DQ template.

* Update dq templates, build dq template data in MessageDecorator

* remove IngestionPipeline template

* move SlackCallbackResource into a separate class. Fix tests.

* simulate timeout and 300.

* remove the validation of messages.

* fix teams MessageDecorator

* remove unused code from SlackMessageDecorator

* fix owners and tags data population in the template

* fix: changes in test case and test case result should load different templates.
2024-11-11 11:05:31 +05:30
sonika-shah
687b564ef6
GEN-1501 : Fix aggregation results on displayName.keyword for dataAssets (#18569) 2024-11-11 10:55:56 +05:30
Siddhant
2d6e21d7a9
fix postgres query. (#18568) 2024-11-10 09:22:23 +00:00
Sriharsha Chintalapani
e7632428f4
Minor: Return pagination of for listHierarchy (#18567) 2024-11-09 23:28:52 -08:00
Sriharsha Chintalapani
a03d915909
Minor: JsonUtils.extractValue shouldn't throw exceptions (#18554) 2024-11-09 09:45:07 -08:00
Siddhant
9d9eccedda
Debuggability Improvements and Fixes (#18564)
* fix: list records from latest to earliest.

* refactor: remove listing unprocessed events from listEvents API.

* refactor: EventSubscriptionDiagnosticInfo.
2024-11-09 16:40:11 +01:00
Sriharsha Chintalapani
7f5f47f618
Add export apis back and new apis should be called exportAsync (#18552)
* Add export apis back and new apis should be called exportAsync

* ui: use exportAsync endpoint for export assets

* fix build failure

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-11-08 10:35:02 -08:00
Siddhant
2f39b76c33
Fix Fetching Successfully Sent Change Events For Debuggability API (#18493)
* fix getSuccessfullySentChangeEventsForAlert

* migrations.

* offset-based pagination

* add startingOffset in EventSubscriptionDiagnosticInfo

* fix missing paginationOffset

* Record successfully sent change events in a separate table, clean up associated successful and failed records on alert deletion.

* Remove oldest 30 records if record count reaches 100 for successful_sent_change_events.

* list unprocessed changeEvents based on alert subscription rules, ensuring only relevant unprocessed events are listed.

* refactor: enhance eventSubscriptionDiagnosticInfo

Added new fields to the eventSubscriptionDiagnosticInfo schema to support detailed diagnostics:
- `relevantUnprocessedEventsCount`: Count of relevant unprocessed events based on alert filtering rules.
- `relevantUnprocessedEventsList`: List of relevant unprocessed events based on alert filtering rules.
- `totalUnprocessedEventsCount`: Count of all unprocessed events.
- `totalUnprocessedEventsList`: List of all unprocessed events.

* Refactor: delete 30 oldest records per alert when the count reaches 100 records

* fix: table creation issue in PostgreSQL

* fix tests.

* scheduleCleanupJob

* remove in-line deletion of records.

* fix postgres table structure issue.
2024-11-08 12:47:16 +01:00
Suman Maharana
da039b197f
Add: Azure Data factory Connector (#18543)
* Added Azure Data factory Connector

* Added Lineage data factory

* removed not required files

* removed not required files

* Removed datafactory ui changes from oss

* resolve merge conflicts

* resolve merge conflicts

* added python requirements
2024-11-08 07:38:45 +01:00
harshsoni2024
01208ef4b4
Connector: add stitch schema (#18452) 2024-11-07 21:08:41 +05:30
sonika-shah
e0d175f78e
GEN-1894 GEN-771 : Improvement - Add displayName field for custom property (#18496) 2024-11-07 16:59:59 +05:30
Sriharsha Chintalapani
9efe137466
add offset limits to hierarchy listing (#18515) 2024-11-07 15:22:11 +05:30
Chirag Madlani
6fa76ecde9
feat: persona based customisation for glossary and glossary term pages (#18454)
* Initial schemas for UI customization

* feat(ui): persona customization for entity details page
including Glossary & Glossary term detail page too

* WIP glossary term details page customization

* update persona epic

* WIP persona customization

* fix errors and warning

* WIP personalize glossary term

* WIP persona

* WIP glossary terms customization

* WIP glossary terms customization

* update glossary term customization

* remove glossary term reference from generic component

* use more of generic provider

* improved UI UX for customization

* complete glossary term customization
start with glossary customization

* apply backend linting

* support customization for glossary page

* fix issue with new tab on glossary page

* address feedbacks

* fix tests

* fix playwright for customize landing page

* fix tests

* fix tests

* address sonar errors

* fix glossary test

* fix glossary test

* fix failing playwright tests

* address comments

* fix conflict error

* add more widgets

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2024-11-07 11:13:12 +05:30
Onkar Ravgan
30fdf8de0b
[ER Diagrams]: Fix EntityRelationship data disappeared after reindexing (#18477)
* fix er diag reindex

* fix col index mapping
2024-11-06 10:25:26 -08:00
Siddhant
36025377fa
fix dependency. (#18536) 2024-11-06 14:02:12 +01:00
sonika-shah
1ade1d4eb6
GEN-1937 : Fix - User count doesn’t update on adding to the team while using search (#18516) 2024-11-06 16:13:23 +05:30
Teddy
d579008c99
GEN 1683 - Add Column Value to be At Expected Location Test (#18524)
* feat: added column value to be in expected location test

* fix: renamed value -> values

* doc: added 1.6 documentatio entry

* style: ran python linting

* fix: move data packaging to pyproject.yaml

* fix: add init file back for data package

* fix: failing test case
2024-11-06 11:17:13 +01:00
Sriharsha Chintalapani
f1ebe816b0
Minor: lineage settings should be readable by all users (#18529) 2024-11-06 15:05:32 +05:30
Mayur Singal
5660a751e3
GEN-2000: Add Support for PowerBI Report Server (#18513) 2024-11-06 14:55:05 +05:30
Siddhant
4d8b303447
fix tests. (#18519) 2024-11-06 11:37:38 +05:30
Sriharsha Chintalapani
e9a888fce7
Fix tests (#18494)
* Fix TestCaseResource testCaseInheritedFields

* Fix TableResourceTest

* Fix tests

* remove setInheritedFields in listing of search entities

* Fix tests

* remove unique entity name creations

* add specific names in search listing tests

* add large size for fetching from ES, as other tests adds to the table_search_index

* Fix tests

* fix tests
2024-11-05 08:55:02 -08:00
Sriharsha Chintalapani
b763091bf6
Make Export CSV Async API, websocket to push data back (#18497)
* Make Export CSV Async API, websocket to push data back

* ui: implement csvExportChannel to receive entity export data via websocket

* ui: update flow to show dialogbox with the response message and once data is available download the file

* minor: remove unused code

* ui: show inline banner for success and error message

* remove unused locale keys

* ui: clear existing csvexport job on sucess

* fix failing UserResourceTest

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
2024-11-05 18:32:41 +05:30
Teddy
3353dd3462
fix: add entityFQN to incident index (#18504) 2024-11-04 18:11:41 +01:00
Sriharsha Chintalapani
f2c9bc6108
Add LineageSettings to control Lineage Default depth, layers (#18498) 2024-11-04 08:50:43 -08:00
sonika-shah
d5497ec2ac
Set max_analyzed_offset in HighlightBuilder to Fix Highlighting Errors for Large Fields (#18495)
* Set max_analyzed_offset in HighlightBuilder to Fix Highlighting Errors for Large Fields

* fix java checkstyle

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-11-04 15:26:49 +05:30
Sriharsha Chintalapani
680ace99b3
Fix #18332: Do not consolidate changes for Parent Entities (#18333)
* Fix #18332: Do not consolidate changes for Parent Entities

* Fix #18332: Do not consolidate changes for Parent Entities

* added no data placeholder when no tags there

* fix failing playwright test due to consolidation changes and added the missing services

* Fix tests

* Fix parent entity change notifications

* Fix tests

* Fix parent entity change notifications

* Fix tests

* Fix tests

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-10-31 11:48:46 -07:00