* Draft Implementation of Custom Workflows
* Multiple Entities in the Same Trigger for Workflow, along with draft implementation of signal id
* Improved User Approval Task and Impl
* Custom Workflows - Draft Implementation 2, improved periodic batch entity filter and other improvements
* feat(governance): Implement transactional custom workflows - improved
This commit introduces a robust, transactional, and extensible framework for custom governance workflows in OpenMetadata.
Key features and improvements include:
Transactional Workflow Management: A new WorkflowTransactionManager ensures atomic operations for creating, updating, and deleting workflow definitions, maintaining consistency between the OpenMetadata database and the Flowable engine.
Safe ID Encoding: Implemented a WorkflowIdEncoder to generate safe, Base64-encoded, and collision-resistant IDs for Flowable processes, preventing errors from ID truncation.
Rollback and Deprecation Tasks:
Added RollbackEntityTask to revert entities to their last approved state.
Introduced DeprecateStaleEntityTask for automated lifecycle management of stale assets.
Enhanced Workflow Engine:
Improved WorkflowHandler to validate workflow definitions before deployment.
Added new custom functions to the rule engine for checking entity update timestamps and calculating field completeness scores.
CI/CD and Build Improvements:
Updated the CI Dockerfile with a multi-stage build and refined dependency installation.
Modified POM files to include necessary dependencies for new features.
* Adding DataCompleteness Task Node, Flowable Debug logs
* Transaction handling for Custom Workflow - Initial Draft
* add new tasks to node definition interface
* Update generated TypeScript types
* Draft Implementation of Multi Reviewer Approval Task with Migration
* Update generated TypeScript types
* Transaction handling fix, id truncation fix by migration, feed repo fix for multi reviewer pattern, copilot comments
* Update generated TypeScript types
* Fixed Multi Reviewer approval to take consideration of the namespaced variables, Fixed RollBackEntity task to follow subprocess like other automated tasks, copilot nitpicks
* Remove conditionalSetEntityAttributes as it is not needed anymore
* Update generated TypeScript types
* Completely remove the setConditionalAttributes to fix compilation errors
* Removed the comments in the schemaChanges
* Created a new Task called CreateDetailedApprovalTaskImpl, Fixed RollBackEntityTask to roll back to either Approved or Rejected state, use namespaced variables, Updated the workflow handler to resolve the tasks and remove them from the user who has approved the task in the feed during multi reviewer approval, TransactionManager updated to keep Transactions in place, Improve the Validation Node input in the Mainworkflow java to handle proper graph traversal, Find Proper Termination Message for the two user events that are conflicting, Include Message in the request approval thread task for proper messages
* Update generated TypeScript types
* Rendering of messages in task thread
* Fix PeriodicBatchEntityTriggerTask to separate workflows based on entities Draft, Fix: SetEntityAttributes Impl to handle multiple fields, Fix: DataCompletenessTask Draft
* Fix DataCompletenessTask BoundaryEvent in Flowable
* Introduced Wf Deployment Strategy for transaction Handling, Improved user tasks for better termination event naming, fix periodic batch entity trigger to fetch the entity instead of all the entities in the trigger, Migrated GlossaryApprovalWorkflow.json with new nodes and edges, Fixed test cases and some edge cases
* Update generated TypeScript types
* Added performTasks for TagRepository and DataProductRepository, Removed test api, removed unnecessary change from workflow json, improved DataCompletenessImpl, Improved CreateDetailedApprovalTaskImpl to show what updated and not updated in the thread response
* Remove Entity Certification and Entity Status tasks and make use of generic set entity attribute task
* Update generated TypeScript types
* Fix the compilation issues!
* Remove setCertification and setEntityAttributes from createWorkflowDefinition
* Test cases for custom workflows related to glossaryTermApprovalWorkflow
* Test cases for custom workflows
* Changed info to debug logs
* Update generated TypeScript types
* DetailedUserApprovalTask changed to reviewChange task, Have validations for workflows where user approval tasks can only be available for entities that support reviewers
* Fix compilation issues and mvn spotless apply
* Update generated TypeScript types
* Remove Extra assignees from user tasks
* Update generated TypeScript types
* Replace Tags/GlossaryTerms during Mutual Exclusivity and Append when there is no Mutual Exclusivity
* Workflow Transaction Manager to handle authorization as part of its functionality, Added Validation for workflows using /validate endpoint
* Increase the Flowable's polling time to 60 seconds to poll timer and async jobs table
* Update generated TypeScript types
* Enum for UserTask Termination Messages, Removed WorkflowUtils to use WorkflowHandler's terminateDuplicate Instances, Approval and rejecters list in the variables, using namespaced variables for updatedBy
* Reverted the enum for userTaskType for now
* Added new tests for dataContract, dataProduct and Tag for user approval tasks
* Glossary Approval Workflow changed to handle jsonLogic according to UI expectations
* Added a new Task type for change description tasks to review and suggest the changes, Added checkUpdatedByReviewer incase of perfom tasks, increase wf test timeout
* Update generated TypeScript types
* TaskWorkflow constructor public to be used by Collate Repo
* AutoApproveServiceTaskImpl incase the assignees are not there for the userTask with ExlusiveGateway builder with hasAssignees
* Fix Compilation Issues - Upgrade of deps to lang3
* ExclusiveGatewayBuilder set exclusive as true, and other minor changes for test and approval tasks
* Added a different wait mechanism for user tasks in WorkflowDefinitionResourceTest.java
* Combined UserApprovalTask and ChangeReviewTask into one to handle suggestions in the same task!
* Update generated TypeScript types
* Test Case Approval Workflow, Removing orphaned changeReviewTask, Test case fix and new test cases
* Update generated TypeScript types
* Treat empty strings, arrays as missing by default
* Update generated TypeScript types
* fix compilation issues by changing the schema properly
* Remove Stale ChangeReview Task
* Update generated TypeScript types
* Enhanced User Approval task to show changed fields along with what changed and the task resolve function as well
* Update generated TypeScript types
* Add Knowledge center page to workflows event consumer, remove legacy backward compatibility for triggers, lenient validations for workflows when there are no nodes
* EntitySpecific Filters for EventBased Trigger, Removing Backward Compatibility logic for filters, Removed unnecessary comments and execution variables, Handle Structured task resolver for dataProduct, dataContract, tag and testCase, Modified GlossaryApprovalWorkflow.json and improved the MigrationUtil
* Update generated TypeScript types
* Bring back entitycertification and glosarystatus task for backward compatibility
* Update generated TypeScript types
* Filter is a map, entity specific filters are stringified, have certification and glossary status tasks in nodeinterface
* Merge Main
* Remove Suggestions for RequestApproval
* Update generated TypeScript types
* Remove Old Deployments of Periodic Batch Entity Trigger and use hiphen to trigger to avoid accidental triggering
* Resolve Merge conflicts, Java Checkstyle
* Update generated TypeScript types
* Fix Migrations
* Added alter table queries in 1.6.0 to avoid cached plan queries in flowable
* Increase timeout in workflow definition resource test
* Increase polling in workflow definition resource test
* Fix java Checkstyle
* comment the flaky test
* COmmented out Flaky Test Cases, fixed a bug on team reviewers
* Resolve java checkstyle after resolving conflicts
* add updated at field in json logic for glossary
* update fields
* add version field
* Update the rule config of new term node with "and" and update the migration as well
* add equal not equal op
* Delete the trigger workflows using like condition
* Defensive Fallback for multiple task instances created by race condition, Terminate duplicate instances of main workflows
* reverting operators
* Approval Capabilities for Metrics
* Update generated TypeScript types
* Move Migrations to 1.10.1 from 1.10.0
* Removed the extra spaces in Migration 1.10
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* fix: combined index on offset and eventTime in changeEvent table
* fix: correct syntax error in Postgres query.
* fix: correct syntax error in Postgres query.
* fix: correct syntax error in Postgres query.
* fix: Update configuration for activity feed.
* fix: Migrations
* remove the customProperty and testSuite feed beautification case from the feed
* fix: Remove custom properties from the source
* fix: add metric in the matchAnySource.
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* Move the alert components to the components folder from pages folder
* auto generated localization changes and linting changes
* Fix the imports
* Add AlertConfigDetails component
* Add tabs in the alert details page
* auto generated localization changes
* Apply permissions for alert and add owner update functionality
* Change the alert update method from PUT to PATCH
* Add the recent events tab
* auto generated localization changes
* Improve the recent events tab for alert details
* Implement the logic to show alert events count on the alert details page
* auto generated localization and linting changes
* Add the pagination logic for the recent events list on the alert details page
* Fix the logic for previous click in NextPreviousWithOffset
* Localization change for other languages
* Hide pagination when not required
* Add pagination and permissions to API; create eventsRecord API; fix timestamp issue
* fix: remove publisherName from test destination message.
* Fix the recent events tab error
* fix: Resolve PATCH API issue with destination ID creation.
* fix: Delete associated alert metrics upon alert deletion
* feat: Detailed HTTP Response for Destination Test
* Add the test external destination functionality
* localization changes for other languages
* fix: Set status for testing destination in EmailPublisher
* Add advanced configurations for webhook destinations
* Localization changes for other languages
* Fix the destination status not showing
Add the POST and PUT radio button in advanced configs
* Localization changes for other languages
* Update the API to fetch the alert event diagnostic counts
* Improve the AlertConfigDetails component to show the alert configs properly
* fix: Display displayName instead of fullyQualifiedName in templates
* Fix the test destination result not showing the first time
* Fix the test destination status not showing for already existing destinations
* Fix the unit tests
* Fix playwright tests
* refactor: Separate sendMessage and sendTestMessage methods; add statusContext
* fix: Use name or fqn as fallback if displayName is null or empty
* fix: upsert successfulChangeEvent
* eliminate the GovernanceWorkflowChangeEvent option from external destination
* Add code comments
* Fix the unit test
---------
Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
Co-authored-by: Siddhanttimeline <siddhanttimeline9@gmail.com>
* Initial implementation for Governance Workflows
* Remove file that shouldn't be there
* Initial implementation for the governance workflows
* Refactor JSONSchema for Governance Workflows
* Refactored WorkflowInstanceStatus to keep the stage properly
* Fix Migration for MySQL for the WorkflowInstanceStates change
* Update WorkflowInstanceState to track all the states propertly
* Initial code tro migrate Request type of Tasks
* copy governance schemas into json directory
* update schema names
* Extract WorkflowInstance into its own resource
* Update WorkflowInstance to have the workflow starting variables
* Changed relatedEntity to be an EntityLink
* First iteration on checkEntityAttributes JSONLogic
* Refactored Workflow into TriggerWorkfow and MainWorkflow
* Fixed WorkflowInstance and WorkflowInstanceState
* Fixed WorkflowInstance and WorkflowInstanceState when process is terminated
* Fix UserTask
* json logic changes to workflow
* Fix Patch request. Add exclude filter to EventBasedEntityTrigger
* update query builder
* Fix Migrations
* Fix Migrations
* modify query builder widget
* fix search props
* update widget and use default fields
* Update edges to always have a true/false condition
* made changes to provider for fields
* push progress
* fix grouping of array field
* Remove duplicated migrations
* Fix migrations
* Fix migrations
* Fix Tests and commit some improvements suggestion
* fix unit tests
* Add Certification Workflow
* Fix Tests
* Fix Tests
* fix tests
* add delay in pagination tests
* fix minor config changes for json logic
* change the glossary page
* Remove Table Certification Workflow seed Data
* add json logic and elastic search parsers
* minor schedule interval changes for workflows
* minor fixes to the parsing logic
* certification ui
* fix tests
* fix tests
* add other search fields
* Add Certification to search index
* Update index mapping for certification
* add quick filter for certification
* update locales
* Fix index mapping
* Add missing index updates
* Fix checkstyle
* increase timeout
* skipping user test
* skipping team test
---------
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
* fix getSuccessfullySentChangeEventsForAlert
* migrations.
* offset-based pagination
* add startingOffset in EventSubscriptionDiagnosticInfo
* fix missing paginationOffset
* Record successfully sent change events in a separate table, clean up associated successful and failed records on alert deletion.
* Remove oldest 30 records if record count reaches 100 for successful_sent_change_events.
* list unprocessed changeEvents based on alert subscription rules, ensuring only relevant unprocessed events are listed.
* refactor: enhance eventSubscriptionDiagnosticInfo
Added new fields to the eventSubscriptionDiagnosticInfo schema to support detailed diagnostics:
- `relevantUnprocessedEventsCount`: Count of relevant unprocessed events based on alert filtering rules.
- `relevantUnprocessedEventsList`: List of relevant unprocessed events based on alert filtering rules.
- `totalUnprocessedEventsCount`: Count of all unprocessed events.
- `totalUnprocessedEventsList`: List of all unprocessed events.
* Refactor: delete 30 oldest records per alert when the count reaches 100 records
* fix: table creation issue in PostgreSQL
* fix tests.
* scheduleCleanupJob
* remove in-line deletion of records.
* fix postgres table structure issue.
* create EventSubscriptionDiagnosticInfo
* API for failedEvents
* migrations
* add index on source
* list all failedEvents optionally filtered by source
* refactor
* refactor
* getSuccessfullySentChangeEventsForAlert API
* increase defaultValue of limit to 100
* resolve conflicts
* listEvents API with 'failed', 'successful,' and 'unprocessed' query parameters
* fix description
* eventSource as enums
* refactor name.
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* MINOR - Renaming MetaPilot to CollateAI
* MINOR - Renaming MetaPilot to CollateAI
* MINOR - Renaming MetaPilot to CollateAI
* MINOR - Renaming MetaPilot to CollateAI
* fix
* moving limits migrations for 1.5.9
* fix(ui): pagination issue with search in services list page
* Add Apps Data store
* Revert Services Changes
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* feat: indexed test case results
* feat: added indexation logic for test case results
* style: ran java linting
* fix: IDE warnigns
* chore: added test case results migration
* style: ran java linting
* fix: postgres migration column json ref
* empty commit to trigger queued
* chore: extracted test case results to its own resource
* chore: fix failing tests
* chore: move testCaseResult state from testSuite and testCase to dynamic field fetched from test case results search index
* chore: clean up test case repository
* style: ran java linting
* chore: removed testCaseResultSummary and testCaseResult state from db
* fix: test failures
* chore: fix index mapping type for result value
* chore: fix test failure
* feat: indexed test case results
* feat: added indexation logic for test case results
* style: ran java linting
* fix: IDE warnigns
* chore: added test case results migration
* style: ran java linting
* fix: postgres migration column json ref
* empty commit to trigger queued
* fix: added index on id field
