* Draft Implementation of Custom Workflows
* Multiple Entities in the Same Trigger for Workflow, along with draft implementation of signal id
* Improved User Approval Task and Impl
* Custom Workflows - Draft Implementation 2, improved periodic batch entity filter and other improvements
* feat(governance): Implement transactional custom workflows - improved
This commit introduces a robust, transactional, and extensible framework for custom governance workflows in OpenMetadata.
Key features and improvements include:
Transactional Workflow Management: A new WorkflowTransactionManager ensures atomic operations for creating, updating, and deleting workflow definitions, maintaining consistency between the OpenMetadata database and the Flowable engine.
Safe ID Encoding: Implemented a WorkflowIdEncoder to generate safe, Base64-encoded, and collision-resistant IDs for Flowable processes, preventing errors from ID truncation.
Rollback and Deprecation Tasks:
Added RollbackEntityTask to revert entities to their last approved state.
Introduced DeprecateStaleEntityTask for automated lifecycle management of stale assets.
Enhanced Workflow Engine:
Improved WorkflowHandler to validate workflow definitions before deployment.
Added new custom functions to the rule engine for checking entity update timestamps and calculating field completeness scores.
CI/CD and Build Improvements:
Updated the CI Dockerfile with a multi-stage build and refined dependency installation.
Modified POM files to include necessary dependencies for new features.
* Adding DataCompleteness Task Node, Flowable Debug logs
* Transaction handling for Custom Workflow - Initial Draft
* add new tasks to node definition interface
* Update generated TypeScript types
* Draft Implementation of Multi Reviewer Approval Task with Migration
* Update generated TypeScript types
* Transaction handling fix, id truncation fix by migration, feed repo fix for multi reviewer pattern, copilot comments
* Update generated TypeScript types
* Fixed Multi Reviewer approval to take consideration of the namespaced variables, Fixed RollBackEntity task to follow subprocess like other automated tasks, copilot nitpicks
* Remove conditionalSetEntityAttributes as it is not needed anymore
* Update generated TypeScript types
* Completely remove the setConditionalAttributes to fix compilation errors
* Removed the comments in the schemaChanges
* Created a new Task called CreateDetailedApprovalTaskImpl, Fixed RollBackEntityTask to roll back to either Approved or Rejected state, use namespaced variables, Updated the workflow handler to resolve the tasks and remove them from the user who has approved the task in the feed during multi reviewer approval, TransactionManager updated to keep Transactions in place, Improve the Validation Node input in the Mainworkflow java to handle proper graph traversal, Find Proper Termination Message for the two user events that are conflicting, Include Message in the request approval thread task for proper messages
* Update generated TypeScript types
* Rendering of messages in task thread
* Fix PeriodicBatchEntityTriggerTask to separate workflows based on entities Draft, Fix: SetEntityAttributes Impl to handle multiple fields, Fix: DataCompletenessTask Draft
* Fix DataCompletenessTask BoundaryEvent in Flowable
* Introduced Wf Deployment Strategy for transaction Handling, Improved user tasks for better termination event naming, fix periodic batch entity trigger to fetch the entity instead of all the entities in the trigger, Migrated GlossaryApprovalWorkflow.json with new nodes and edges, Fixed test cases and some edge cases
* Update generated TypeScript types
* Added performTasks for TagRepository and DataProductRepository, Removed test api, removed unnecessary change from workflow json, improved DataCompletenessImpl, Improved CreateDetailedApprovalTaskImpl to show what updated and not updated in the thread response
* Remove Entity Certification and Entity Status tasks and make use of generic set entity attribute task
* Update generated TypeScript types
* Fix the compilation issues!
* Remove setCertification and setEntityAttributes from createWorkflowDefinition
* Test cases for custom workflows related to glossaryTermApprovalWorkflow
* Test cases for custom workflows
* Changed info to debug logs
* Update generated TypeScript types
* DetailedUserApprovalTask changed to reviewChange task, Have validations for workflows where user approval tasks can only be available for entities that support reviewers
* Fix compilation issues and mvn spotless apply
* Update generated TypeScript types
* Remove Extra assignees from user tasks
* Update generated TypeScript types
* Replace Tags/GlossaryTerms during Mutual Exclusivity and Append when there is no Mutual Exclusivity
* Workflow Transaction Manager to handle authorization as part of its functionality, Added Validation for workflows using /validate endpoint
* Increase the Flowable's polling time to 60 seconds to poll timer and async jobs table
* Update generated TypeScript types
* Enum for UserTask Termination Messages, Removed WorkflowUtils to use WorkflowHandler's terminateDuplicate Instances, Approval and rejecters list in the variables, using namespaced variables for updatedBy
* Reverted the enum for userTaskType for now
* Added new tests for dataContract, dataProduct and Tag for user approval tasks
* Glossary Approval Workflow changed to handle jsonLogic according to UI expectations
* Added a new Task type for change description tasks to review and suggest the changes, Added checkUpdatedByReviewer incase of perfom tasks, increase wf test timeout
* Update generated TypeScript types
* TaskWorkflow constructor public to be used by Collate Repo
* AutoApproveServiceTaskImpl incase the assignees are not there for the userTask with ExlusiveGateway builder with hasAssignees
* Fix Compilation Issues - Upgrade of deps to lang3
* ExclusiveGatewayBuilder set exclusive as true, and other minor changes for test and approval tasks
* Added a different wait mechanism for user tasks in WorkflowDefinitionResourceTest.java
* Combined UserApprovalTask and ChangeReviewTask into one to handle suggestions in the same task!
* Update generated TypeScript types
* Test Case Approval Workflow, Removing orphaned changeReviewTask, Test case fix and new test cases
* Update generated TypeScript types
* Treat empty strings, arrays as missing by default
* Update generated TypeScript types
* fix compilation issues by changing the schema properly
* Remove Stale ChangeReview Task
* Update generated TypeScript types
* Enhanced User Approval task to show changed fields along with what changed and the task resolve function as well
* Update generated TypeScript types
* Add Knowledge center page to workflows event consumer, remove legacy backward compatibility for triggers, lenient validations for workflows when there are no nodes
* EntitySpecific Filters for EventBased Trigger, Removing Backward Compatibility logic for filters, Removed unnecessary comments and execution variables, Handle Structured task resolver for dataProduct, dataContract, tag and testCase, Modified GlossaryApprovalWorkflow.json and improved the MigrationUtil
* Update generated TypeScript types
* Bring back entitycertification and glosarystatus task for backward compatibility
* Update generated TypeScript types
* Filter is a map, entity specific filters are stringified, have certification and glossary status tasks in nodeinterface
* Merge Main
* Remove Suggestions for RequestApproval
* Update generated TypeScript types
* Remove Old Deployments of Periodic Batch Entity Trigger and use hiphen to trigger to avoid accidental triggering
* Resolve Merge conflicts, Java Checkstyle
* Update generated TypeScript types
* Fix Migrations
* Added alter table queries in 1.6.0 to avoid cached plan queries in flowable
* Increase timeout in workflow definition resource test
* Increase polling in workflow definition resource test
* Fix java Checkstyle
* comment the flaky test
* COmmented out Flaky Test Cases, fixed a bug on team reviewers
* Resolve java checkstyle after resolving conflicts
* add updated at field in json logic for glossary
* update fields
* add version field
* Update the rule config of new term node with "and" and update the migration as well
* add equal not equal op
* Delete the trigger workflows using like condition
* Defensive Fallback for multiple task instances created by race condition, Terminate duplicate instances of main workflows
* reverting operators
* Approval Capabilities for Metrics
* Update generated TypeScript types
* Move Migrations to 1.10.1 from 1.10.0
* Removed the extra spaces in Migration 1.10
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* Fix#23754: test suite owner is getting blocked when trying to add test cases to a bundle suite
* add playwright test for test suite (#23768)
* addressing comment
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* Update `TableDiffParamsSetter` to move data at table level
This means that `key_columns` and `extra_columns` will be defined per table instead of "globally", just like `data_diff` expects
* Update `TableDiffValidator` to use table's `key_columns`
Call `data_diff` and run validations using each table's `key_columns`
* Create migration to update `tableDiff` test definition
* Fix Playwright test
* Add the migration classes and data for recognizers
This is so that we can run a migration that sets `json->recognizers` of `PII.Sensitive` and `PII.NonSensitive` tags from json values.
The issue with normal migrations was that the value of recognizers was too long to be persisted in the server migrations log.
Created a common `migration.utils.v1110.MigrationProcessBase`
* Ensure building automatically with the right parameters
* Update typescript types
* Fix: Domain assets count mismatch between API and UI
* exclude SearchDerivedFields from reindex
* refactor fallback and have new field assetsCount
* Update generated TypeScript types
* refactor: clean up groupEntitiesByType function
* add tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix#23710: GlossaryTerm - Circular references cause API to hang: Self-referential or circular parent relationships in glossary terms cause infinite loops when querying with directChildrenOf parameter
* Fix Test Cases
* Prevent Renaming the term to duplicates
* Visited array in invalidate cache
* Fix type errors
* Fix json type check
* Update the tsconfig
* update the tsconfig.json to align with ui
* Revert "update the tsconfig.json to align with ui"
This reverts commit 7dc4e3d6f0a31924d013eb1b1abcda80c6a02f99.
---------
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
Co-authored-by: Ram Narayan Balaji <81347100+yan-3005@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* MINOR - Add SLA fields
* Update generated TypeScript types
* added ui support for the timezone and columnName
* MINOR - Add SLA fields
* Update generated TypeScript types
* change the timezone as the enum and added field placeholder
* display column name on ui
* Fix unit test
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* Add `reason` property to `TagLabel`
This is to understand what score was used for selecting the entity
* Build `TagLabel`s with `reason`
* Increase `PIIProcessor._tolerance`
This is so we correctly filter out low scores from classifiers while still maintaining the normalization that filters out confusing outcomes.
e.g: an output with scores 0.3, 0.7 and 0.75, would initially filter the 0.3 and then discard the other two because they're both relatively high results.
* Make database and DAO changes needed to persist `TagLabel.reason`
* Update generated TypeScript types
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Setup SPI Pattern for seed data path on OM commercial version
* Implement and test Notification Template validation and reset endpoints
* Relocate system-default notification templates
* Update generated TypeScript types
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
* MINOR - Rename contract consumers
* Update generated TypeScript types
* supported contract security data display ui with test
* fix the row_filter data being send on API if they don't posses data
* fix the playwright test because of breaking of adding new rule
* reverted the commented code
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* Add support for translations in multi lang
* Add Tag Feedback System
* Update generated TypeScript types
* Fix typing issues and add tests to reocgnizer factory
* Updated `TagResourceTest.assertFieldChange` to fix broken test
This is because change description values had been serialized into strings and for some reason the keys ended up in a different order. So instead of performing String comparison, we do Json comparisons
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eugenio Doñaque <eugenio.donaque@getcollate.io>
* Field level validation in SSO configuration
* Update generated TypeScript types
* Ldap field error response
* add form field level error validations
* field level error highlight style fix
* fix array field level error highlight issue
* minor style fix for array fields
* resolve merge conflict
* error highlight for text area fields
* fix test
* Fixed tests and improvement
* added unit test
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: shrushti2000 <shrushtipolekar@gmail.com>
* Fix#23445: User with Owner role unable to update Incident Status via API
* fix: failing build
* fix: permission on test case incidents
* playwright: added e2e test for issue #23445 (#23459)
* fix: entitylink parsing for entity resource
* fix: restore severity state in PATCH test
---------
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* Revert Entity Status Migrations as we already have the default value
* Batch Migrations and by default show entityStatus field in the response
* Setting default value of enum in the status.json and fixing test cases for new default enum UNPROCESSED
* Update generated TypeScript types
* DataContract by default has Unprocessed status, Overriden method in the GlossaryTermRepository
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add the drive service settings option
* Fix the ci error
* Add Directory Details page
* Add Explore page support for new drive assets
* Add Directory Children table
* Update page.json
* Add File details page
* Add spreadsheet details page
* Add Worksheet details page
* Update explore tree name
* Fix the search suggestions not showing for new assets
* Add new assets in missing setting options
* Update service util class base for drive service
* Add version support for directory
* Add version pages for File, Spreadsheet and Worksheet
* Add FQN for worksheet columns
* Add columns table for the worksheet
* Fix the Spreadsheet, Directory and Worksheet version pages not showing children.
* Fix the domain not updating for drive assets
* Fix announcement not working for drive assets
* Add playwright for Directory asset
* Fix the index mappings and failing playwright test for directory
* Add google drive service icon
* Fix the delete and restore functionality for drive assets
* Fix patch tags for worksheet and add tests
* Fix failures
* search indices
* Update service details page for drive service
* Add playwright tests for remaining drive assets
* Fix listing APIs to return when root=true mentioned
* Update API to fetch drive service assets and fix playwright for Worksheet service
* Fix the index mappings
* Add the new fields for advanced search
* Add the playwright tests for drive assets
* fix: drive service test connection
* fix: drive service test connection
* Fix the ServiceDetailsPage tests
* Fix the playwright tests
* Fix the search settings issue for drive assets
* add missing icons, fix search index issue for worksheet, resolve the sonar issues
* Fix the entity permission spec
* Fix FileResourceTest to create GoogleDrive instead of SharePoint to match the type creation in the test
* Work on comments
---------
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Keshav Mohta <keshavmohta09@gmail.com>
Co-authored-by: Ram Narayan Balaji <ramnarayanb3005@gmail.com>
* Domain Only Access Role to be evaluated as part of PolicyEvaluator
* Domain Only Access Role to be evaluated as part of PolicyEvaluator
* Cleanup unnecessary comments
* Add migration fix for domain only policy
* fix playwright domain rbac
* fix hasDomain role issues for TestCaseResourceContext
* allow user to create only with their domain
* move DomainOnlyAccessPolicy migrations to 1.9.10 from 1.9.9
* For resources that don't support domains (like DataInsights), always returns true
* fix adding test case to bundleSuite
* revert supportsDomains changes in RuleEvaluator and handle it in postFiltering steps
* Inherit domains when creating task from test case incident, and skip few entities for domain check + other remaining DQ related domain fixes
* for CreateResourceContext consider assigned domains + inherited domains
* fix permission for table metrics page
* remove SKIP_DOMAIN_CHECK_ENTITY_LIST
* fix test
---------
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
