* Initial Implementation for Change Events for Workflow Generated Changes and Manual Task Resolutions
* Test case
* Add impersonatedBy to changeEvent schema, changeEvent generations, workflow consumer filtering, always override impersonatedBy even if not null
* Update generated TypeScript types
* Improved debug logging in WorkflowEventConsumer and tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Feat(ui): support contract tab for other entities
* fix the tab being showed in entities along with the next and previous button with the functionality
* modify the contract badge status to all entities in data asset header
* supported topic,dashboardDataModel,ApiEndpoint in schema tab with the json changes
* Update generated TypeScript types
* fix the contract.ts file being deleted
* re-push the missing data in dataContract.json
* supported the contract tab in persona changes and also supported for Chart Entity
* fix the unit test and type error in schema for contract tab
* added playwright test for dataContract for other entities
* fix the sonar test
* added databas and schema contract test along with the previous failure
* added Database and DatabaseSchema contract test and fix other test issue due to merge conflict
* commeted out the tab support for charts now, since BE is still pending on it
* code polish, render schema table on condition with test and schema form default expand all keys
* Update generated TypeScript types
* fix the button props naming
* fix the buttons labels
* modify the playwright and include support for all entities checking
* fix the schema fields for other entities
* added support for fiel, directory, spreadsheet and worksheet and fix some playwright cases
* fix unit and playwright test
* fix some sonar issue
* fix some code smell and minor improvement
* fix unit test
* remove unwanted file
---------
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Update schemas
* Remove the allowedEmailRegistrationDomains, allowedDomains, useRolesFromProvider fields from hidden state
* Refactor the SSO Configuration Form and add tests
* Fix code smells and refactor the code for SSOConfigurationForm
* Fix the code smells
* Remove the custom functions to create patch for SSO configurations
* Add mock for structuredClone
* Update generated TypeScript types
* Empty commit
* Fix the unnecessary cleanup of data before saving
* Update the default values for oidc configs
* Fix unit test
* Remove the unnecessary util function
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Implement Ingestion side to return a flag when all values are unique
* Update generated TypeScript types
* feat: Enhance CardinalityDistributionChart to display messages when all values are unique
- Added logic to check if all values are unique for both first day and current day data.
- Implemented a placeholder message when all values are unique, indicating no distribution available.
- Updated tests to cover scenarios for unique values and ensure correct rendering of charts and messages.
- Added localization for the new message in multiple languages.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* refactor: used hashing to reduce api calls, replace distinct with group by to optimize lineage queries & minor code optimizations
* Update generated TypeScript types
* fix: self.job_table_lineage defaultdict function
* refactor: improved hashing
* fix: added _table_lookup_cache and _dlt_table_cache in tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix: migration
* fix: playwright test DBT -> dbt
* feat: added rentention for profile and dq data
* feat: fix failing tests
* feat: address error in postgres delete sql
* feat: fixed missing parameter in psql query
* fix: added the deletion step in test case
* feat: fixed postgres query for deletion before cutoffs
* Fix: Add explicit fqnHash and lineage related mappings with ignore_above 512 to prevent truncation in Elasticsearch indices
* increase limit for other lineage field mapping
* add missing upstreamLineage in jp, ru,zh index
* Fix: Search Slowness when painless scripts aggregates for terms and classifications
* Fix Sql
* Add fields to security service index
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
* Search Query for Tags to restrict tags other than Approved or Unprocessed
* Restrict tag search query in the UI to not show tags other than Approved/UnProcessed
* Fxi Search Queries for Tag Search and use protected for tests
* add activity feed
* Refactored the error message of validation, refactored code
* use generic widgets in data product details
* Update generated TypeScript types
* commit progress
* add activity feed to domains
* commit progress
* add status badge
* commit progress
* Add customizable layout for tag class base
* nit
* Fix data product reviewer fetch
* Add reviewer field in tag page
* Add knowledge center task route for notification
* fix knowledge center page enum
* nit
* Fix tag page unit test
* fix test
* fix expert field visiable issue
* fix expert field
* Fix tag and add domain entity test
* nit
* fix test is for domain add
* fix tags owner test id
* fix tag disabled permission issue
* Add Conversation feature for Data Products
* Update generated TypeScript types
* Merge branch 'main' into entity-specific-restrictions
* nit
* fix glossary integration test
---------
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* feat: add owner assignment support at metadata ingestion level
* docs: Translate comments to English in test_owner
* refactor: move the test_owner-related files into correct positions
* feat: Add support for more source types
* Revert "feat: Add support for more source types"
This reverts commit a7649dcb3204cf98b7f4f9be02fbb982d2532193.
* feat: Add owner field support in sourceConfig for Database and Dashboard ingestion (fixes#22392)
* refactor code with the required style
* add owner field in related json file
* feat: add topology-based owner config for database/schema/table
* Format the code by the pre-commit tools
* fix some errors
* add a doc to explain this feature
* translate all Chinese comments to English and consolidate documentation
* remove redundant code
* refactor code
* refactor code
* refactor code
* refactor code
* Add some tests for owner-config and enhance this feat
* Add some tests for owner-config and enhance this feat
* fix some error
* fix some error
* refactor code
* Remove the yaml and bash test files and test owner config with pytest style
* format the python code
* refactor ingestion code
* refactor code
* fix some error in test_owner_utils
---------
Co-authored-by: Ma,Yutao <yutao.ma@sap.com>
* Initial implementation for Dimensionality on Data Quality Tests
* Fix ColumnValuesToBeUnique and create TestCaseResult API
* Refactor dimension result
* Initial E2E Implementation without Impact Score
* Dimensionality Thin Slice
* Update generated TypeScript types
* Update generated TypeScript types
* Removed useless method to use the one we already had
* Fix Pandas Dimensionality checks
* Remove useless comments
* Implement PR comments, fix Tests
* Improve the code a bit
* Fix imports
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add assets API and deprecate inline assets field for Domain and Dataproduct
* fix mvn test
* fix py test and add new tests
* fix py test
* fix py test
* fix timeout for workflow test
* address pr feedback
* Update generated TypeScript types
* minor- remove unused function
---------
Co-authored-by: Bhanu Agrawal <bhanuagrawal2018@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* refactor searchQuery to avoid AND and OR operators
* fix searchQuery function
* fix serviceDetails.tsx
* refactor for searchQuery
* update searchData calls with searchQuery
* fix unit test
* refactor queryFilter for dataContractUtil.ts
* fix authProvider for global domain filter
* address PR comments
* revert dataProduct fqn change
* refactor all query filtersusing getTermQuery function
* fix test
* fix tests
* fix e2e tests
* fix test
* fix service listing
* fix test
* fix service spec
* fix glossary spec and tag spec
* fix domains spec
* fix tag spec
* fix failing specs
* fix failed specs
* fix failing tests
* fix domain spec
* fix classification & service listing spec
* change query filter for following & change message for no online users
* add missing followers field in entity index mapping
* add missing owners field in entity index mapping
* address comments
* fix failing test
---------
Co-authored-by: Pranita <pfulsundar8@gmail.com>
Co-authored-by: Akash Jain <15995028+akash-jain-10@users.noreply.github.com>
Co-authored-by: sonikashah <sonikashah94@gmail.com>
* Setup new client for ES/OS
* Migrated createIndex and addIndexAlias methods
* Migrated createAliases method to new ES/OS client
* Migrated updateIndex and deleteIndex methods
* Updated indexExists methods
* refactor: extract index management operations into dedicated manager classes
* Add tests for ElasticSearch and OpenSearch index managers
* chore: fix code style issues"
* Added integeration tests for ES/OS index manager
* Fix log level
* Fixed priority key mapping for test_case_result_index
* fix test to use field name in the query as domains.id not domain.id
* Fixed failing tests
* Set headers so 8.x client can work with 7.17x or higher server for ES
* Added OpenSearch compatible stemmer configuration
* Fix java code style
* Fixed java checkstyle issue
* Added support for 7.17.x backward compatibility
* Fixed failing tests
* Updated client availability checks in both index manager classes to use the consistent !isClientAvailable pattern
* Removed comment
* chore: Fix line formatting in Playwright test files
- Adjusted line length in Domains.spec.ts for better readability
- Fixed indentation and formatting in Lineage.spec.ts
- Added missing newline at end of files
* Added new line
* Migrated other index related methods and add tests
* Draft Implementation of Custom Workflows
* Multiple Entities in the Same Trigger for Workflow, along with draft implementation of signal id
* Improved User Approval Task and Impl
* Custom Workflows - Draft Implementation 2, improved periodic batch entity filter and other improvements
* feat(governance): Implement transactional custom workflows - improved
This commit introduces a robust, transactional, and extensible framework for custom governance workflows in OpenMetadata.
Key features and improvements include:
Transactional Workflow Management: A new WorkflowTransactionManager ensures atomic operations for creating, updating, and deleting workflow definitions, maintaining consistency between the OpenMetadata database and the Flowable engine.
Safe ID Encoding: Implemented a WorkflowIdEncoder to generate safe, Base64-encoded, and collision-resistant IDs for Flowable processes, preventing errors from ID truncation.
Rollback and Deprecation Tasks:
Added RollbackEntityTask to revert entities to their last approved state.
Introduced DeprecateStaleEntityTask for automated lifecycle management of stale assets.
Enhanced Workflow Engine:
Improved WorkflowHandler to validate workflow definitions before deployment.
Added new custom functions to the rule engine for checking entity update timestamps and calculating field completeness scores.
CI/CD and Build Improvements:
Updated the CI Dockerfile with a multi-stage build and refined dependency installation.
Modified POM files to include necessary dependencies for new features.
* Adding DataCompleteness Task Node, Flowable Debug logs
* Transaction handling for Custom Workflow - Initial Draft
* add new tasks to node definition interface
* Update generated TypeScript types
* Draft Implementation of Multi Reviewer Approval Task with Migration
* Update generated TypeScript types
* Transaction handling fix, id truncation fix by migration, feed repo fix for multi reviewer pattern, copilot comments
* Update generated TypeScript types
* Fixed Multi Reviewer approval to take consideration of the namespaced variables, Fixed RollBackEntity task to follow subprocess like other automated tasks, copilot nitpicks
* Remove conditionalSetEntityAttributes as it is not needed anymore
* Update generated TypeScript types
* Completely remove the setConditionalAttributes to fix compilation errors
* Removed the comments in the schemaChanges
* Created a new Task called CreateDetailedApprovalTaskImpl, Fixed RollBackEntityTask to roll back to either Approved or Rejected state, use namespaced variables, Updated the workflow handler to resolve the tasks and remove them from the user who has approved the task in the feed during multi reviewer approval, TransactionManager updated to keep Transactions in place, Improve the Validation Node input in the Mainworkflow java to handle proper graph traversal, Find Proper Termination Message for the two user events that are conflicting, Include Message in the request approval thread task for proper messages
* Update generated TypeScript types
* Rendering of messages in task thread
* Fix PeriodicBatchEntityTriggerTask to separate workflows based on entities Draft, Fix: SetEntityAttributes Impl to handle multiple fields, Fix: DataCompletenessTask Draft
* Fix DataCompletenessTask BoundaryEvent in Flowable
* Introduced Wf Deployment Strategy for transaction Handling, Improved user tasks for better termination event naming, fix periodic batch entity trigger to fetch the entity instead of all the entities in the trigger, Migrated GlossaryApprovalWorkflow.json with new nodes and edges, Fixed test cases and some edge cases
* Update generated TypeScript types
* Added performTasks for TagRepository and DataProductRepository, Removed test api, removed unnecessary change from workflow json, improved DataCompletenessImpl, Improved CreateDetailedApprovalTaskImpl to show what updated and not updated in the thread response
* Remove Entity Certification and Entity Status tasks and make use of generic set entity attribute task
* Update generated TypeScript types
* Fix the compilation issues!
* Remove setCertification and setEntityAttributes from createWorkflowDefinition
* Test cases for custom workflows related to glossaryTermApprovalWorkflow
* Test cases for custom workflows
* Changed info to debug logs
* Update generated TypeScript types
* DetailedUserApprovalTask changed to reviewChange task, Have validations for workflows where user approval tasks can only be available for entities that support reviewers
* Fix compilation issues and mvn spotless apply
* Update generated TypeScript types
* Remove Extra assignees from user tasks
* Update generated TypeScript types
* Replace Tags/GlossaryTerms during Mutual Exclusivity and Append when there is no Mutual Exclusivity
* Workflow Transaction Manager to handle authorization as part of its functionality, Added Validation for workflows using /validate endpoint
* Increase the Flowable's polling time to 60 seconds to poll timer and async jobs table
* Update generated TypeScript types
* Enum for UserTask Termination Messages, Removed WorkflowUtils to use WorkflowHandler's terminateDuplicate Instances, Approval and rejecters list in the variables, using namespaced variables for updatedBy
* Reverted the enum for userTaskType for now
* Added new tests for dataContract, dataProduct and Tag for user approval tasks
* Glossary Approval Workflow changed to handle jsonLogic according to UI expectations
* Added a new Task type for change description tasks to review and suggest the changes, Added checkUpdatedByReviewer incase of perfom tasks, increase wf test timeout
* Update generated TypeScript types
* TaskWorkflow constructor public to be used by Collate Repo
* AutoApproveServiceTaskImpl incase the assignees are not there for the userTask with ExlusiveGateway builder with hasAssignees
* Fix Compilation Issues - Upgrade of deps to lang3
* ExclusiveGatewayBuilder set exclusive as true, and other minor changes for test and approval tasks
* Added a different wait mechanism for user tasks in WorkflowDefinitionResourceTest.java
* Combined UserApprovalTask and ChangeReviewTask into one to handle suggestions in the same task!
* Update generated TypeScript types
* Test Case Approval Workflow, Removing orphaned changeReviewTask, Test case fix and new test cases
* Update generated TypeScript types
* Treat empty strings, arrays as missing by default
* Update generated TypeScript types
* fix compilation issues by changing the schema properly
* Remove Stale ChangeReview Task
* Update generated TypeScript types
* Enhanced User Approval task to show changed fields along with what changed and the task resolve function as well
* Update generated TypeScript types
* Add Knowledge center page to workflows event consumer, remove legacy backward compatibility for triggers, lenient validations for workflows when there are no nodes
* EntitySpecific Filters for EventBased Trigger, Removing Backward Compatibility logic for filters, Removed unnecessary comments and execution variables, Handle Structured task resolver for dataProduct, dataContract, tag and testCase, Modified GlossaryApprovalWorkflow.json and improved the MigrationUtil
* Update generated TypeScript types
* Bring back entitycertification and glosarystatus task for backward compatibility
* Update generated TypeScript types
* Filter is a map, entity specific filters are stringified, have certification and glossary status tasks in nodeinterface
* Merge Main
* Remove Suggestions for RequestApproval
* Update generated TypeScript types
* Remove Old Deployments of Periodic Batch Entity Trigger and use hiphen to trigger to avoid accidental triggering
* Resolve Merge conflicts, Java Checkstyle
* Update generated TypeScript types
* Fix Migrations
* Added alter table queries in 1.6.0 to avoid cached plan queries in flowable
* Increase timeout in workflow definition resource test
* Increase polling in workflow definition resource test
* Fix java Checkstyle
* comment the flaky test
* COmmented out Flaky Test Cases, fixed a bug on team reviewers
* Resolve java checkstyle after resolving conflicts
* add updated at field in json logic for glossary
* update fields
* add version field
* Update the rule config of new term node with "and" and update the migration as well
* add equal not equal op
* Delete the trigger workflows using like condition
* Defensive Fallback for multiple task instances created by race condition, Terminate duplicate instances of main workflows
* reverting operators
* Approval Capabilities for Metrics
* Update generated TypeScript types
* Move Migrations to 1.10.1 from 1.10.0
* Removed the extra spaces in Migration 1.10
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* Add the migration classes and data for recognizers
This is so that we can run a migration that sets `json->recognizers` of `PII.Sensitive` and `PII.NonSensitive` tags from json values.
The issue with normal migrations was that the value of recognizers was too long to be persisted in the server migrations log.
Created a common `migration.utils.v1110.MigrationProcessBase`
* Ensure building automatically with the right parameters
* Update typescript types
* Fix: Domain assets count mismatch between API and UI
* exclude SearchDerivedFields from reindex
* refactor fallback and have new field assetsCount
* Update generated TypeScript types
* refactor: clean up groupEntitiesByType function
* add tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Update the index mappings
* Add the Data Product field for filtering
Improve the field suggestions shown for filter values
* Fix unit tests
* update the application locale file
* Update suggestion APIs for tags, certification and tier filters
* Add tests for the new suggestion changes.
* Add the version and dataProduct fields for index mapping back
* fix: update autocomplete queries to use structured query format for classification filtering
* Fix the classification not showing in search api response
* Fix the failing playwright Advanced search
* Add lowercase_normalizer for the tier field similar to the tag field in the index mappings
* Fix the Curated Assets test
* Fix the failing playwright tests
* Fix the playwright tests
* update the API await logic
* Fix the Advanced search tests
* Fix the translation file eslint
* Work on the comments
* Add data product display name normalization for jp and ru langs
* fix: remove default provider from Superset connection schema
* feat: add Superset form handling and connection testing functionality
* Fix the unit test
* Fix Search Index Application spec failure
* MINOR - Add SLA fields
* Update generated TypeScript types
* added ui support for the timezone and columnName
* MINOR - Add SLA fields
* Update generated TypeScript types
* change the timezone as the enum and added field placeholder
* display column name on ui
* Fix unit test
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
