* fixed fqn parsing problem in clickhouse and added more logging
* ran py format commands
* fixed python formatting issues
---------
Co-authored-by: Nancy Amandi <nancy.amandi@moniepoint.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
* Initial implementation for Dimensionality on Data Quality Tests
* Fix ColumnValuesToBeUnique and create TestCaseResult API
* Refactor dimension result
* Initial E2E Implementation without Impact Score
* Dimensionality Thin Slice
* Update generated TypeScript types
* Update generated TypeScript types
* Removed useless method to use the one we already had
* Fix Pandas Dimensionality checks
* Remove useless comments
* Implement PR comments, fix Tests
* Improve the code a bit
* Fix imports
* Implement Dimensionality for ColumnMeanToBeBetween
* Removed useless comments and improved minor things
* Implement UnitTests
* Fixes
* Moved import pandas to type checking
* Fix Min/Max being optional
* Fix Unittests
* small fixes
* Fix Unittests
* Fix Issue with counting total rows on mean
* Improve code
* Fix Merge
* Removed unused type
* Refactor to reduce code repetition and complexity
* Fix conflict
* Rename method
* Refactor some metrics
* Implement Dimensionality to ColumnLengthToBeBetween
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Update schemas
* Remove the allowedEmailRegistrationDomains, allowedDomains, useRolesFromProvider fields from hidden state
* Refactor the SSO Configuration Form and add tests
* Fix code smells and refactor the code for SSOConfigurationForm
* Fix the code smells
* Remove the custom functions to create patch for SSO configurations
* Add mock for structuredClone
* Update generated TypeScript types
* Empty commit
* Fix the unnecessary cleanup of data before saving
* Update the default values for oidc configs
* Fix unit test
* Remove the unnecessary util function
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Refactor previous tests for shared resources
* Add validation result models
This also includes a method for merging them, useful when running validation in batches
* Added `DataFrameValidationEngine` for running tests
This also includes a registry for mapping test names to pandas test classes
* Implement the DataFrameValidator facade
This includes the logic to load tests from different sources (OpenMetadata or code) and pass them down to the engine.
It also includes tests for the integration with OpenMetadata
* Add examples for the API
* Apply comments
* Implement Ingestion side to return a flag when all values are unique
* Update generated TypeScript types
* feat: Enhance CardinalityDistributionChart to display messages when all values are unique
- Added logic to check if all values are unique for both first day and current day data.
- Implemented a placeholder message when all values are unique, indicating no distribution available.
- Updated tests to cover scenarios for unique values and ensure correct rendering of charts and messages.
- Added localization for the new message in multiple languages.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* refactor: used hashing to reduce api calls, replace distinct with group by to optimize lineage queries & minor code optimizations
* Update generated TypeScript types
* fix: self.job_table_lineage defaultdict function
* refactor: improved hashing
* fix: added _table_lookup_cache and _dlt_table_cache in tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Initial implementation for Dimensionality on Data Quality Tests
* Fix ColumnValuesToBeUnique and create TestCaseResult API
* Refactor dimension result
* Initial E2E Implementation without Impact Score
* Dimensionality Thin Slice
* Update generated TypeScript types
* Update generated TypeScript types
* Removed useless method to use the one we already had
* Fix Pandas Dimensionality checks
* Remove useless comments
* Implement PR comments, fix Tests
* Improve the code a bit
* Fix imports
* Implement Dimensionality for ColumnMeanToBeBetween
* Removed useless comments and improved minor things
* Implement UnitTests
* Fixes
* Moved import pandas to type checking
* Fix Min/Max being optional
* Fix Unittests
* small fixes
* Fix Unittests
* Fix Issue with counting total rows on mean
* Improve code
* Fix Merge
* Removed unused type
* Refactor to reduce code repetition and complexity
* Fix conflict
* Rename method
* Refactor some metrics
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix: address data quality feedbacks
* test: enhance DataObservabilityTab tests with async rendering and data quality report mock
* fix: update search query to use wildcard characters for better matching
* fix: adjust column width and add vertical scroll to ColumnProfileTable
* Refactor SVG files to remove width and height attributes
- Updated aborted-test.svg to remove width and height attributes.
- Updated column-count.svg to remove width and height attributes.
- Updated created-date.svg to remove width and height attributes.
- Updated failed-test.svg to remove width and height attributes.
- Updated profile-sample.svg to remove width and height attributes.
- Updated row-count.svg to remove width and height attributes.
- Updated success-test.svg to remove width and height attributes.
- Updated total-size.svg to remove width and height attributes.
- Updated total-test.svg to remove width and height attributes.
* Remove CSS comments from email notification envelope
* Add markdown & HTML processing dependencies
* Notification channel foundation & channel renderers
* Implement core logic for the notification templates engine
* Test core logic for the notification templates engine
* Add missing icons for persona data assets settings
* Fix ui errors
* Update the default values for oidc configs
* Revert "Update the default values for oidc configs"
This reverts commit 73cab6d00b15ecfc96a43b228d3b5e5b0379c79a.
* Add missing icon for classification in persona details page
* Fix sonar issue
* Setup new client for ES/OS
* Migrated createIndex and addIndexAlias methods
* Migrated createAliases method to new ES/OS client
* Migrated updateIndex and deleteIndex methods
* Updated indexExists methods
* refactor: extract index management operations into dedicated manager classes
* Add tests for ElasticSearch and OpenSearch index managers
* chore: fix code style issues"
* Added integeration tests for ES/OS index manager
* Fix log level
* Fixed priority key mapping for test_case_result_index
* fix test to use field name in the query as domains.id not domain.id
* Migrated createEntity and createEntities method
* Fixed failing tests
* Set headers so 8.x client can work with 7.17x or higher server for ES
* Added OpenSearch compatible stemmer configuration
* Fix java code style
* Fixed java checkstyle issue
* Added support for 7.17.x backward compatibility
* Fixed failing tests
* Migrated createTimeSeriesEntity and deleteEntity methods
* Refactor search client architecture with entity management abstraction
* Updated log level to ERROR form WARN
* deleteEntity method impl removed from ES client
* migrated deleteEntityByFields method to new es/os client
* migrated deleteEntityByFQNPrefix method to use new ES/OS client
* deleteEntityByFQNPrefix method removed
* Migrated deleteByScript method to new ES/OS client
* Removed deleteByScript method from SearchClient
* Migrated softDeleteOrRestoreEntity method to new ES/OS client
* Reverted BulkResponse static import
* Migrated softDeleteOrRestoreChildren method to new ES/OS client
* Migrated updateEntity method to new ES/OS client
* Migrated updateChildren method to new ES/OS client
* Removed unused methods
* Migrated getDocByID method to new ES/OS client
* Added info logging
* Fixed parsing issue while passing string doc
* Migrated updateEntityRelationship method to new ES/OS client
* Migrated reindexWithEntityIds method to new ES/OS client
* Added log to show error message for updateEntityRelationship
* Refactored entity manager methods
* Added async client, fixed null obj parsing issue
* Refactoring
* Refactored duplicate methods createEntity and createTimeSeriesEntity
* Added integration tests for ES/OS entity managers
* Updated client availability checks in both index manager classes to use the consistent !isClientAvailable pattern
* Removed comment
* chore: Fix line formatting in Playwright test files
- Adjusted line length in Domains.spec.ts for better readability
- Fixed indentation and formatting in Lineage.spec.ts
- Added missing newline at end of files
* Added new line
* Migrated other index related methods and add tests
* MIgrated updateByFqnPrefix and updateLineage and deleteByRangeQuery methods
* migrated deleteByRangeAndTerm method
* Added integration tests
* Migrated updateColumnsInUpstreamLineage and deleteColumnsInUpstreamLineage methods
* Migrated updateGlossaryTermByFqnPrefix method
* Removed unused method and updated error handling
* Migrated reindexEntities method to new ES/OS client
* Added integration tests for reindexEntities method
* Added interface and impl for generic methods migration
* added default impl
* Migrated deleteDataStream method
* Migrated deleteILMPolicy method
* Migrated deleteIndexTemplate method
* Migrated deleteComponentTemplate emthod
* migrated dettachIlmPolicyFromIndexes method
* migrated removeILMFromComponentTemplate method
* Migrated cluster metric methods
* Removed from ESClient/OSClient to GenericManager
* Migrated getSearchHealthStatus
* Add integration tests
* Migrated buildDIChart and fetchDIChartFields methods
* Migrated listDataInsightChartResult emthod
* Migrated getQueryCostRecords
* Migrated getSchemaEntityRelationship
* Added data insight aggregator tests
* Fixed failing tests
* Migrated aggregate method
* Migrated genericAggregation and aggregate methods
* Fixed failing data insight query
* Updated ref for OS aggregation manager
* Fixed issues with aggregations migration
* Fixed tests
* fix aggregate api response
* Fixed aggregation response structure
* Initial implementation for Dimensionality on Data Quality Tests
* Fix ColumnValuesToBeUnique and create TestCaseResult API
* Refactor dimension result
* Initial E2E Implementation without Impact Score
* Dimensionality Thin Slice
* Update generated TypeScript types
* Update generated TypeScript types
* Removed useless method to use the one we already had
* Fix Pandas Dimensionality checks
* Remove useless comments
* Implement PR comments, fix Tests
* Improve the code a bit
* Fix imports
* Implement Dimensionality for ColumnMeanToBeBetween
* Removed useless comments and improved minor things
* Implement UnitTests
* Fixes
* Moved import pandas to type checking
* Fix Min/Max being optional
* Fix Unittests
* small fixes
* Fix Unittests
* Fix Issue with counting total rows on mean
* Improve code
* Fix Merge
* Removed unused type
* Fix Tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix: migration
* fix: playwright test DBT -> dbt
* feat: added rentention for profile and dq data
* feat: fix failing tests
* feat: address error in postgres delete sql
* feat: fixed missing parameter in psql query
* fix: added the deletion step in test case
* feat: fixed postgres query for deletion before cutoffs
