mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-12-31 09:25:40 +00:00
c89bc2adac
* Oh boy, factory-boy Created a bunch of `factory-boy` factories that help creating mock test data easily * Update `try_bind` docker utility to ease debugging * Resolve conflicts between `Classification` tags * Refactor `TagClassifier` into another entity This is so: 1. We're not tied to the `ColumnClassifier` interface that forced returning `Mapping[T, float]` (unnecessary since we're returning `List[ScoredTag]` 2. The tag analyzer uses the same `recognizer_factories` registry we used for `PIIProcessor` 3. Create a separate service that abstracts using `TagScorer` and `TagAnalyzer` to return `TagScore`s (makes testing upstream code easier) * Interface to retrieve available `Tag`s and `Classification`s * Refactor `TagProcessor` to support multi-classification - Depends `ClassificationManagerInterface` to retrieve `Tag`s and `Classification`s - Uses a callable dependency to score tags for a column - Accepts a classification filter parameter - Leverages `ConflictResolver` to resolve conflicts between tags of the same `Classification` * Add an integration test for the `TagProcessor` * Ensure `PII` classification is configured with migrations # Conflicts: # bootstrap/sql/migrations/native/1.11.1/mysql/postDataMigrationSQLScript.sql # bootstrap/sql/migrations/native/1.11.1/postgres/postDataMigrationSQLScript.sql * Move `FakeClassificationManager` to `_openmetadata_testutils` This is because importing from `tests` breaks in the CI when running pytests from the root of the repo * Fix broken mutually exclusive classifications This is because the implementation did not take into account previous tags when resolving conflicts. This caused that running the classifier twice for a classification, with a mutually exclusive configuration, would end up breaking the exclusivity