* Kafka Connect Lineage Improvements
* Remove specific Kafka topic example from docstring
Removed example from the documentation regarding the earnin.bank.dev topic.
* fix: update comment to reflect accurate example for database server name handling
* fix: improve expected FQN display in warning messages for missing Kafka topics
* fix: update table entity retrieval method in KafkaconnectSource
* fix: enhance lineage information checks and improve logging for missing configurations in KafkaconnectSource
* Kafka Connect Lineage Improvements
* address comments; work without the table.include.list
---------
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* chore: implement logger levels tests for depreciation
* fix: use METADATA_LOGGER instead of warnings
* use unit test syntax
* isort
* black
* fix test
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Update `TableDiffParamsSetter` to move data at table level
This means that `key_columns` and `extra_columns` will be defined per table instead of "globally", just like `data_diff` expects
* Update `TableDiffValidator` to use table's `key_columns`
Call `data_diff` and run validations using each table's `key_columns`
* Create migration to update `tableDiff` test definition
* Fix Playwright test
* Refactor presidio utils
Extract the spacy model functionality from the analyzer building function
* Added a new `TagClassifier`
This classifier uses tags to dynamically build presidio `RecognizerRegistry`s
* Added a new `TagProcessor`
This processor uses `TagClassifier` to label a column based on the tags' recognizers
* Create `TagProcessor` based on workflow configuration
* Create decorator to apply threshold to recognizers
This is so that we can apply thresholds on recognizer results without subclassing or having to keep a map between the presidio recognizer and the recognizer configuration
* Fix broken test
* Add `reason` property to `TagLabel`
This is to understand what score was used for selecting the entity
* Build `TagLabel`s with `reason`
* Increase `PIIProcessor._tolerance`
This is so we correctly filter out low scores from classifiers while still maintaining the normalization that filters out confusing outcomes.
e.g: an output with scores 0.3, 0.7 and 0.75, would initially filter the 0.3 and then discard the other two because they're both relatively high results.
* Make database and DAO changes needed to persist `TagLabel.reason`
* Update generated TypeScript types
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add support for translations in multi lang
* Add Tag Feedback System
* Update generated TypeScript types
* Fix typing issues and add tests to reocgnizer factory
* Updated `TagResourceTest.assertFieldChange` to fix broken test
This is because change description values had been serialized into strings and for some reason the keys ended up in a different order. So instead of performing String comparison, we do Json comparisons
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eugenio Doñaque <eugenio.donaque@getcollate.io>
* Update test data for `tests.integration.trino`
This is to create tables with complex data types.
Using raw SQL because creating tables with pandas didn't get the right types for the structs
* Update tests to reproduce the issue
Also included the new tables in the other tests to make sure complex data types do not break anything else
Reference: [issue 16983](https://github.com/open-metadata/OpenMetadata/issues/16983)
* Added `TypeDecorator`s handle `trino.types.NamedRowTuple`
This is because pydantic couldn't figure out how to create python objects when receiving `NamedRowTuple`s, which broke the sampling process.
This makes sure the data we receive from the trino interface is compatible with Pydantic
* feat: databricks oauth and azure ad auth setup
* refactor: add auth type changes in databricks.md
* fix: test after oauth changes
* refactor: unity catalog connection to databricks connection code
* Feat: show dbt project name
* Update generated TypeScript types
* added dbtSourceProject in data asset header properties
* Added tests
* Addressed comments
* Update generated TypeScript types
* move from dataAssetHeader to the dbt tab itself
* added unit test for added code
* test name change
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* fix: added code for separate engine and session for each project in rofiler and classification and refactor billing project approach
* fix: added entity.database check, bigquery sampling tests
* fix: system metrics logic when bigquery billing project is provided
* Initial Implementation of Adding Status and Reviewers to assets for workflows
* Update generated TypeScript types
* Copilot Review Comments Addressed
* Removed DataProduct Reviewer Inheritance as it is irrelevant
* Commit: Classification has status and reviewers, DataContract uses the same status enums, changed the logic to be APPROVED instead of Active, DataContract can have null status as seen in tests, Changed Workflow to use workflowStatus instead of status as it is contradicting with the approval status, Fixed Tests
* Default for reviewers is null
* Default for reviewers is createSchema
* Addressed CoPilots comments
* Update generated TypeScript types
* Workflow status to workflowStatus in db and migrations
* Revert "Workflow status to workflowStatus in db and migrations"
This reverts commit 676e8789358654bc6f980f855c372f33c22fc40b.
* Changed status to entityStatus in the schema files
* Java Implementation of Default Status, Search Client improvements and Test fixes and new tests
* Adding entityStatus and reviewers in the searchIndex mappings and common attributes
* Data Migration scripts to change the glossaryTerm and dataContract structure
* Update generated TypeScript types
* Fixed zh/spreadsheet index json error
* Fix Postgre migration script
* Changed the entityStatus.json to status.json
Removed the duplicates of entityStatus in the indexMapping
Modified the sample data to take in EntityStatus.Approved instead of ContractStatus.Active
* Update generated TypeScript types
* dummy commit
* Fix UI Build Issues with the New EntityStatus
Fix py tests
* Migrations for all the entities that need entityStatus
* Update generated TypeScript types
* Removed Post Migration scripts
* Fix UI and py for entityStatus
* Update generated TypeScript types
* Fix: DataContractResourceTest
* Fix UI and py for importing entityStatus
* UI to show and fetch Reviewers
* cleanup
* Removed Overridden SetDefaultStatus in GlossaryTermRepository
* Removed unnecessary validation
* Added entityStatus in search_entity_index_mapping.json
* Fixed DataContractResourceTest
* mvn spotless apply and fix migration scripts
* fix tests
* fix type error
* fix advanced search tests
* Status comparison using enums and supportsStatus to supportsEntityStatus
* mvn spotless apply
* fix merge conflict
* update entity status
* fix tests
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
* MINOR: Improve UDF Lineage Processing & Better Logging Time & MultiProcessing (#20848)
* Fix multiprocessing with better memory management and Airflow 2+ compatibility
* Add support for both multiprocessing and multithreading for relevant platforms
* Handle conflicting cross-db lineage changes of service_name parameter change
* Handle stored proc queries without caching all and increase the thread timeout times to cover 100% lineage
* Fix `get_table_query` inheritance and pylint
* Remove mocks from db_utils tests
* Better db_utils test and fix the service_names parameter in case of schema_fallback
---------
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
