* fix: added code for separate engine and session for each project in rofiler and classification and refactor billing project approach
* fix: added entity.database check, bigquery sampling tests
* fix: system metrics logic when bigquery billing project is provided
* MINOR: Improve UDF Lineage Processing & Better Logging Time & MultiProcessing (#20848)
* Fix multiprocessing with better memory management and Airflow 2+ compatibility
* Add support for both multiprocessing and multithreading for relevant platforms
* Handle conflicting cross-db lineage changes of service_name parameter change
* Handle stored proc queries without caching all and increase the thread timeout times to cover 100% lineage
* Fix `get_table_query` inheritance and pylint
* Remove mocks from db_utils tests
* Better db_utils test and fix the service_names parameter in case of schema_fallback
---------
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Fix Oracle DataDiff and Change Oracle Connection to BaseConnection
* Add small unittest
* Fix Test
* Fix logic, to void other engines to denormalize table/schema names
* Add calculated view columns' formula parsing logic with correct source reference
* Handle top level column formula parsing and pass formula expression in column lineage detail
---------
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
* fix: ingestion fails for Iceberg tables with nested partition column
* test: added test to cover nested partition column for iceberg
* refactor: used if-else in tablePartition check
* fix: partition_column_name & column_partition_type typo
* Add lineage to Exasol connector
* Update test_connection to return TestConnectionResult
* Add exasol tests & dependencies to tests in setup.py
* Opensearch is required for testing, so add it there
* Modify metadata
* Update documentation for lineage
* Apply formatting changes to code
* Apply make py_format
* fix(dq): enable ''Column values to be in set'' test case for boolean columns
Add BOOLEAN to supportedDataTypes array in columnValuesToBeInSet.json
to allow boolean column validation with predefined allowed values.
This enables users to enforce strict true/false validation on boolean
columns directly at the column level, resolving issue #22099.
Co-authored-by: IceS2 <IceS2@users.noreply.github.com>
* Add tests to the new feature
* Add migrations and columnValuesToBeNotInSet
---------
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: IceS2 <IceS2@users.noreply.github.com>
* Process temp table graph in stored procedure processor and add db/schema filtering on lineage
* Add tests for stored procedure lineage processing
* Fix tests and py_format
* Fix the filters and log stored proc query count info
* Enhance patch request handling by adding 'skip_on_failure' parameter
* Introduced 'skip_on_failure' option in build_patch and OMetaPatchMixin methods to control behavior on patch operation failures.
* Updated documentation to reflect the new parameter and its default value.
* Improved error handling to log warnings instead of raising exceptions when 'skip_on_failure' is set to True.
* fix: add tests for patch request with skip on failure
* refactor: streamline mock patching and improve test readability in patch request tests
* Consolidated import statements for unittest mock.
* Enhanced readability by reducing line breaks and simplifying mock patching syntax.
* Ensured consistent use of commas in function calls for clarity.
* Updated tests to maintain functionality while improving code style.
* fix: improve error handling in patch operations
* Enhanced logging for patch operation failures in both build_patch and OMetaPatchMixin methods.
* Added detailed entity information in warning and error messages to aid in debugging.
* Ensured consistent behavior when 'skip_on_failure' is set, providing clearer feedback on operation outcomes.
* fix: clean up whitespace in patch request error handling
* Removed unnecessary whitespace in the build_patch function to improve code readability.
* Ensured consistent formatting in warning and error messages for better clarity during logging.
* fix: enhance error handling and improve test assertions in patch request
* Updated the condition for checking 'changeDescription' in the _remove_change_description function for better clarity.
* Modified exception handling in tests to raise RuntimeError instead of a generic Exception, providing more specific error feedback.
* Improved assertions in tests to check for the presence of error messages, enhancing the robustness of error handling verification.
* Adjusted test cases to reflect changes in expected patch operation counts and ensure accurate validation of patch operations.
* fix: enhance patch operation with skip_on_failure handling
* Added 'skip_on_failure' parameter to OMetaPatchMixin methods to control behavior on patch failures.
* Improved error handling to log warnings and provide detailed feedback when patch operations are skipped.
* Updated tests to verify the new behavior of skipping failures and improved assertions for clarity.
* feat: implemented microstrategy lineage & dbServicePrefix
* feat: added dbServicePrefixes support in other dashboards
* fix: test_metabase and powerbi extra code remove
* fix: python checkstyle
* refactor: added prefix support for other connectors - superset, tableau, etc
* refactor: added migration for prefix change and fix dbServicePrefixes field description
* refactor: added prefix changes in superset db source
* doc: add prefix in tableau doc
* fix: typescript files and postgres migration for prefix
* fix: moved migration in 1.8.2
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* 🎉 Init
* replace TARGET with EXPOSURE
* refactor and document
* add docs
* handle missing type/entity not matching
* linter
* update docs
* refactor for using label for communicating FQN as name field cannot contain special characters other than underscore. Storing dots in the name works for now but there is a deprecation warning and it will fail in the future.
* improve docs
* improve docs
* improve logging
* refactor for usage of meta.open_metadata_fqn
* linting
* update docs
* update docs
* fix docs
* 🎉 Add tests
* fix(airflow): correctly extract owners from serialized Airflow DAGs
Airflow serialization format wraps tasks under `__var` and `__type`.
Previously, the OpenMetadata Airflow connector failed to extract task owners properly in this format.
This patch:
- Flattens `__var` when parsing task owners
- Fallbacks to `default_args["owner"]` if no task-level owner is explicitly present
- Ensures correct DAG owner is picked as the most common task owner
- Handles compatibility with older Airflow versions
Fixes: #21106
* test(airflow): add tests for owner extraction from serialized Airflow DAGs
Adds new test cases to validate owner extraction logic:
- Owners from serialized task format (`__var`)
- Fallback to `default_args['owner']` if task owners are missing
- Resolution of most common owner
- Compatibility with unstructured or missing owners
* remove test version specific comment
* simplify comments and warnings
* fix return statement
* fixing formatting
* adding handling of default args
* fixing and adding more tests
* feat: add support for DBX system metrics
* feat: add support for DBX system metrics
* fix: added WRITE back
* fix: failing test cases
* fix: failing test
