* fix(dq): enable ''Column values to be in set'' test case for boolean columns
Add BOOLEAN to supportedDataTypes array in columnValuesToBeInSet.json
to allow boolean column validation with predefined allowed values.
This enables users to enforce strict true/false validation on boolean
columns directly at the column level, resolving issue #22099.
Co-authored-by: IceS2 <IceS2@users.noreply.github.com>
* Add tests to the new feature
* Add migrations and columnValuesToBeNotInSet
---------
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: IceS2 <IceS2@users.noreply.github.com>
* Process temp table graph in stored procedure processor and add db/schema filtering on lineage
* Add tests for stored procedure lineage processing
* Fix tests and py_format
* Fix the filters and log stored proc query count info
* Enhance patch request handling by adding 'skip_on_failure' parameter
* Introduced 'skip_on_failure' option in build_patch and OMetaPatchMixin methods to control behavior on patch operation failures.
* Updated documentation to reflect the new parameter and its default value.
* Improved error handling to log warnings instead of raising exceptions when 'skip_on_failure' is set to True.
* fix: add tests for patch request with skip on failure
* refactor: streamline mock patching and improve test readability in patch request tests
* Consolidated import statements for unittest mock.
* Enhanced readability by reducing line breaks and simplifying mock patching syntax.
* Ensured consistent use of commas in function calls for clarity.
* Updated tests to maintain functionality while improving code style.
* fix: improve error handling in patch operations
* Enhanced logging for patch operation failures in both build_patch and OMetaPatchMixin methods.
* Added detailed entity information in warning and error messages to aid in debugging.
* Ensured consistent behavior when 'skip_on_failure' is set, providing clearer feedback on operation outcomes.
* fix: clean up whitespace in patch request error handling
* Removed unnecessary whitespace in the build_patch function to improve code readability.
* Ensured consistent formatting in warning and error messages for better clarity during logging.
* fix: enhance error handling and improve test assertions in patch request
* Updated the condition for checking 'changeDescription' in the _remove_change_description function for better clarity.
* Modified exception handling in tests to raise RuntimeError instead of a generic Exception, providing more specific error feedback.
* Improved assertions in tests to check for the presence of error messages, enhancing the robustness of error handling verification.
* Adjusted test cases to reflect changes in expected patch operation counts and ensure accurate validation of patch operations.
* fix: enhance patch operation with skip_on_failure handling
* Added 'skip_on_failure' parameter to OMetaPatchMixin methods to control behavior on patch failures.
* Improved error handling to log warnings and provide detailed feedback when patch operations are skipped.
* Updated tests to verify the new behavior of skipping failures and improved assertions for clarity.
* feat: implemented microstrategy lineage & dbServicePrefix
* feat: added dbServicePrefixes support in other dashboards
* fix: test_metabase and powerbi extra code remove
* fix: python checkstyle
* refactor: added prefix support for other connectors - superset, tableau, etc
* refactor: added migration for prefix change and fix dbServicePrefixes field description
* refactor: added prefix changes in superset db source
* doc: add prefix in tableau doc
* fix: typescript files and postgres migration for prefix
* fix: moved migration in 1.8.2
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* 🎉 Init
* replace TARGET with EXPOSURE
* refactor and document
* add docs
* handle missing type/entity not matching
* linter
* update docs
* refactor for using label for communicating FQN as name field cannot contain special characters other than underscore. Storing dots in the name works for now but there is a deprecation warning and it will fail in the future.
* improve docs
* improve docs
* improve logging
* refactor for usage of meta.open_metadata_fqn
* linting
* update docs
* update docs
* fix docs
* 🎉 Add tests
* fix(airflow): correctly extract owners from serialized Airflow DAGs
Airflow serialization format wraps tasks under `__var` and `__type`.
Previously, the OpenMetadata Airflow connector failed to extract task owners properly in this format.
This patch:
- Flattens `__var` when parsing task owners
- Fallbacks to `default_args["owner"]` if no task-level owner is explicitly present
- Ensures correct DAG owner is picked as the most common task owner
- Handles compatibility with older Airflow versions
Fixes: #21106
* test(airflow): add tests for owner extraction from serialized Airflow DAGs
Adds new test cases to validate owner extraction logic:
- Owners from serialized task format (`__var`)
- Fallback to `default_args['owner']` if task owners are missing
- Resolution of most common owner
- Compatibility with unstructured or missing owners
* remove test version specific comment
* simplify comments and warnings
* fix return statement
* fixing formatting
* adding handling of default args
* fixing and adding more tests
* feat: add support for DBX system metrics
* feat: add support for DBX system metrics
* fix: added WRITE back
* fix: failing test cases
* fix: failing test
* Fix : query_cost_record_search_index Search exception for elasticsearch instance
* add sample query to cover test scenarios
* update mapping and fix test
* metadata dbt
* fix:
- default path to current directory
- addional warning and exception handling for missing metadata config vars
* test: add unit tests for DBT Ingestion CLI
* refactor
* PR review:
- using Pydantic to parse and validate the openmetadata config in dbt's .yml
- extended test-cases
- giving user more configuration options for ingestion
* py refactoring
* add: dbt-auto ingest docs
* Improvements:
- using environement variables for loading sensitve variables
- added docs for auto dbt-ingestion for dbt-core
- more test cases
* fix:
- test case for reading JWT token inside the the method
* refactor: py code formatting
* refactor: py formatting
* ingest-dbt docs updated
* refined test cases
* Chore:
- sonar vulnerability issue review
- using existing URL class for host validation
---------
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Implement Dependency Injection for the Ingestion Framework
* Fix Test
* Fix Profile Test Connection
* Add Injection to Metrics in Profiler
* Add Injection to the Profiler
* Fix UnitTests
* Fix Pytests
* Fix Tests
* Fix types
* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Implement Dependency Injection for the Ingestion Framework
* Fix Test
* Fix Profile Test Connection
* Fix test, making the injection test run last
* Update connections.py
* Changed NewType to an AbstractClass to avoid linting issues
* remove comment
* Fix bug in service spec
* Update PyTest version to avoid importlib.reader wrong import
* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Fix Test
* Fix Profile Test Connection
* Remove unit test
* Remove comment
* Fix tests and missing changes
* Add PIICategoryTags and some utilities on top of them.
* Fix static-check
* Add test for fqn representation
* Add NEREntityGeneralTags.json from Collate
* Add test to check PIICategoryTags agree with the ones used by OM server
* Add LabelExtractor
* Fix style
* Add ignore superflous-parens for pylint
* Ass comment as per PR review
* Fix not-updated PII-IT
* Remove duplicated IT test for PII
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Add logic to handle WorkflowContext on Ingestion
* Revert base.py changes
* Removed comment
* Fix basedpyright complaints
* Make ContextManager automatically add its context to the PipelineStatus
* Small changes
