* Process temp table graph in stored procedure processor and add db/schema filtering on lineage
* Add tests for stored procedure lineage processing
* Fix tests and py_format
* Fix the filters and log stored proc query count info
* Enhance patch request handling by adding 'skip_on_failure' parameter
* Introduced 'skip_on_failure' option in build_patch and OMetaPatchMixin methods to control behavior on patch operation failures.
* Updated documentation to reflect the new parameter and its default value.
* Improved error handling to log warnings instead of raising exceptions when 'skip_on_failure' is set to True.
* fix: add tests for patch request with skip on failure
* refactor: streamline mock patching and improve test readability in patch request tests
* Consolidated import statements for unittest mock.
* Enhanced readability by reducing line breaks and simplifying mock patching syntax.
* Ensured consistent use of commas in function calls for clarity.
* Updated tests to maintain functionality while improving code style.
* fix: improve error handling in patch operations
* Enhanced logging for patch operation failures in both build_patch and OMetaPatchMixin methods.
* Added detailed entity information in warning and error messages to aid in debugging.
* Ensured consistent behavior when 'skip_on_failure' is set, providing clearer feedback on operation outcomes.
* fix: clean up whitespace in patch request error handling
* Removed unnecessary whitespace in the build_patch function to improve code readability.
* Ensured consistent formatting in warning and error messages for better clarity during logging.
* fix: enhance error handling and improve test assertions in patch request
* Updated the condition for checking 'changeDescription' in the _remove_change_description function for better clarity.
* Modified exception handling in tests to raise RuntimeError instead of a generic Exception, providing more specific error feedback.
* Improved assertions in tests to check for the presence of error messages, enhancing the robustness of error handling verification.
* Adjusted test cases to reflect changes in expected patch operation counts and ensure accurate validation of patch operations.
* fix: enhance patch operation with skip_on_failure handling
* Added 'skip_on_failure' parameter to OMetaPatchMixin methods to control behavior on patch failures.
* Improved error handling to log warnings and provide detailed feedback when patch operations are skipped.
* Updated tests to verify the new behavior of skipping failures and improved assertions for clarity.
* feat: implemented microstrategy lineage & dbServicePrefix
* feat: added dbServicePrefixes support in other dashboards
* fix: test_metabase and powerbi extra code remove
* fix: python checkstyle
* refactor: added prefix support for other connectors - superset, tableau, etc
* refactor: added migration for prefix change and fix dbServicePrefixes field description
* refactor: added prefix changes in superset db source
* doc: add prefix in tableau doc
* fix: typescript files and postgres migration for prefix
* fix: moved migration in 1.8.2
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* 🎉 Init
* replace TARGET with EXPOSURE
* refactor and document
* add docs
* handle missing type/entity not matching
* linter
* update docs
* refactor for using label for communicating FQN as name field cannot contain special characters other than underscore. Storing dots in the name works for now but there is a deprecation warning and it will fail in the future.
* improve docs
* improve docs
* improve logging
* refactor for usage of meta.open_metadata_fqn
* linting
* update docs
* update docs
* fix docs
* 🎉 Add tests
* fix(airflow): correctly extract owners from serialized Airflow DAGs
Airflow serialization format wraps tasks under `__var` and `__type`.
Previously, the OpenMetadata Airflow connector failed to extract task owners properly in this format.
This patch:
- Flattens `__var` when parsing task owners
- Fallbacks to `default_args["owner"]` if no task-level owner is explicitly present
- Ensures correct DAG owner is picked as the most common task owner
- Handles compatibility with older Airflow versions
Fixes: #21106
* test(airflow): add tests for owner extraction from serialized Airflow DAGs
Adds new test cases to validate owner extraction logic:
- Owners from serialized task format (`__var`)
- Fallback to `default_args['owner']` if task owners are missing
- Resolution of most common owner
- Compatibility with unstructured or missing owners
* remove test version specific comment
* simplify comments and warnings
* fix return statement
* fixing formatting
* adding handling of default args
* fixing and adding more tests
* feat: add support for DBX system metrics
* feat: add support for DBX system metrics
* fix: added WRITE back
* fix: failing test cases
* fix: failing test
* Fix : query_cost_record_search_index Search exception for elasticsearch instance
* add sample query to cover test scenarios
* update mapping and fix test
* metadata dbt
* fix:
- default path to current directory
- addional warning and exception handling for missing metadata config vars
* test: add unit tests for DBT Ingestion CLI
* refactor
* PR review:
- using Pydantic to parse and validate the openmetadata config in dbt's .yml
- extended test-cases
- giving user more configuration options for ingestion
* py refactoring
* add: dbt-auto ingest docs
* Improvements:
- using environement variables for loading sensitve variables
- added docs for auto dbt-ingestion for dbt-core
- more test cases
* fix:
- test case for reading JWT token inside the the method
* refactor: py code formatting
* refactor: py formatting
* ingest-dbt docs updated
* refined test cases
* Chore:
- sonar vulnerability issue review
- using existing URL class for host validation
---------
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Implement Dependency Injection for the Ingestion Framework
* Fix Test
* Fix Profile Test Connection
* Add Injection to Metrics in Profiler
* Add Injection to the Profiler
* Fix UnitTests
* Fix Pytests
* Fix Tests
* Fix types
* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Implement Dependency Injection for the Ingestion Framework
* Fix Test
* Fix Profile Test Connection
* Fix test, making the injection test run last
* Update connections.py
* Changed NewType to an AbstractClass to avoid linting issues
* remove comment
* Fix bug in service spec
* Update PyTest version to avoid importlib.reader wrong import
* Initial implementation for our Connection Class
* Implement the Initial Connection class
* Add Unit Tests
* Fix Test
* Fix Profile Test Connection
* Remove unit test
* Remove comment
* Fix tests and missing changes
* Add PIICategoryTags and some utilities on top of them.
* Fix static-check
* Add test for fqn representation
* Add NEREntityGeneralTags.json from Collate
* Add test to check PIICategoryTags agree with the ones used by OM server
* Add LabelExtractor
* Fix style
* Add ignore superflous-parens for pylint
* Ass comment as per PR review
* Fix not-updated PII-IT
* Remove duplicated IT test for PII
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Add logic to handle WorkflowContext on Ingestion
* Revert base.py changes
* Removed comment
* Fix basedpyright complaints
* Make ContextManager automatically add its context to the PipelineStatus
* Small changes
* Reproduce failing behaviour with non-date-time data
* Add a presidio patch for DateTimes
* Fix type-check error
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
* Make pytest to user code from src rather than from install package
* Fix test_amundsen: missing None
* Update pytest configuration to use importlib mode
* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings
* Refactor referencedByQueries validation to use field_validator as per deprecation warning
* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning
* Move superset test to integration test as they are using testcontainers
* Update coverage source path
* Fix wrong import.
* Add install_dev_env target to Makefile for development dependencies
* Add test-unit as extra in setup.py
* Modify dependencies in dev environment.
* Ignore all airflow tests
* Remove coverage in unit_ingestion_dev_env. Revert coverage source to prevent broken CI.
* Add nox for running unit test
* FIx PowerBI integration test to use pathlib for resource paths and not os.getcwd to prevent failures when not executed from the right path
* Move test_helpers.py to unit test, as it is not an integration test.
* Remove utils empty folder in integration tests
* Refactor testcontainers configuration to avoid pitfalls with max_tries setting
* Add nox unit testing basic setup
* Add format check session
* Refactor nox-unit and add plugins tests
* Add GHA for py-nox-ci
* Add comment to GHA
* Restore conftest.py file
* Clarify comment
* Simplify function
* Fix matrix startegy and nox mismatch
* Improve python version strategy with nox and GHA
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
* Fix test_amundsen: missing None
* Fix custom_basemodel_validation to check model_fields on type(values) to prevent noisy warnings
* Refactor referencedByQueries validation to use field_validator as per deprecation warning
* Update ColumnJson to use model_rebuild rather as replacement for forward reference updates as per deprecation warning
* Move superset test to integration test as they are using testcontainers
* Add install_dev_env target to Makefile for development dependencies
* Add test-unit as extra in setup.py
* Skip failing IT test. Requires further investigation.
* feat: add query logger as an event listent in debug mode
* fix: added ingestion.src plugin to pylint
* minor: add partition sampled table
* test: added test for partitioned BQ table
* Remove log_query function from logger.py
* style: ran python linting
* Remove 'ORGANIZATION' PII Tag as it is no longer supported by our PII detectors.
* Updata presidio version to fix wrong regex for indian passport
* Increase sample size of Indian passport numbers
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
* Add PII Tag and Sensitivity Level enums.
* Add feature-extraction for PII classification tasks
* Add faker as test dependency
* Add unit tests for presidio tag extractor
* Add PIISensitivityTags enum and update sensitivity mapping logic
* Add Presidio utility functions for PII analysis
* Extend column name regexs for PII
* Add tests for PAN, NIF, SSN entities
* Fix version of faker to prevent flaky tests. Fix failing tests.
* Add Generated to State enum
* Integrate PIISensitive classifier to PIIProcessor
* Add PII Tag and Sensitivity Level enums.
* Add feature-extraction for PII classification tasks
* Add faker as test dependency
* Add unit tests for presidio tag extractor
* Add PIISensitivityTags enum and update sensitivity mapping logic
* Add Presidio utility functions for PII analysis
* Extend column name regexs for PII
* Add colum name split
* Move pii algorithms to dedicated package
* Add tests for PAN, NIF, SSN entities
* Fix linting
* Add comment on why we need to set specific lanaguage to Presidio recognizers, as per PR suggestion.
* Fix version of faker to prevent flaky tests. Fix failing tests.
* Fix wrong import
---------
Co-authored-by: Pere Menal <pere.menal@getcollate.io>
