This PR fixes a bug in the Kinesis messaging source where the `Topic` class was incorrectly imported from `metadata.generated.schema.type.schema` instead of the correct entity
definition path `metadata.generated.schema.entity.data.topic`.
**Problem:**
The `yield_topic_sample_data` method used the `type.schema.Topic` definition when calling `fqn.build` and `metadata.get_by_name`. These functions expect the main entity
class.
**Fix:**
Changed the import statement to use `metadata.generated.schema.entity.data.topic.Topic`.
This ensures the correct type definition is used when interacting with the FQN utility and metadata API, preventing potential downstream issues.
* fix: made databricks httpPath required and added a migration file for the same
* fix: added sql migration in postDataMigration file and fix databricks tests
* fix: added httpPath in test_source_connection.py and test_source_parsing.py files
* fix: added httpPath in test_databricks_lineage.py
* fix: table name in postgres migration
For tableau ingestion, code does not handle properly null upstream custom table queries
and null values for table OM entities.
Co-authored-by: Abdallah Serghine <abdallah.serghine@olx.pl>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Unpinned google-cloud-secret-manager version in ingestion dependencies
* Restrict google-cloud-secret-manager version to <2.20.1 because of mlflow-skinny dependency issue
---------
Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
* fix: close client after query
use context clients in SQL sampler to close the connection once the query is complete
* use self.context_client in all sql sampler implementations
* use sqlalchemy's built-in session management
* format
* format
* use get_client directly
- Refactored the handling of nested columns in `sql_column_handler.py` to prioritize source-provided children, ensuring they override any derived children.
- Removed the overridden `_process_col_type` method in `bigquery/metadata.py` to streamline column type handling, enforcing the use of the standard path for BigQuery.
This update improves the accuracy of column metadata processing and simplifies the codebase.
* Refactor SQL column processing and enhance BigQuery project ID handling
* Introduced a new `process_column` function in `sql_column_handler.py` to streamline column processing logic.
* Updated `BigquerySource` to improve project ID retrieval from service connections, ensuring compatibility with various credential types.
* Added handling for nested columns in BigQuery schema processing.
* Enhanced error handling and logging for better debugging during project ID setup.
* Add support for GCP Application Default Credentials in BigQuery ingestion
* Enhanced `BigquerySource` to include handling for GCP Application Default Credentials (ADC).
* Updated JSON schema for GCP credentials to define `gcpADC` and its properties.
* Improved logging for credential setup in `set_google_credentials` function.
* Added comments and TODOs for future enhancements related to project ID fetching from the resource manager.
* Update .gitignore to include cursor rules files
* Added .cursorrules and .cursor/ to the .gitignore to prevent tracking of cursor rule files in the repository.
* This change helps maintain a cleaner repository by excluding unnecessary files from version control.
* refactor: Bigquery Credentials to allow multiple project ids
* fix: Handle unknown array data types in SQL column processing
