This PR fixes a bug in the Kinesis messaging source where the `Topic` class was incorrectly imported from `metadata.generated.schema.type.schema` instead of the correct entity
definition path `metadata.generated.schema.entity.data.topic`.
**Problem:**
The `yield_topic_sample_data` method used the `type.schema.Topic` definition when calling `fqn.build` and `metadata.get_by_name`. These functions expect the main entity
class.
**Fix:**
Changed the import statement to use `metadata.generated.schema.entity.data.topic.Topic`.
This ensures the correct type definition is used when interacting with the FQN utility and metadata API, preventing potential downstream issues.
For tableau ingestion, code does not handle properly null upstream custom table queries
and null values for table OM entities.
Co-authored-by: Abdallah Serghine <abdallah.serghine@olx.pl>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* fix: close client after query
use context clients in SQL sampler to close the connection once the query is complete
* use self.context_client in all sql sampler implementations
* use sqlalchemy's built-in session management
* format
* format
* use get_client directly
- Refactored the handling of nested columns in `sql_column_handler.py` to prioritize source-provided children, ensuring they override any derived children.
- Removed the overridden `_process_col_type` method in `bigquery/metadata.py` to streamline column type handling, enforcing the use of the standard path for BigQuery.
This update improves the accuracy of column metadata processing and simplifies the codebase.
* Refactor SQL column processing and enhance BigQuery project ID handling
* Introduced a new `process_column` function in `sql_column_handler.py` to streamline column processing logic.
* Updated `BigquerySource` to improve project ID retrieval from service connections, ensuring compatibility with various credential types.
* Added handling for nested columns in BigQuery schema processing.
* Enhanced error handling and logging for better debugging during project ID setup.
* Add support for GCP Application Default Credentials in BigQuery ingestion
* Enhanced `BigquerySource` to include handling for GCP Application Default Credentials (ADC).
* Updated JSON schema for GCP credentials to define `gcpADC` and its properties.
* Improved logging for credential setup in `set_google_credentials` function.
* Added comments and TODOs for future enhancements related to project ID fetching from the resource manager.
* Update .gitignore to include cursor rules files
* Added .cursorrules and .cursor/ to the .gitignore to prevent tracking of cursor rule files in the repository.
* This change helps maintain a cleaner repository by excluding unnecessary files from version control.
* refactor: Bigquery Credentials to allow multiple project ids
* fix: Handle unknown array data types in SQL column processing
* feat: allow user to turn off rnadomized sample
* style: ran python linting
* fix: models default value for randomizedSample
* style: ran linting
* doc: move config to advanced
* Fix#19667: OpenSearch Connector
* Fix#19667: OpenSearch Connector
* do not ingest any system level indexes
* fix pyformat
* Add AWS auth
* Use common schema and fix ssl config in client
* Add openseach connector docs and update schema
* Remove api key auth type and complete docs checklist
* Remove unnecessary httpx dependency and pyformat
* Add compatible version of httpx for elasticsearch
* Fix pylint fails and py-tests validation error
---------
Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
