For tableau ingestion, code does not handle properly null upstream custom table queries
and null values for table OM entities.
Co-authored-by: Abdallah Serghine <abdallah.serghine@olx.pl>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Unpinned google-cloud-secret-manager version in ingestion dependencies
* Restrict google-cloud-secret-manager version to <2.20.1 because of mlflow-skinny dependency issue
---------
Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
* fix: close client after query
use context clients in SQL sampler to close the connection once the query is complete
* use self.context_client in all sql sampler implementations
* use sqlalchemy's built-in session management
* format
* format
* use get_client directly
- Refactored the handling of nested columns in `sql_column_handler.py` to prioritize source-provided children, ensuring they override any derived children.
- Removed the overridden `_process_col_type` method in `bigquery/metadata.py` to streamline column type handling, enforcing the use of the standard path for BigQuery.
This update improves the accuracy of column metadata processing and simplifies the codebase.
* Refactor SQL column processing and enhance BigQuery project ID handling
* Introduced a new `process_column` function in `sql_column_handler.py` to streamline column processing logic.
* Updated `BigquerySource` to improve project ID retrieval from service connections, ensuring compatibility with various credential types.
* Added handling for nested columns in BigQuery schema processing.
* Enhanced error handling and logging for better debugging during project ID setup.
* Add support for GCP Application Default Credentials in BigQuery ingestion
* Enhanced `BigquerySource` to include handling for GCP Application Default Credentials (ADC).
* Updated JSON schema for GCP credentials to define `gcpADC` and its properties.
* Improved logging for credential setup in `set_google_credentials` function.
* Added comments and TODOs for future enhancements related to project ID fetching from the resource manager.
* Update .gitignore to include cursor rules files
* Added .cursorrules and .cursor/ to the .gitignore to prevent tracking of cursor rule files in the repository.
* This change helps maintain a cleaner repository by excluding unnecessary files from version control.
* refactor: Bigquery Credentials to allow multiple project ids
* fix: Handle unknown array data types in SQL column processing
* feat: allow user to turn off rnadomized sample
* style: ran python linting
* fix: models default value for randomizedSample
* style: ran linting
* doc: move config to advanced
* Fix#19667: OpenSearch Connector
* Fix#19667: OpenSearch Connector
* do not ingest any system level indexes
* fix pyformat
* Add AWS auth
* Use common schema and fix ssl config in client
* Add openseach connector docs and update schema
* Remove api key auth type and complete docs checklist
* Remove unnecessary httpx dependency and pyformat
* Add compatible version of httpx for elasticsearch
* Fix pylint fails and py-tests validation error
---------
Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
