38 Commits

Author SHA1 Message Date
Keshav Mohta
0796c6274b
Fixes: Databricks httpPath Required (#20611)
* fix: made databricks httpPath required and added a migration file for the same

* fix: added sql migration in postDataMigration file and fix databricks tests

* fix: added httpPath in test_source_connection.py and test_source_parsing.py files

* fix: added httpPath in test_databricks_lineage.py

* fix: table name in postgres migration
2025-04-07 13:33:55 +05:30
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
mgorsk1
3d2dfeb583
feat: use native trino client authentication classes (#16196)
---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 12:54:42 +05:30
Nicola Coretti
7ebc62dca7
feat: Add support for exasol datasource (#17166)
* Add flake.nix

* Add lockfile for flake

* Update nix environment and document usage

* Add schema for exasol connector

* Add Exasol definitions to databaseService

* Fix error in exasol connector schema

* Add additional connection options/settings to exasol connector

* Add exasol-connector to ui

* Add depdencies for exasol-connector

* Update notes

* Update ingestion code

* Add Basic Documentation for Exasol Connector

* Update flake file

* Add developer notes

* Add python script which can be used as entry point for debugging in ide

* Add config file which can be used for debugging (manual execution)

* Update debug script

* Update developer notes

* Remove old developer notes

* Add .venv to gitignore

* Update dev notes

* Update development notes

* Update ExasolSource

* Establish basic connection to Exasol DB from connector

* Update exasol connector connection settings

* Add service_spec for exasol plugin

* Remove development files

* Remove unused module

* Applied code formatter

* Update exasol dependency constraint(s)

* Add unit test for exasol connection url(s)

* Fixed test expectations for exasol connection url test(s)

* Adjust the test query for the Exasol connection test
2024-10-31 08:11:30 +01:00
Mayur Singal
a677910982
Fix #16211: Fix special charechter encoding in trino username (#16258)
* Fix #16211: Fix special charechter encoding in trino username

* fix test and add comments
2024-05-15 14:48:22 +05:30
Onkar Ravgan
3114ef9b5c
Add support for JWT auth in Trino (#12823)
* Added jwt field in trino

* fixed postgres migration

* fixed pytests
2023-08-17 07:22:58 +02:00
Pere Miquel Brull
a82c2f001c
Fix #9704 - Add inline TNS string (#12711)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-08-07 11:04:13 +02:00
vanshika18
72e4d0070f
Support MSSLQ handle multiple databases with pyodbc connection (#12526) 2023-07-26 10:43:24 +05:30
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Keith Sirmons
97b58c65f5
Impalaconnection (#11151)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala

* added cast to decimal for column when running average in mean.py

* fixed lint error

* fixed ui ordering of precision and scale.
Precision should be ordred in front of scale since the precision is set first in decimal data types

* first pass for impala connector

* updated default auth_mechanism to be one of the enum values.

* updated UI documentation to match fields for the impalaconneciton.

refined impalaConnection to bring use_ssl to a boolean instead or relying on an extra connection option being manually added.

Removed reference to hive for type mapping

added impala to the pip setup

* py_format updates

* removed print statement

* Lints and fixes

* Updated database documentation to follow new style

* Flag as BETA

* Remove tests

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-21 09:57:13 +02:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning (#10603)
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Mayur Singal
0fafbf70ba
Support postgres multiple DB ingestion for user with no access (#10207) 2023-02-23 15:42:40 +05:30
Pere Miquel Brull
84348d4748
Fix #8866 - bump datamodel-codegen (#9623)
* Fix #8866 - bump datamodel-codegen

* Update connection options and arguments structure

* Add builders test

* Format

* Allow Any values in componentConfig

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-01-09 13:20:32 +01:00
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients (#9545) 2023-01-02 13:52:27 +01:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change (#8909) 2022-11-21 12:35:23 +01:00
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Milan Bariya
4d16be2608
Fix: Hive connection issue (#8583)
* Fix: Hive connection issue

* Change based on comments
2022-11-08 16:22:37 +01:00
Milan Bariya
1565aa7733
FIX: Trino&Presto catalogs as databases (#8189)
* FIX: Trino&Presto catalogs as databases

* FIX: Trino&Presto catalogs as databases

* Change based on comments
2022-10-18 20:00:17 +05:30
Milan Bariya
5b3c877dd1
FIX:connection failed because of having '@' character in uname and pass (#8172) 2022-10-17 16:57:55 +05:30
Milan Bariya
cd43f1fdcf
Fix: connector for hive has double dash (#7953) 2022-10-05 12:01:13 +02:00
NiharDoshi99
f0f0476396
Added http & https scheme options in Hive connector (#6899)
* Added other scheme options in Hive connector

* Added other scheme options in Hive connector
2022-08-24 18:30:30 +05:30
Mayur Singal
7c004a6785
Fix #6823: Fix hive attribute error (#6824)
* Fix #6823: Fix hive attribute error

* Empty commit to start test
2022-08-19 14:22:54 +02:00
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579)
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
2022-08-05 10:28:04 +05:30
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441)
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
2022-08-02 09:13:46 +02:00
Mayur Singal
ebc9a24ee0
Connection options fix (#5971)
* Connection Options Fix

* Fixed Condition
2022-07-08 17:44:18 +02:00
Pere Miquel Brull
02fece727b
Fix #5307 - Define a better topology for Ingestion Sources (#5366)
Fix #5307 - Define a better topology for Ingestion Sources (#5366)
2022-06-13 12:17:54 +02:00
Milan Bariya
ec1aca926a
Fix #5210: Review database field for connectors (#5225)
* Fix #5210: Review database field for connectors
2022-06-06 10:48:23 +05:30
Abhishek Pandey
12e8a1fcf6
required-fields-updated-in-snowflake-and-athena (#5143)
required-fields-updated-in-snowflake-and-athena (#5143)
2022-05-30 14:57:26 +02:00
Abhishek Pandey
8fd69e7eb8
source-connection-test-updated (#5126) 2022-05-25 07:36:00 +02:00
codingwithabhi
da92d07c78
source-connection-unitest-added (#4916) 2022-05-13 13:48:04 +05:30
Milan Bariya
d1f21911f5
Add: Unit Test source for multiple conn (#4845)
Add: Unit Test source for multiple conn (#4845)
2022-05-12 14:31:09 +02:00
Milan Bariya
0acf88e1ba
ISSUE-1072: Add Apache Pinot support (#4649)
* ISSUE-1072: Add Apache Pinot support

* ISSUE-1072: Add Apache Pinot support

* Rename pinotdbConnection.json to pinotDBConnection.json
2022-05-11 11:13:28 +02:00
Mayur Singal
7e684a91eb
Fixed #4769: Trino Connection Argument Fix (#4794)
* Fixed #4769: Trino Connection Argument Fix

* Added tests

* Imporved test with proxy and without conn args
2022-05-09 12:46:03 +02:00
Mayur Singal
355e6dc6b0
Fixed Druid Source Connector (#4476) 2022-04-25 06:46:35 -07:00
Onkar Ravgan
0536cdedcf
Fixes for vertica password issue and added connection url test (#4323)
* rebase with main

* optimized password logic and added tests

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-04-22 07:38:35 +02:00
Mayur Singal
c48bc18065
Fix Hive & DB2 (#4147) 2022-04-14 10:27:05 -07:00
Mayur Singal
a3ceea6b63
Hive Source Fix (#3919)
* Hive Source Fix

* Added auth option test & unified unit test
2022-04-08 22:29:48 +05:30