3385 Commits

Author SHA1 Message Date
Mayur Singal
5660a751e3
GEN-2000: Add Support for PowerBI Report Server (#18513) 2024-11-06 14:55:05 +05:30
Suman Maharana
426ad2000b
Fix #17778 : Databricks query run optimisation (#18467)
* Fix : Databricks query run  optimization

* Fixed dialect error

* fix get columns

* py format

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-06 10:10:01 +05:30
IceS2
dccba20101
Return s3 endpoint as str() instead of Url (#18521) 2024-11-05 17:39:50 +00:00
Katarzyna Kałek
47c75fe6a7
Enhanced Glue ingestion with external table features (#18511)
* added fileFormat, locationPath and external table lineage to Glue ingestion

* Improve Lineage Label

---------

Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-05 21:48:20 +05:30
Imri Paran
84391e7078
MINOR: tests: fix Tuple in bigquery e2e cli (#18499)
* tests: fix Tuple in bigquery e2e cli

* tests: fix Tuple in bigquery e2e cli

* fix workflow condition
2024-11-04 09:54:10 -08:00
Teddy
9a685d5f19
fix: pass row and result computation for inSet test (#18466) 2024-10-31 08:15:18 +00:00
Nicola Coretti
7ebc62dca7
feat: Add support for exasol datasource (#17166)
* Add flake.nix

* Add lockfile for flake

* Update nix environment and document usage

* Add schema for exasol connector

* Add Exasol definitions to databaseService

* Fix error in exasol connector schema

* Add additional connection options/settings to exasol connector

* Add exasol-connector to ui

* Add depdencies for exasol-connector

* Update notes

* Update ingestion code

* Add Basic Documentation for Exasol Connector

* Update flake file

* Add developer notes

* Add python script which can be used as entry point for debugging in ide

* Add config file which can be used for debugging (manual execution)

* Update debug script

* Update developer notes

* Remove old developer notes

* Add .venv to gitignore

* Update dev notes

* Update development notes

* Update ExasolSource

* Establish basic connection to Exasol DB from connector

* Update exasol connector connection settings

* Add service_spec for exasol plugin

* Remove development files

* Remove unused module

* Applied code formatter

* Update exasol dependency constraint(s)

* Add unit test for exasol connection url(s)

* Fixed test expectations for exasol connection url test(s)

* Adjust the test query for the Exasol connection test
2024-10-31 08:11:30 +01:00
Imri Paran
016a840b2f
MINOR Fix snowflake profiler by using case-insensitive strings (#18438)
* use snowflake system metrics computer instead of source

* reverted pylint

* use case-insensitive strings equality for snowflake filters
2024-10-29 18:33:36 +01:00
Suman Maharana
67a9e63439
Minor: Fixed dbtcloud test connection and improved docs (#18408) 2024-10-29 14:39:52 +05:30
Onkar Ravgan
4a0c8406e9
[ER Diagrams] Add ER diagram APIs and sample data (#18021)
* Add ER diag APIs and sample data

* fix pylint

* formatting fixes2

* fixed es client return

* fixed os client return

* supported TableDetailPage tabs as classBase for supporting collate only tabs

* Added schema Apis

* change the base class to .ts and move the component in the util files

* beautify function arguments

* Added optimizations

* Ingestion changes

* svg dimension change

* supported class base tab in databaseSchema

* supported classBase action button in schema table name column

* added further keys data for constraint modal

* fix sonar issue

* remove old method to override edit action on column and shifted to DisplayNameModal for fields

* supported table right panel component to further extends on collate side

* minor fix around duplicate constraint

* added support to update table constraints and column constraints in the UI

* code optimization and minor fixes

* review comments and multi col fix

* added queryFilter option in NodeSuggestion and tableConstrainst to fetch and use only in service tables

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-10-28 20:26:19 +05:30
Mayur Singal
9d91325af8
Lineage-1: Move view lineage processing to lineage workflow (#18220)
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-10-28 18:18:22 +05:30
Nicola Coretti
24de281026
Fix docstring in the Doris metadata module (#18421) 2024-10-28 17:58:04 +05:30
Mayur Singal
4083838056
MINOR: Couchbase Secondary Index Fix (#18398) 2024-10-24 20:34:55 +05:30
harshsoni2024
1a8bba6058
GEN-1911: Quicksight lineage source fix (#18348) 2024-10-24 11:41:37 +05:30
Imri Paran
95982b9395
[GEN-356] Use ServiceSpec for loading sources based on connectors (#18322)
* ref(profiler): use di for system profile

- use source classes that can be overridden in system profiles
- use a manifest class instead of factory to specify which class to resolve for connectors
- example usage can be seen in redshift and snowflake

* - added manifests for all custom profilers
- used super() dependency injection in order for system metrics source
- formatting

* - implement spec for all source types
- added docs for the new specification
- added some pylint ignores in the importer module

* remove TYPE_CHECKING in core.py

* - deleted valuedispatch function
- deleted get_system_metrics_by_dialect
- implemented BigQueryProfiler with a system metrics source
- moved import_source_class to BaseSpec

* - removed tests related to the profiler factory

* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger

* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger

* fixed tests

* format

* bigquery system profile e2e tests

* fixed module docstring

* - removed import_side_effects from redshift. we still use it in postgres for the orm conversion maps.
- removed leftover methods

* - tests for BaseSpec
- moved get_class_path to importer

* - moved constructors around to get rid of useless kwargs

* - changed test_system_metric

* - added linage and usage to service_spec
- fixed postgres native lineage test

* add comments on collaborative constructors
2024-10-24 07:47:50 +02:00
Ayush Shah
51347a981a
fixes: Mode test connection returns data in dict instead of json (#18386) 2024-10-24 11:11:39 +05:30
Vijay Lakshmanan
4f2ef6fe5c
Fixes #16263: Fixed Mode dashboard ingestion API call (#18355) 2024-10-23 12:03:08 +05:30
Pere Miquel Brull
5e80ad9fc3
MINOR - Only timeout on main threads (#18341) 2024-10-21 15:18:33 +02:00
Teddy
dcf71aa0ea
fix: lazy load classes from factory method (#18321) 2024-10-21 11:29:03 +02:00
Mayur Singal
a4d62f6d85
MINOR: Add location path to table entity (#18307) 2024-10-21 10:31:27 +05:30
Pere Miquel Brull
c2929e67e6
MINOR - Return TestConnectionResult from test_connection_fn (#18320)
* MINOR - Return TestConnectionResult from test_connection fn

* MINOR - Return TestConnectionResult from test_connection fn
2024-10-18 09:54:07 +02:00
Pere Miquel Brull
5074f6588f
MINOR - Validate app runner init (#18316) 2024-10-18 09:40:06 +02:00
Katarzyna Kałek
c9995eecb6
FIX #18309: fixed task deserialization in Airflow metadata ingestion (#18310)
* fixed task deserialization in Airflow metadata ingestion

* fixed formatting

---------

Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
2024-10-17 14:51:55 -07:00
Pere Miquel Brull
7012e73d75
GEN-1166 - Improve Ingestion Workflow Error Summary (#18280)
* GEN-1166 - Improve Ingestion Workflow Error Summary

* fix test

* docs

* comments
2024-10-16 18:15:50 +02:00
Pere Miquel Brull
89b6c1c1cd
MINOR - Pass timeout to test connection and return TestConnectionStep (#18236)
* update connections

* MINOR - Pass timeout in test connection and return TestConnectionStep

* format

* comments

* comments
2024-10-16 18:15:28 +02:00
harshsoni2024
4f89dc582b
salesforce table description from label if not through query (#18286) 2024-10-16 12:56:44 +05:30
harshsoni2024
51448452d0
MINOR: Fix pinotdb col. datatype error (#18268) 2024-10-16 11:35:27 +05:30
Mayur Singal
592d7396bc
MINOR: Fix Couchbase columns not fetched (#18284) 2024-10-16 09:53:57 +05:30
Ayush Shah
40bd3bd3fa
Fixes #18186: Quicksight Ingestion Error handled (#18218) 2024-10-16 09:52:07 +05:30
Imri Paran
be82086e25
MINOR: add column case sensitivity parameter (#18115)
* fix(data-quality): table diff

- added handling for case-insensitive columns
- added handling for different numeric types (int/float/Decimal)
- added handling of boolean test case parameters

* add migrations for table diff

* add migrations for table diff

* removed cross type diff for now. it appears to be flaky

* fixed migrations

* use casefold() instead of lower()

* - implemented utils.get_test_case_param_value
- fixed params for case sensitive column

* handle bool test case parameters

* format

* testing

* format

* list -> List

* list -> List

* - change caseSensitiveColumns default to fase
- added migration to stay backward compatible

* - removed migration files
- updated logging message for table diff migration

* changed bool test case parameters default to always be false

* format

* docs: data diff

- added the caseSensitiveColumns parameter

requires: https://github.com/open-metadata/OpenMetadata/pull/18115

* fixed test_get_bool_test_case_param
2024-10-15 16:29:43 +02:00
Onkar Ravgan
e6705f25b3
fixed dbt tag name (#18273) 2024-10-15 16:43:03 +05:30
harshsoni2024
eb49d7a5bc
fix query for mysql con. (#18272) 2024-10-15 14:03:49 +05:30
Mayur Singal
8322c0f684
Fix #17963: Fix PinotDB Ingestion (#18266)
* Fix #17963: Fix PinotDB Ingestion

* fix conn args
2024-10-15 08:36:40 +05:30
Onkar Ravgan
2ee015e426
Add array supp for json schema parser (#18255) 2024-10-15 07:30:16 +05:30
Suman Maharana
dd08bc9ffd
GEN-895: Added Glue Pipeline Lineage (#18063) 2024-10-14 13:08:17 +05:30
Suman Maharana
142a506120
Fixed dbt Manifest and Run results parsing (#18234) 2024-10-14 13:07:22 +05:30
Imri Paran
3c7f995677
MINOR: fix(looker): exclude version
https://github.com/looker-open-source/sdk-codegen/issues/1518
2024-10-11 18:47:43 +00:00
Suman Maharana
69b34684b5
Fixed mysql E2E (#18229) 2024-10-11 10:49:03 +00:00
Imri Paran
d0ca05efbf
MINOR: add data quality tests (#18193)
* test: add dq tests

* wip

* fixed test_all_definition_exists
2024-10-11 12:07:58 +05:30
Imri Paran
bff50664b9
fix(data-quality): table diff url (#18221)
fixed url for table diff when using service connection
2024-10-11 01:07:29 +05:30
Pere Miquel Brull
f9e99f49e4
TEST - Add ES pagination with multiple filters (#18162)
* add filtering test

* adding test to paginate with filters
2024-10-10 17:14:22 +02:00
Imri Paran
68e71cb3dc
GEN-970: Refactor redshift system metrics to support freshness test (#17981)
* ref(profiler): redshift system metrics

- moved redshift system metrics to the redshift source module
- use Timestamp in data quality
- added plugin feature to test utils

* use timezone.utc

* format

* reverted unintended snowflake changes

* fixed import test_system_metrics.py

* revert

* fixed import in tests
2024-10-10 08:32:07 +02:00
IceS2
02d9494e7f
MINOR: Update DB2 dependencies to fix issue about not having ibm-db installed (#18192)
* Update DB2 dependencies to fix issue about not having ibm-db installed

* Fix checkstyle
2024-10-09 18:39:29 +02:00
devanshGarg-lumiq
fcd34afab5
Fix Redshift profiler for external tables in Redshift Spectrum (#18182) 2024-10-09 12:58:57 +02:00
Mayur Singal
e21843b752
Fix #18177: Fix Hive Metastore connection issue (#18178) 2024-10-09 11:34:41 +02:00
Prajwal214
bbd159b947
Minor: Updated pyiceberg version to 0.5.1 (#18155)
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
2024-10-08 14:55:57 +05:30
Sachin Chaurasiya
457f3d919a
GEN-1322: API Entity - Remove Beta (#17967)
* GEN-1322: API Entity - Remove Beta

* minor: add doc for the metadata pipeline

* api service refactor

* api service refactor backend changes

* add apiconnection in test service connection

* pytest fix

* fix java file formatting

* Fix casing of REST in ApiServiceRest.spec.ts

* Refactor REST to Rest in API classes

* minor change

* minor change

* minor change

* fix cashing for API to Api

* add playwright test for api service ingestion

* fix: playwright test

---------

Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
2024-10-08 14:39:55 +05:30
harshsoni2024
421ad601e9
GEN-1552: Postgres stored procedures support (#18083) 2024-10-07 14:28:03 +05:30
Ethan
49fceb4674
Fixes #18104 : change parse_obj and assertEquals which was deprecated (#18105)
* change deprecationwarning

* fix format python

* fix replace module

* change : java function name
2024-10-07 09:02:41 +02:00
Imri Paran
71720ebc51
fix(table-diff): support cross database (#18085)
fixed table diff url to include database in all cases
2024-10-04 15:31:17 +02:00