2535 Commits

Author SHA1 Message Date
Mayur Singal
1be238266a
MINOR: Apply multi threading in View Lineage Processing (#17922) 2024-09-19 15:06:38 +02:00
Pere Miquel Brull
6a1cd0ef8b
GEN-1493 - Fix paginate_es in opensearch (#17858)
* GEN-1493 - Fix opensearch pagination

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
2024-09-17 14:21:10 +02:00
Pere Miquel Brull
91ddfcf07a
FIX - Postgres Python CI (#17859)
* fix deprecation

* remove print

* tests use testcontainers for mysql

* testcontainers for 3.8 dont work
2024-09-17 11:57:32 +02:00
Ayush Shah
2feb7d04ca
MINOR: Add Matillion SSL + Docs implementation (#17792) 2024-09-16 12:57:12 +05:30
Sriharsha Chintalapani
c34545adeb
Issue-15768: Support Metric Entity (#17680)
* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Fix tests

* Fix tests

* Fix tests

* Minor: Fix tests

* ui: add metricsAPI rest utils

* ui: metric list page part 1

* feat: Add metric translations for multiple languages

* chore: Add "metric" field to SearchIndexingApplication schema

* ui: add create metric page

* ui: metric details page patch 1

* ui: add custom property and lineage support for metric entity

* ui: add expression component

* ui: add metric summary component

* chore: Update tab labels in MetricDetails and MetricVersion components

* ui: show other info like metric type, granularity, etc

* feat: Add support for metric entity in search dropdown

* feat: Rename custom property to Metric in MetricEntity.md

* feat: Add OwnerLabel component to MetricListPage

* Fix expression field in Metric

* chore: update expression to metricExpression

* ui: add metric header component with edit option

* Add metric to SearchIndexApp

* chore: Update expression to metricExpression

* ui: allow metric expression edit

* ui: update metric icon

* minor improvements

* Fix lineage indexing for Metric

* Update GlobalSettingsClassBase.ts to use MetricIcon for metrics in the global settings menu

* Fix error handling in MetricListPage component

* add related metrics

* minor improvements

* Fix relatedTerms patch

* Fix relatedTerms validation

* Add Boolean for deleted

* filter active entity from related metric list

* playwrite e2e part 1

* Refactor MetricSummary component to include RelatedMetrics in the summary panel

* test: add playwright test for metric special cases

* Add 'Metrics' to Explore Tree

* test: add e2e for add metric page

* test: add test for metric listing page content

* Add Boolean for deleted, remove deleted from suggests

* Refactor LineageProvider to handle deleted flag properly

* add playwright for metric listing

* fix test

* Add colored metric icon and update its usage in GlobalSettingsClassBase

* Fixed py_test test_ometa_endpoint for metric

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: SumanMaharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-09-16 11:36:31 +05:30
Pere Miquel Brull
e820cb0450
GEN-1410 - Improve PII Logging information (#17835) 2024-09-16 07:05:30 +02:00
k.nakagaki
3d8e30142c
Fixes 8428: make it possible to choice a sampling method type when we create profile ingestion for the Snowflake (#17831)
* Add test for existing code

* Add sampling method at ingestion.

* add samplingMethodType into UI

* modify init method to use new parameter.

* create descriptions

* execute isort

* fix an unintended change.

* apply py_format

* close section

* specify  init arguments

* fix bug

* apply py_format

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-09-15 21:51:17 +02:00
Mayur Singal
1fdb667ee3
MINOR: make type optional in ES Response (#17832)
* MINOR: make type optional in ES Mixin

* make none as default

* pyformat
2024-09-13 10:17:12 +02:00
Mayur Singal
f2c75be78d
MINOR: Optimise Snowflake Test Connection (#17779) 2024-09-13 12:55:55 +05:30
Imri Paran
59854de58d
[GEN-1406]: import plugin and clear plugin message for failed source import (#17788)
* support side effects on source classes by always importing source class

* streamlined error message

* fixed service type extraction for test suite pipeline

* - replaced "custom" with constant
- added quotes for the plugin exception for copy/paste ergonomics
2024-09-12 16:13:03 +02:00
Imri Paran
b70b3ce913
added config logging with secrets redacted (#17770) 2024-09-12 10:19:53 +02:00
Pere Miquel Brull
f4d0eba08f
MINOR - Add ES pagination with error handling (#17776)
* MINOR - Add ES pagination with error handling

* format

* format

* add nullable

* prepare API

* fix pagination

* format
2024-09-12 07:14:56 +02:00
harshsoni2024
e7cfa4889c
MINOR: profiler e2e fix (#17809) 2024-09-11 22:04:22 +05:30
Mayur Singal
454467620c
MINOR: Fix Redshift View Def regex_replace Error (#17810) 2024-09-11 21:02:14 +05:30
harshsoni2024
4f57f4e710
GEN-309: OpenAPI Connector (#17754)
* configure api service metadata

* add rest api service

* fix test con. pyformat changes

* add models, fix test con.

* improve test con.

* add docs, side doc

* fix model data parse, url error fix

* add tests

* fix pytest errors

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-09-11 13:36:53 +05:30
harshsoni2024
3922f45a3e
fix typo in profiler metric (#17784) 2024-09-10 22:31:23 +05:30
Suman Maharana
094bae7097
Fixes #15565 : Advanced Avro schema recursion depth issue (#17683)
* Fix: Avro schema recursion depth issue

* py_format

* Addressed comments
2024-09-10 18:26:42 +05:30
Mayur Singal
b142882848
MINOR: Fix redshift ingestion warnings in case of ext table (#17767) 2024-09-10 15:32:35 +05:30
Mayur Singal
9107c41a23
GEN-1251: Fix Expected ColumnLineage but got dict (#17708) 2024-09-09 17:52:51 +05:30
Pere Miquel Brull
c309906a1b
MINOR - Bump Presidio Analyzer and validate support for legal entities (#17750) 2024-09-06 16:07:08 +02:00
Pere Miquel Brull
1a276450fa
MINOR - Fix lineage GET for names with / and standardize quote calls (#17748)
* MINOR - Fix lineage GET for names with `/` and standardize quote calls

* format

* fix import
2024-09-06 12:12:44 +02:00
Imri Paran
84be1a3162
Fix 17698: use resolution logic for snowflake system metrics profiler (#17699)
* fix(profiler): snowflake

resolve tables using the snowflake engine instead of OpenMetadata

* added env for cleaning up dbs in E2E

* moved system metric method to profiler. all the rest says in snowflake

* format

* revert unnecessary changes

* removed test for previous resolution method

* use shutdown39
2024-09-06 07:25:10 +00:00
Ayush Shah
b2f21fa070
Fixes #17085: Update Oracle count & unique count function to handle HexByteString/Blob types (#17596) 2024-09-06 12:42:41 +05:30
Pere Miquel Brull
8191202850
MINOR - Better PII classification for JSON data (#17734)
* MINOR - Better PII classification for JSON data

* linting
2024-09-06 08:54:23 +02:00
Onkar Ravgan
6cd07071d7
Fixed dbt cloud run_res name (#17694) 2024-09-04 16:58:29 +05:30
Pere Miquel Brull
2237d5a8d5
MINOR - PII Scanner tests and log levels (#17686)
* MINOR - PII Scanner tests and log levels

* MINOR - PII Scanner tests and log levels
2024-09-04 12:11:07 +02:00
nicor88
235337dbcd
fixes 17648: add tags and description for tableau published data source (#17678)
* tableau data-source tags and description

* tweaks

* PR review

* change test description

* black formatting
2024-09-03 11:25:27 +02:00
Suman Maharana
e93cf23d6c
Fixes: Parse postgres json column fields (#17645) 2024-09-03 11:10:47 +05:30
Mayur Singal
77d98c9675
MINOR: Fix athena lineage models (#17662) 2024-09-02 14:19:13 +05:30
Pere Miquel Brull
bcb29b46da
MINOR - Implement SAP Hana Lineage (#17615)
* MINOR - SAP Hana Lineage

* skeleton

* parser

* lineage

* manage formulas

* add cvs

* add cvs

* better typing

* enum

* handle cvs

* saphana docs
2024-08-30 07:42:43 +02:00
nicor88
5eae1e371c
fix ingestion of owner in dbt, via email (#17613) 2024-08-28 18:24:50 +05:30
Pere Miquel Brull
2180a6c7f1
FIX - profiler interface system metrics validation & e2e YAML includeDDL (#17562) 2024-08-23 09:00:18 +02:00
Imri Paran
b48c6a0485
feat(postgres): add money profile (#17558)
add support for profiling of money type
2024-08-22 14:53:34 -07:00
Pere Miquel Brull
519b3c32e3
MINOR - Speek up redshift test connection (#17553) 2024-08-22 11:12:09 -07:00
Onkar Ravgan
bbe92e2af3
MINOR: Fix none password issue for mysql and postgresql databases (#17548)
* fix none password issue

* added warning
2024-08-22 17:34:34 +05:30
kwgdaig
43a244fbf1
ISSUE-17045: Modified to create column linage even when upstream columns and data source columns are one-to-many (#17112)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-08-22 09:07:05 +02:00
Mayur Singal
dd17ee739a
MINOR: Fix output handler time ingestion (#17429)
* MINOR: Fix output handler time ingestion

* chore: fixes Lint error

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2024-08-21 21:07:26 +05:30
Ayush Shah
7ada44a315
Minor: Remove trailing slash from FivetranClient base URL (#17528)
* fix: Remove trailing slash from FivetranClient base URL

* chore: use a method created for removing trailing slash
2024-08-21 13:02:16 +00:00
Ayush Shah
c08b83f41d
MINOR: Fixes AttributeError: 'DatalakeGcsClient' object has no attribute 'project' (#17526) 2024-08-21 17:51:35 +05:30
Imri Paran
5133c31d31
MINOR: kafka integration tests (#17457)
* tests: kafka integration

kafka integration tests with schema registry

* added ignore kafka for python 3.8

* fixed tests
2024-08-21 16:05:09 +05:30
IceS2
ac6b192fb3
MINOR: Fix pydantic v2 issues with domo (#17507)
* Fix pydantic v2 issues with domo

* Fix sourceURL for domo charts
2024-08-21 11:10:12 +02:00
Do Manh Ha
a868596db7
Fixes #17461: Unquote and interpret escaped characters in BigQuery dataset description (#17462)
* fix(bigquery): unquote and convert any escaped characters to their actual representations

* test: bigquery description with multiple line

---------

Co-authored-by: Imri Paran <imri.paran@gmail.com>
2024-08-20 17:56:19 +02:00
Teddy
354f879866
fix: db_service not passed to metric instantiating (#17504) 2024-08-20 07:57:03 -07:00
Teddy
4ed2035a50
fix: remove root when accessing fqn of entity reference (#17491) 2024-08-20 15:17:16 +02:00
Ayush Shah
9880f06b2c
Fixes #17489: Allow non numeric numbers to be sent via Json, Replace NaN value… (#17490)
* fix: Allow non numeric numbers to be sent via Json, Replace NaN values with None in SQAProfilerInterface

Replace NaN values with None in the SQAProfilerInterface class to maintain database parity. NaN values will be cast to null in OpenMetadata. This change ensures that data handling processes account for this conversion.

* fix: histogram overflow error

* test: Add Unit Test for Null and Null Ratio Metric

* chore: Address comments

* chore: Address comments

* fix: checkstyle and message

* fix: failing tests as null count works as expected
2024-08-20 16:33:55 +05:30
Imri Paran
5da7bb049c
MINOR: fix table profiler on empty tables in trino (#17471)
* fix(profiler): trino

coalesce row count to 0 if result is null. this value gets returned for empty tables

* fixed test_metadata.py
2024-08-20 08:42:10 +00:00
Imri Paran
2722eadc33
fix: gcs (#17486)
1. update docs using gcp credentials in path.
2. updated example `clientIt` in docs
3. fixed client to work with implicit project
4. fixed workflow to warn about missing buckets
2024-08-19 23:14:46 -07:00
Imri Paran
31c2ec8c57
MINOR: fix qlikcloud test connection (#17459)
* fix: qlikcloud test connection

* patch test_connection for qlik cloud unit tests
2024-08-19 23:14:09 -07:00
IceS2
ddd8c41864
Fix DB2 Schema Trailing Whitespaces (#17475) 2024-08-19 23:13:14 -07:00
Imri Paran
7508848376
fix(dq): data types for unique columns (#17431)
1. remove json and array from supported data types of unique column test.
2. migrations.
3. tests.
2024-08-19 14:28:42 +02:00