3502 Commits

Author SHA1 Message Date
Pere Miquel Brull
91ddfcf07a
FIX - Postgres Python CI (#17859)
* fix deprecation

* remove print

* tests use testcontainers for mysql

* testcontainers for 3.8 dont work
2024-09-17 11:57:32 +02:00
Onkar Ravgan
507efc647f
fixed alationsink test (#17883) 2024-09-17 10:36:33 +02:00
Ayush Shah
23c6aa3ca5
fix: add children to array of struct (#17850) 2024-09-16 18:02:17 +05:30
Pere Miquel Brull
9a96ddd4ae
GEN-1494 - Add CI for py-tests with Postgres and Opensearch (#17853)
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1410 - Improve PII Logging information

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* allow_module_level=True
2024-09-16 09:35:38 +02:00
Ayush Shah
2feb7d04ca
MINOR: Add Matillion SSL + Docs implementation (#17792) 2024-09-16 12:57:12 +05:30
Sriharsha Chintalapani
c34545adeb
Issue-15768: Support Metric Entity (#17680)
* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Fix tests

* Fix tests

* Fix tests

* Minor: Fix tests

* ui: add metricsAPI rest utils

* ui: metric list page part 1

* feat: Add metric translations for multiple languages

* chore: Add "metric" field to SearchIndexingApplication schema

* ui: add create metric page

* ui: metric details page patch 1

* ui: add custom property and lineage support for metric entity

* ui: add expression component

* ui: add metric summary component

* chore: Update tab labels in MetricDetails and MetricVersion components

* ui: show other info like metric type, granularity, etc

* feat: Add support for metric entity in search dropdown

* feat: Rename custom property to Metric in MetricEntity.md

* feat: Add OwnerLabel component to MetricListPage

* Fix expression field in Metric

* chore: update expression to metricExpression

* ui: add metric header component with edit option

* Add metric to SearchIndexApp

* chore: Update expression to metricExpression

* ui: allow metric expression edit

* ui: update metric icon

* minor improvements

* Fix lineage indexing for Metric

* Update GlobalSettingsClassBase.ts to use MetricIcon for metrics in the global settings menu

* Fix error handling in MetricListPage component

* add related metrics

* minor improvements

* Fix relatedTerms patch

* Fix relatedTerms validation

* Add Boolean for deleted

* filter active entity from related metric list

* playwrite e2e part 1

* Refactor MetricSummary component to include RelatedMetrics in the summary panel

* test: add playwright test for metric special cases

* Add 'Metrics' to Explore Tree

* test: add e2e for add metric page

* test: add test for metric listing page content

* Add Boolean for deleted, remove deleted from suggests

* Refactor LineageProvider to handle deleted flag properly

* add playwright for metric listing

* fix test

* Add colored metric icon and update its usage in GlobalSettingsClassBase

* Fixed py_test test_ometa_endpoint for metric

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: SumanMaharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-09-16 11:36:31 +05:30
Pere Miquel Brull
e820cb0450
GEN-1410 - Improve PII Logging information (#17835) 2024-09-16 07:05:30 +02:00
k.nakagaki
3d8e30142c
Fixes 8428: make it possible to choice a sampling method type when we create profile ingestion for the Snowflake (#17831)
* Add test for existing code

* Add sampling method at ingestion.

* add samplingMethodType into UI

* modify init method to use new parameter.

* create descriptions

* execute isort

* fix an unintended change.

* apply py_format

* close section

* specify  init arguments

* fix bug

* apply py_format

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-09-15 21:51:17 +02:00
Mayur Singal
1fdb667ee3
MINOR: make type optional in ES Response (#17832)
* MINOR: make type optional in ES Mixin

* make none as default

* pyformat
2024-09-13 10:17:12 +02:00
Mayur Singal
f2c75be78d
MINOR: Optimise Snowflake Test Connection (#17779) 2024-09-13 12:55:55 +05:30
Imri Paran
59854de58d
[GEN-1406]: import plugin and clear plugin message for failed source import (#17788)
* support side effects on source classes by always importing source class

* streamlined error message

* fixed service type extraction for test suite pipeline

* - replaced "custom" with constant
- added quotes for the plugin exception for copy/paste ergonomics
2024-09-12 16:13:03 +02:00
Imri Paran
3282b057d8
ci: pin spacy version (#17814)
spacy 3.8 requires numpy 2.0 which is not compatible with openmetadata-ingestion requirements:
184e508d9c
2024-09-12 15:12:53 +05:30
Imri Paran
b70b3ce913
added config logging with secrets redacted (#17770) 2024-09-12 10:19:53 +02:00
Pere Miquel Brull
f4d0eba08f
MINOR - Add ES pagination with error handling (#17776)
* MINOR - Add ES pagination with error handling

* format

* format

* add nullable

* prepare API

* fix pagination

* format
2024-09-12 07:14:56 +02:00
Imri Paran
a3d6c1dd20
MINOR: tests(datalake): use minio (#17805)
* tests(datalake): use minio

1. use minio instead of moto for mimicking s3 behavior.
2. removed moto dependency as it is not compatible with aiobotocore (https://github.com/getmoto/moto/issues/7070#issuecomment-1828484982)

* - moved test_datalake_profiler_e2e.py to datalake/test_profiler
- use minio instead of moto

* fixed tests

* fixed tests

* removed default name for minio container
2024-09-12 07:13:01 +02:00
harshsoni2024
e7cfa4889c
MINOR: profiler e2e fix (#17809) 2024-09-11 22:04:22 +05:30
Mayur Singal
454467620c
MINOR: Fix Redshift View Def regex_replace Error (#17810) 2024-09-11 21:02:14 +05:30
harshsoni2024
4f57f4e710
GEN-309: OpenAPI Connector (#17754)
* configure api service metadata

* add rest api service

* fix test con. pyformat changes

* add models, fix test con.

* improve test con.

* add docs, side doc

* fix model data parse, url error fix

* add tests

* fix pytest errors

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-09-11 13:36:53 +05:30
harshsoni2024
3922f45a3e
fix typo in profiler metric (#17784) 2024-09-10 22:31:23 +05:30
Suman Maharana
094bae7097
Fixes #15565 : Advanced Avro schema recursion depth issue (#17683)
* Fix: Avro schema recursion depth issue

* py_format

* Addressed comments
2024-09-10 18:26:42 +05:30
Mayur Singal
b142882848
MINOR: Fix redshift ingestion warnings in case of ext table (#17767) 2024-09-10 15:32:35 +05:30
Teddy
e4c01c5702
fix: region typo in test (#17766) 2024-09-09 17:54:07 +05:30
Mayur Singal
9107c41a23
GEN-1251: Fix Expected ColumnLineage but got dict (#17708) 2024-09-09 17:52:51 +05:30
Pere Miquel Brull
c309906a1b
MINOR - Bump Presidio Analyzer and validate support for legal entities (#17750) 2024-09-06 16:07:08 +02:00
Pere Miquel Brull
1a276450fa
MINOR - Fix lineage GET for names with / and standardize quote calls (#17748)
* MINOR - Fix lineage GET for names with `/` and standardize quote calls

* format

* fix import
2024-09-06 12:12:44 +02:00
Imri Paran
84be1a3162
Fix 17698: use resolution logic for snowflake system metrics profiler (#17699)
* fix(profiler): snowflake

resolve tables using the snowflake engine instead of OpenMetadata

* added env for cleaning up dbs in E2E

* moved system metric method to profiler. all the rest says in snowflake

* format

* revert unnecessary changes

* removed test for previous resolution method

* use shutdown39
2024-09-06 07:25:10 +00:00
Ayush Shah
b2f21fa070
Fixes #17085: Update Oracle count & unique count function to handle HexByteString/Blob types (#17596) 2024-09-06 12:42:41 +05:30
Pere Miquel Brull
8191202850
MINOR - Better PII classification for JSON data (#17734)
* MINOR - Better PII classification for JSON data

* linting
2024-09-06 08:54:23 +02:00
Teddy
a7b6279f2e
GEN-1333 Add TS validation on DQ and Porfiler data ingestion (#17731)
* fix: added ts validation for DQ and profiler data

* fix: change ts to ms in pytest

* style: ran python linting
2024-09-06 08:16:31 +02:00
Onkar Ravgan
6cd07071d7
Fixed dbt cloud run_res name (#17694) 2024-09-04 16:58:29 +05:30
Pere Miquel Brull
2237d5a8d5
MINOR - PII Scanner tests and log levels (#17686)
* MINOR - PII Scanner tests and log levels

* MINOR - PII Scanner tests and log levels
2024-09-04 12:11:07 +02:00
nicor88
235337dbcd
fixes 17648: add tags and description for tableau published data source (#17678)
* tableau data-source tags and description

* tweaks

* PR review

* change test description

* black formatting
2024-09-03 11:25:27 +02:00
Suman Maharana
e93cf23d6c
Fixes: Parse postgres json column fields (#17645) 2024-09-03 11:10:47 +05:30
Mayur Singal
77d98c9675
MINOR: Fix athena lineage models (#17662) 2024-09-02 14:19:13 +05:30
Pere Miquel Brull
bcb29b46da
MINOR - Implement SAP Hana Lineage (#17615)
* MINOR - SAP Hana Lineage

* skeleton

* parser

* lineage

* manage formulas

* add cvs

* add cvs

* better typing

* enum

* handle cvs

* saphana docs
2024-08-30 07:42:43 +02:00
Ayush Shah
9245c8965f
fix: Bump Ingestion and Dockerfile versions (#17624) 2024-08-29 12:03:30 +05:30
nicor88
5eae1e371c
fix ingestion of owner in dbt, via email (#17613) 2024-08-28 18:24:50 +05:30
Pere Miquel Brull
2180a6c7f1
FIX - profiler interface system metrics validation & e2e YAML includeDDL (#17562) 2024-08-23 09:00:18 +02:00
Imri Paran
b48c6a0485
feat(postgres): add money profile (#17558)
add support for profiling of money type
2024-08-22 14:53:34 -07:00
Pere Miquel Brull
519b3c32e3
MINOR - Speek up redshift test connection (#17553) 2024-08-22 11:12:09 -07:00
Onkar Ravgan
bbe92e2af3
MINOR: Fix none password issue for mysql and postgresql databases (#17548)
* fix none password issue

* added warning
2024-08-22 17:34:34 +05:30
kwgdaig
43a244fbf1
ISSUE-17045: Modified to create column linage even when upstream columns and data source columns are one-to-many (#17112)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-08-22 09:07:05 +02:00
Imri Paran
2dd613b2a7
tests: lineage (#17436)
add test for cell with 100k characters
2024-08-21 20:28:08 -07:00
Mayur Singal
dd17ee739a
MINOR: Fix output handler time ingestion (#17429)
* MINOR: Fix output handler time ingestion

* chore: fixes Lint error

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2024-08-21 21:07:26 +05:30
Ayush Shah
7ada44a315
Minor: Remove trailing slash from FivetranClient base URL (#17528)
* fix: Remove trailing slash from FivetranClient base URL

* chore: use a method created for removing trailing slash
2024-08-21 13:02:16 +00:00
Ayush Shah
c08b83f41d
MINOR: Fixes AttributeError: 'DatalakeGcsClient' object has no attribute 'project' (#17526) 2024-08-21 17:51:35 +05:30
Imri Paran
5133c31d31
MINOR: kafka integration tests (#17457)
* tests: kafka integration

kafka integration tests with schema registry

* added ignore kafka for python 3.8

* fixed tests
2024-08-21 16:05:09 +05:30
IceS2
ac6b192fb3
MINOR: Fix pydantic v2 issues with domo (#17507)
* Fix pydantic v2 issues with domo

* Fix sourceURL for domo charts
2024-08-21 11:10:12 +02:00
Imri Paran
c055620ff4
tests: lineage (#17509)
added test cases for lineage with and without includeDDL
2024-08-21 07:47:30 +00:00
Do Manh Ha
a868596db7
Fixes #17461: Unquote and interpret escaped characters in BigQuery dataset description (#17462)
* fix(bigquery): unquote and convert any escaped characters to their actual representations

* test: bigquery description with multiple line

---------

Co-authored-by: Imri Paran <imri.paran@gmail.com>
2024-08-20 17:56:19 +02:00