2916 Commits

Author SHA1 Message Date
Imri Paran
64f8571a77
pin testcontainers by minor version (#17938) 2024-09-24 09:18:36 +02:00
Aldwyn Cabarrubias
b9f881f2fa
Change time format for the created_at of the DBT cloud pipeline status (#17916)
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
2024-09-23 14:50:48 +05:30
Teddy
b222de66fe
Minor fix pylint version (#17944)
* fix import issue

* fix: fix pylint version

* style:ran python linting

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-20 18:50:28 +02:00
Pere Miquel Brull
4cccaae446
GEN-996 - Allow PII Processor without storing Sample Data (#17927)
* GEN-996 - Allow PII Processor without storing Sample Data

* fix import

* fix import
2024-09-20 16:05:29 +02:00
Ayush Shah
8087f48e15
Fixes #17869: Add validations in place for salesforce connection (#17870) 2024-09-20 16:33:38 +05:30
Pere Miquel Brull
1e56c76c0e
FIX #17896 - Python lineage SDK to work with Uuid & FQN models (#17928) 2024-09-20 10:37:41 +02:00
Imri Paran
d09bca26f6
MINOR: fix mssql integration test (#17923)
* change tag for sql server due to https://github.com/microsoft/mssql-docker/issues/441 (or some similar issue)

* use 2022-latest

* fixed mssql tests

* format

* used new columns

* use the custom sql server
2024-09-20 08:52:40 +02:00
Imri Paran
21af02d8d5
Fix 17903: fix(data-quality): snowflake data diff (#17907)
* fix(data-quality): snowflake data diff

- fixed schema in snowflake URL for data diff
- added e2e for snowflake data quality

* reverted unintended change
2024-09-19 17:55:48 +02:00
Mayur Singal
1be238266a
MINOR: Apply multi threading in View Lineage Processing (#17922) 2024-09-19 15:06:38 +02:00
Imri Paran
760b8eb742
fix(ingestion): deltalake (#17910)
add pin for deltalake<0.20 to avoid ad35eda798 (diff-f81b8b91e9721dc0a3235b6674a57c0700153b92b59f3988d357950a9c8c4760R1142-R1144)
2024-09-19 08:25:19 +02:00
Teddy
7df05c3e57
Minor python test failure (#17904)
* fix import issue

* chore: add raise from status on ingestion step in tests

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-18 17:41:05 +02:00
Teddy
33c50efdbf
GEN-1192 - Move Test Case to its Own Resource (#17862)
* feat: indexed test case results

* feat: added indexation logic for test case results

* style: ran java linting

* fix: IDE warnigns

* chore: added test case results migration

* style: ran java linting

* fix: postgres migration column json ref

* empty commit to trigger queued

* chore: extracted test case results to its own resource

* chore: fix failing tests

* chore: move testCaseResult state from testSuite and testCase to dynamic field fetched from test case results search index

* chore: clean up test case repository

* style: ran java linting

* chore: removed testCaseResultSummary and testCaseResult state from db

* fix: test failures

* chore: fix index mapping type for result value

* chore: fix test failure
2024-09-18 11:58:59 +02:00
Pere Miquel Brull
f6fdd86d5c
Docs - Ingestion License (#17893) 2024-09-17 08:58:53 -07:00
Pere Miquel Brull
6a1cd0ef8b
GEN-1493 - Fix paginate_es in opensearch (#17858)
* GEN-1493 - Fix opensearch pagination

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
2024-09-17 14:21:10 +02:00
Pere Miquel Brull
91ddfcf07a
FIX - Postgres Python CI (#17859)
* fix deprecation

* remove print

* tests use testcontainers for mysql

* testcontainers for 3.8 dont work
2024-09-17 11:57:32 +02:00
Onkar Ravgan
507efc647f
fixed alationsink test (#17883) 2024-09-17 10:36:33 +02:00
Ayush Shah
23c6aa3ca5
fix: add children to array of struct (#17850) 2024-09-16 18:02:17 +05:30
Pere Miquel Brull
9a96ddd4ae
GEN-1494 - Add CI for py-tests with Postgres and Opensearch (#17853)
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1410 - Improve PII Logging information

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* allow_module_level=True
2024-09-16 09:35:38 +02:00
Ayush Shah
2feb7d04ca
MINOR: Add Matillion SSL + Docs implementation (#17792) 2024-09-16 12:57:12 +05:30
Sriharsha Chintalapani
c34545adeb
Issue-15768: Support Metric Entity (#17680)
* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Fix tests

* Fix tests

* Fix tests

* Minor: Fix tests

* ui: add metricsAPI rest utils

* ui: metric list page part 1

* feat: Add metric translations for multiple languages

* chore: Add "metric" field to SearchIndexingApplication schema

* ui: add create metric page

* ui: metric details page patch 1

* ui: add custom property and lineage support for metric entity

* ui: add expression component

* ui: add metric summary component

* chore: Update tab labels in MetricDetails and MetricVersion components

* ui: show other info like metric type, granularity, etc

* feat: Add support for metric entity in search dropdown

* feat: Rename custom property to Metric in MetricEntity.md

* feat: Add OwnerLabel component to MetricListPage

* Fix expression field in Metric

* chore: update expression to metricExpression

* ui: add metric header component with edit option

* Add metric to SearchIndexApp

* chore: Update expression to metricExpression

* ui: allow metric expression edit

* ui: update metric icon

* minor improvements

* Fix lineage indexing for Metric

* Update GlobalSettingsClassBase.ts to use MetricIcon for metrics in the global settings menu

* Fix error handling in MetricListPage component

* add related metrics

* minor improvements

* Fix relatedTerms patch

* Fix relatedTerms validation

* Add Boolean for deleted

* filter active entity from related metric list

* playwrite e2e part 1

* Refactor MetricSummary component to include RelatedMetrics in the summary panel

* test: add playwright test for metric special cases

* Add 'Metrics' to Explore Tree

* test: add e2e for add metric page

* test: add test for metric listing page content

* Add Boolean for deleted, remove deleted from suggests

* Refactor LineageProvider to handle deleted flag properly

* add playwright for metric listing

* fix test

* Add colored metric icon and update its usage in GlobalSettingsClassBase

* Fixed py_test test_ometa_endpoint for metric

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: SumanMaharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-09-16 11:36:31 +05:30
Pere Miquel Brull
e820cb0450
GEN-1410 - Improve PII Logging information (#17835) 2024-09-16 07:05:30 +02:00
k.nakagaki
3d8e30142c
Fixes 8428: make it possible to choice a sampling method type when we create profile ingestion for the Snowflake (#17831)
* Add test for existing code

* Add sampling method at ingestion.

* add samplingMethodType into UI

* modify init method to use new parameter.

* create descriptions

* execute isort

* fix an unintended change.

* apply py_format

* close section

* specify  init arguments

* fix bug

* apply py_format

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-09-15 21:51:17 +02:00
Mayur Singal
1fdb667ee3
MINOR: make type optional in ES Response (#17832)
* MINOR: make type optional in ES Mixin

* make none as default

* pyformat
2024-09-13 10:17:12 +02:00
Mayur Singal
f2c75be78d
MINOR: Optimise Snowflake Test Connection (#17779) 2024-09-13 12:55:55 +05:30
Imri Paran
59854de58d
[GEN-1406]: import plugin and clear plugin message for failed source import (#17788)
* support side effects on source classes by always importing source class

* streamlined error message

* fixed service type extraction for test suite pipeline

* - replaced "custom" with constant
- added quotes for the plugin exception for copy/paste ergonomics
2024-09-12 16:13:03 +02:00
Imri Paran
3282b057d8
ci: pin spacy version (#17814)
spacy 3.8 requires numpy 2.0 which is not compatible with openmetadata-ingestion requirements:
184e508d9c
2024-09-12 15:12:53 +05:30
Imri Paran
b70b3ce913
added config logging with secrets redacted (#17770) 2024-09-12 10:19:53 +02:00
Pere Miquel Brull
f4d0eba08f
MINOR - Add ES pagination with error handling (#17776)
* MINOR - Add ES pagination with error handling

* format

* format

* add nullable

* prepare API

* fix pagination

* format
2024-09-12 07:14:56 +02:00
Imri Paran
a3d6c1dd20
MINOR: tests(datalake): use minio (#17805)
* tests(datalake): use minio

1. use minio instead of moto for mimicking s3 behavior.
2. removed moto dependency as it is not compatible with aiobotocore (https://github.com/getmoto/moto/issues/7070#issuecomment-1828484982)

* - moved test_datalake_profiler_e2e.py to datalake/test_profiler
- use minio instead of moto

* fixed tests

* fixed tests

* removed default name for minio container
2024-09-12 07:13:01 +02:00
harshsoni2024
e7cfa4889c
MINOR: profiler e2e fix (#17809) 2024-09-11 22:04:22 +05:30
Mayur Singal
454467620c
MINOR: Fix Redshift View Def regex_replace Error (#17810) 2024-09-11 21:02:14 +05:30
harshsoni2024
4f57f4e710
GEN-309: OpenAPI Connector (#17754)
* configure api service metadata

* add rest api service

* fix test con. pyformat changes

* add models, fix test con.

* improve test con.

* add docs, side doc

* fix model data parse, url error fix

* add tests

* fix pytest errors

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-09-11 13:36:53 +05:30
harshsoni2024
3922f45a3e
fix typo in profiler metric (#17784) 2024-09-10 22:31:23 +05:30
Suman Maharana
094bae7097
Fixes #15565 : Advanced Avro schema recursion depth issue (#17683)
* Fix: Avro schema recursion depth issue

* py_format

* Addressed comments
2024-09-10 18:26:42 +05:30
Mayur Singal
b142882848
MINOR: Fix redshift ingestion warnings in case of ext table (#17767) 2024-09-10 15:32:35 +05:30
Teddy
e4c01c5702
fix: region typo in test (#17766) 2024-09-09 17:54:07 +05:30
Mayur Singal
9107c41a23
GEN-1251: Fix Expected ColumnLineage but got dict (#17708) 2024-09-09 17:52:51 +05:30
Pere Miquel Brull
c309906a1b
MINOR - Bump Presidio Analyzer and validate support for legal entities (#17750) 2024-09-06 16:07:08 +02:00
Pere Miquel Brull
1a276450fa
MINOR - Fix lineage GET for names with / and standardize quote calls (#17748)
* MINOR - Fix lineage GET for names with `/` and standardize quote calls

* format

* fix import
2024-09-06 12:12:44 +02:00
Imri Paran
84be1a3162
Fix 17698: use resolution logic for snowflake system metrics profiler (#17699)
* fix(profiler): snowflake

resolve tables using the snowflake engine instead of OpenMetadata

* added env for cleaning up dbs in E2E

* moved system metric method to profiler. all the rest says in snowflake

* format

* revert unnecessary changes

* removed test for previous resolution method

* use shutdown39
2024-09-06 07:25:10 +00:00
Ayush Shah
b2f21fa070
Fixes #17085: Update Oracle count & unique count function to handle HexByteString/Blob types (#17596) 2024-09-06 12:42:41 +05:30
Pere Miquel Brull
8191202850
MINOR - Better PII classification for JSON data (#17734)
* MINOR - Better PII classification for JSON data

* linting
2024-09-06 08:54:23 +02:00
Teddy
a7b6279f2e
GEN-1333 Add TS validation on DQ and Porfiler data ingestion (#17731)
* fix: added ts validation for DQ and profiler data

* fix: change ts to ms in pytest

* style: ran python linting
2024-09-06 08:16:31 +02:00
Onkar Ravgan
6cd07071d7
Fixed dbt cloud run_res name (#17694) 2024-09-04 16:58:29 +05:30
Pere Miquel Brull
2237d5a8d5
MINOR - PII Scanner tests and log levels (#17686)
* MINOR - PII Scanner tests and log levels

* MINOR - PII Scanner tests and log levels
2024-09-04 12:11:07 +02:00
nicor88
235337dbcd
fixes 17648: add tags and description for tableau published data source (#17678)
* tableau data-source tags and description

* tweaks

* PR review

* change test description

* black formatting
2024-09-03 11:25:27 +02:00
Suman Maharana
e93cf23d6c
Fixes: Parse postgres json column fields (#17645) 2024-09-03 11:10:47 +05:30
Mayur Singal
77d98c9675
MINOR: Fix athena lineage models (#17662) 2024-09-02 14:19:13 +05:30
Pere Miquel Brull
bcb29b46da
MINOR - Implement SAP Hana Lineage (#17615)
* MINOR - SAP Hana Lineage

* skeleton

* parser

* lineage

* manage formulas

* add cvs

* add cvs

* better typing

* enum

* handle cvs

* saphana docs
2024-08-30 07:42:43 +02:00
Ayush Shah
9245c8965f
fix: Bump Ingestion and Dockerfile versions (#17624) 2024-08-29 12:03:30 +05:30