3385 Commits

Author SHA1 Message Date
harshsoni2024
1cd97fdac8
get primitive datatype in case of array, struct (#18102) 2024-10-04 18:49:50 +05:30
Suman Maharana
d9b15ef6cb
Minor: Resolve protected namespace warning (#18109) 2024-10-04 13:01:41 +05:30
Akash Jain
2436337969
fix: PostgreSQL installation in Ingestion Docker (#18114) 2024-10-04 11:31:01 +05:30
Mayur Singal
f69db851b8
Fix #18070: Add support for glue database descriptions (#18094) 2024-10-03 21:53:39 -07:00
IceS2
a05eba91a5
MINOR: Fix Ingestion Dockerfile Compatibility by changing IF syntax (#18098)
* Fix Ingestion Dockerfile Compatibility by changing IF syntax

* Increased shell compatibility by using only one '='
2024-10-04 02:52:48 +05:30
Mayur Singal
088282b0c1
MINOR: MSTR URI Fix (#18095) 2024-10-03 17:10:23 +02:00
Onkar Ravgan
23c6f1a6c1
AlationSink conn improvements (#18091) 2024-10-03 16:20:35 +05:30
Suman Maharana
bc6f4824ea
Added DBT tests with versionless and fixed v7 parsing (#18028) 2024-09-27 19:53:27 +05:30
IceS2
c9d37f6afc
Fix missing cast to str (#18016) 2024-09-27 16:03:45 +02:00
harshsoni2024
9442cac1a5
allow powerbi datamodel children in col.lineage (#18030) 2024-09-27 19:28:59 +05:30
Teddy
8dc6b7d282
MINOR - Fix General Profiler Bugs (#17995)
* fix import issue

* fix: better handle None values in profiler processing

* fix: profiler errors

* chore: fix comment

* style: fix python linting failure

* fix: null byte error with the database client

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-27 14:36:30 +02:00
sam-mccarty-mavenclinic
0dd3e97170
Fix 17911: Looker parsing improvements for liquid templating and view/model aliasing (#17912)
* Looker parsing improvements for liquid templating and view/model aliasing

* add python-liquid dependency to looker plugin requirements

* move to static method with 'openmetadata' context and add rendering tests

* remove backtick stripping

---------

Co-authored-by: Imri Paran <imri.paran@gmail.com>
2024-09-27 13:55:15 +02:00
Mayur Singal
e373fddbde
MINOR: MSTR connector import fix (#18025) 2024-09-27 16:57:09 +05:30
Pere Miquel Brull
d26449576a
GEN-1234 - Clean up suggestions when a user is deleted (#17988)
* GEN-1234 - Clean up suggestions when a user is deleted

* add method

* add method

* fix postgres query
2024-09-26 16:22:36 +02:00
Imri Paran
25284e0232
MINOR: fix snowflake system metrics (#17989)
* fix snowflake system metrics

* format

* add link to logs and commit
fixed the dq cli test

* reverted bad formatting

* fixed models.py

* removed version pinning for data diff in tests
2024-09-26 11:55:17 +00:00
Suman Maharana
37b6dc8290
Add Sigma Dashboard Connector (#17855)
* Add Sigma Dashboard Connector

* changed to id instead of name in dashboard entity

* Address Comments

* addressed comments

* Added Docs

* yaml file changes

* fix ui changes
2024-09-26 16:29:35 +05:30
Suman Maharana
d2bce8adf4
Minor: fix lineage & queries in dbt (#17999) 2024-09-26 10:50:24 +05:30
IceS2
d36f01abf6
Fix tearDown by using the proper file loader (#17994) 2024-09-25 17:37:56 +02:00
Suman Maharana
b3bfa6433e
Fix Manifest is not parsed correctly on dbt versionless (#17975) 2024-09-24 23:39:17 +05:30
Imri Paran
64f8571a77
pin testcontainers by minor version (#17938) 2024-09-24 09:18:36 +02:00
Aldwyn Cabarrubias
b9f881f2fa
Change time format for the created_at of the DBT cloud pipeline status (#17916)
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
2024-09-23 14:50:48 +05:30
Teddy
b222de66fe
Minor fix pylint version (#17944)
* fix import issue

* fix: fix pylint version

* style:ran python linting

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-20 18:50:28 +02:00
Pere Miquel Brull
4cccaae446
GEN-996 - Allow PII Processor without storing Sample Data (#17927)
* GEN-996 - Allow PII Processor without storing Sample Data

* fix import

* fix import
2024-09-20 16:05:29 +02:00
Ayush Shah
8087f48e15
Fixes #17869: Add validations in place for salesforce connection (#17870) 2024-09-20 16:33:38 +05:30
Pere Miquel Brull
1e56c76c0e
FIX #17896 - Python lineage SDK to work with Uuid & FQN models (#17928) 2024-09-20 10:37:41 +02:00
Imri Paran
d09bca26f6
MINOR: fix mssql integration test (#17923)
* change tag for sql server due to https://github.com/microsoft/mssql-docker/issues/441 (or some similar issue)

* use 2022-latest

* fixed mssql tests

* format

* used new columns

* use the custom sql server
2024-09-20 08:52:40 +02:00
Imri Paran
21af02d8d5
Fix 17903: fix(data-quality): snowflake data diff (#17907)
* fix(data-quality): snowflake data diff

- fixed schema in snowflake URL for data diff
- added e2e for snowflake data quality

* reverted unintended change
2024-09-19 17:55:48 +02:00
Mayur Singal
1be238266a
MINOR: Apply multi threading in View Lineage Processing (#17922) 2024-09-19 15:06:38 +02:00
Imri Paran
760b8eb742
fix(ingestion): deltalake (#17910)
add pin for deltalake<0.20 to avoid ad35eda798 (diff-f81b8b91e9721dc0a3235b6674a57c0700153b92b59f3988d357950a9c8c4760R1142-R1144)
2024-09-19 08:25:19 +02:00
Teddy
7df05c3e57
Minor python test failure (#17904)
* fix import issue

* chore: add raise from status on ingestion step in tests

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-09-18 17:41:05 +02:00
Teddy
33c50efdbf
GEN-1192 - Move Test Case to its Own Resource (#17862)
* feat: indexed test case results

* feat: added indexation logic for test case results

* style: ran java linting

* fix: IDE warnigns

* chore: added test case results migration

* style: ran java linting

* fix: postgres migration column json ref

* empty commit to trigger queued

* chore: extracted test case results to its own resource

* chore: fix failing tests

* chore: move testCaseResult state from testSuite and testCase to dynamic field fetched from test case results search index

* chore: clean up test case repository

* style: ran java linting

* chore: removed testCaseResultSummary and testCaseResult state from db

* fix: test failures

* chore: fix index mapping type for result value

* chore: fix test failure
2024-09-18 11:58:59 +02:00
Pere Miquel Brull
f6fdd86d5c
Docs - Ingestion License (#17893) 2024-09-17 08:58:53 -07:00
Pere Miquel Brull
6a1cd0ef8b
GEN-1493 - Fix paginate_es in opensearch (#17858)
* GEN-1493 - Fix opensearch pagination

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch
2024-09-17 14:21:10 +02:00
Pere Miquel Brull
91ddfcf07a
FIX - Postgres Python CI (#17859)
* fix deprecation

* remove print

* tests use testcontainers for mysql

* testcontainers for 3.8 dont work
2024-09-17 11:57:32 +02:00
Onkar Ravgan
507efc647f
fixed alationsink test (#17883) 2024-09-17 10:36:33 +02:00
Ayush Shah
23c6aa3ca5
fix: add children to array of struct (#17850) 2024-09-16 18:02:17 +05:30
Pere Miquel Brull
9a96ddd4ae
GEN-1494 - Add CI for py-tests with Postgres and Opensearch (#17853)
* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* GEN-1410 - Improve PII Logging information

* GEN-1494 - Add CI for py-tests with Postgres and Opensearch

* allow_module_level=True
2024-09-16 09:35:38 +02:00
Ayush Shah
2feb7d04ca
MINOR: Add Matillion SSL + Docs implementation (#17792) 2024-09-16 12:57:12 +05:30
Sriharsha Chintalapani
c34545adeb
Issue-15768: Support Metric Entity (#17680)
* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Issue-15768: Support Metric Entity

* Fix tests

* Fix tests

* Fix tests

* Minor: Fix tests

* ui: add metricsAPI rest utils

* ui: metric list page part 1

* feat: Add metric translations for multiple languages

* chore: Add "metric" field to SearchIndexingApplication schema

* ui: add create metric page

* ui: metric details page patch 1

* ui: add custom property and lineage support for metric entity

* ui: add expression component

* ui: add metric summary component

* chore: Update tab labels in MetricDetails and MetricVersion components

* ui: show other info like metric type, granularity, etc

* feat: Add support for metric entity in search dropdown

* feat: Rename custom property to Metric in MetricEntity.md

* feat: Add OwnerLabel component to MetricListPage

* Fix expression field in Metric

* chore: update expression to metricExpression

* ui: add metric header component with edit option

* Add metric to SearchIndexApp

* chore: Update expression to metricExpression

* ui: allow metric expression edit

* ui: update metric icon

* minor improvements

* Fix lineage indexing for Metric

* Update GlobalSettingsClassBase.ts to use MetricIcon for metrics in the global settings menu

* Fix error handling in MetricListPage component

* add related metrics

* minor improvements

* Fix relatedTerms patch

* Fix relatedTerms validation

* Add Boolean for deleted

* filter active entity from related metric list

* playwrite e2e part 1

* Refactor MetricSummary component to include RelatedMetrics in the summary panel

* test: add playwright test for metric special cases

* Add 'Metrics' to Explore Tree

* test: add e2e for add metric page

* test: add test for metric listing page content

* Add Boolean for deleted, remove deleted from suggests

* Refactor LineageProvider to handle deleted flag properly

* add playwright for metric listing

* fix test

* Add colored metric icon and update its usage in GlobalSettingsClassBase

* Fixed py_test test_ometa_endpoint for metric

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: SumanMaharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-09-16 11:36:31 +05:30
Pere Miquel Brull
e820cb0450
GEN-1410 - Improve PII Logging information (#17835) 2024-09-16 07:05:30 +02:00
k.nakagaki
3d8e30142c
Fixes 8428: make it possible to choice a sampling method type when we create profile ingestion for the Snowflake (#17831)
* Add test for existing code

* Add sampling method at ingestion.

* add samplingMethodType into UI

* modify init method to use new parameter.

* create descriptions

* execute isort

* fix an unintended change.

* apply py_format

* close section

* specify  init arguments

* fix bug

* apply py_format

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-09-15 21:51:17 +02:00
Mayur Singal
1fdb667ee3
MINOR: make type optional in ES Response (#17832)
* MINOR: make type optional in ES Mixin

* make none as default

* pyformat
2024-09-13 10:17:12 +02:00
Mayur Singal
f2c75be78d
MINOR: Optimise Snowflake Test Connection (#17779) 2024-09-13 12:55:55 +05:30
Imri Paran
59854de58d
[GEN-1406]: import plugin and clear plugin message for failed source import (#17788)
* support side effects on source classes by always importing source class

* streamlined error message

* fixed service type extraction for test suite pipeline

* - replaced "custom" with constant
- added quotes for the plugin exception for copy/paste ergonomics
2024-09-12 16:13:03 +02:00
Imri Paran
3282b057d8
ci: pin spacy version (#17814)
spacy 3.8 requires numpy 2.0 which is not compatible with openmetadata-ingestion requirements:
184e508d9c
2024-09-12 15:12:53 +05:30
Imri Paran
b70b3ce913
added config logging with secrets redacted (#17770) 2024-09-12 10:19:53 +02:00
Pere Miquel Brull
f4d0eba08f
MINOR - Add ES pagination with error handling (#17776)
* MINOR - Add ES pagination with error handling

* format

* format

* add nullable

* prepare API

* fix pagination

* format
2024-09-12 07:14:56 +02:00
Imri Paran
a3d6c1dd20
MINOR: tests(datalake): use minio (#17805)
* tests(datalake): use minio

1. use minio instead of moto for mimicking s3 behavior.
2. removed moto dependency as it is not compatible with aiobotocore (https://github.com/getmoto/moto/issues/7070#issuecomment-1828484982)

* - moved test_datalake_profiler_e2e.py to datalake/test_profiler
- use minio instead of moto

* fixed tests

* fixed tests

* removed default name for minio container
2024-09-12 07:13:01 +02:00
harshsoni2024
e7cfa4889c
MINOR: profiler e2e fix (#17809) 2024-09-11 22:04:22 +05:30
Mayur Singal
454467620c
MINOR: Fix Redshift View Def regex_replace Error (#17810) 2024-09-11 21:02:14 +05:30