2916 Commits

Author SHA1 Message Date
Mayur Singal
cf9b1a55ec
MINOR: athena usage & lineage minor fix (#16156)
* MINOR: athena usage & lineage minor fix

* py format
2024-05-07 18:59:39 +02:00
Mayur Singal
5cae2137d3
MINOR: improve s3 test connection (#16158) 2024-05-07 18:59:00 +02:00
Onkar Ravgan
4a6849a05d
MINOR: Added custom property EntityReference support to python sdk (#16132)
* Added cust prop entityref to python sdk

* Added name and displayName fields to entityref
2024-05-07 17:35:39 +05:30
Antoine Balliet
2b7853581c
feat: compute dbt model lineage across databases (#16079)
* feat: compute dbt model lineage across databases

* fix lint

* try other way

* remove useless function

* Update metadata.py

* review

* chore: remove config.serviceName usage
2024-05-07 11:49:50 +05:30
Mayur Singal
e13a1af395
MINOR: S3 Add support for single bucket ingestion (#16096) 2024-05-07 11:33:57 +05:30
Pere Miquel Brull
1969906643
MINOR - Add missing ES mappings (#16137) 2024-05-06 16:46:40 +02:00
Mohit Yadav
0769d71ee7
Fixes Test Suite Reference in Table Schema (#16129)
* Fixes Test Suite Reference in Table Schema

* fix: fix test suite to interact with entity reference

---------

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2024-05-06 19:03:23 +05:30
Suman Maharana
488078da8a
Add DDL query ingest (#15860) 2024-05-06 18:03:50 +05:30
IceS2
795879d776
MINOR: Fix issue with SQLAlchemy types not being correctly mapped to OM Type on the profiler (#16122)
* Fix issue with SQLAlchemy types not being correctly mapped to OM Types on the profiler

* Fix checkstyle
2024-05-03 17:05:52 +02:00
Mayur Singal
215a274bd0
Fix #14130: Add support for azure auth in trino (#16117) 2024-05-03 15:17:25 +05:30
Onkar Ravgan
f1004e7960
Fix #16091: Fixed case issue in dbt (#16105)
* dbt col name case fix

* Update ingestion/src/metadata/ingestion/ometa/mixins/patch_mixin.py

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-05-03 14:29:33 +05:30
Pere Miquel Brull
a0db32c18a
MINOR - Docs and deprecations (#16118)
* MINOR - Docs and deprecations

* clean
2024-05-03 10:18:29 +02:00
Onkar Ravgan
3c083cdb68
updated pbi e2e counts (#16109) 2024-05-03 06:57:15 +02:00
Yuki Shiga
f2cfbd5982
Fixes #16075: Fix Incorrect Spark Property Name in DeltaLake Connection Configuration (#16097)
* fix typo (sparks.driver.extraClassPath -> spark.driver.extraClassPath)

* Fix incorrect Spark session username configuration in DeltaLake connection
2024-05-02 18:04:31 +02:00
Onkar Ravgan
87c8254c38
Fix #15454: Added protobuf parser complex schema support (#16071)
* Added protobuf parser complex schema support

* Added options keyword in proto testing
2024-04-30 17:59:27 +05:30
Pere Miquel Brull
78324bd17c
MINOR - Set pipeline status early (#16077) 2024-04-30 09:22:04 +02:00
Teddy
e1bda12b79
fix: cache constraint results (#16078) 2024-04-30 08:12:04 +02:00
Onkar Ravgan
c63d441aa1
Fix #12929: Added config to include/exclude undeployed pipelines while ingestion (#15797) 2024-04-30 10:54:06 +05:30
Pere Miquel Brull
a5e062cf30
MINOR - Metadata Actions - New Actions (#16037)
* MINOR - Metadata Actions - New Actions

* Add actions

* Add actions

* Add actions

* Add actions

* Add actions

* Add actions

* Add support for chart search in UI

* Refactor searchEntity function in AlertsUtil.tsx and getSearchIndexEntityTypeMapping function in SearchClassBase.ts

* add domain

* add domain

* format

* Update setSourceAsValue parameter in searchEntity function

* add missing fields in the advanced search field selection for all the assets

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2024-04-30 07:11:03 +02:00
Ayush Shah
6055955f4e
Fix DataLake Client (#16072) 2024-04-30 10:25:03 +05:30
Mayur Singal
eac0466b4d
Fix #16064: Add support for dynamic table type (#16069) 2024-04-29 18:56:52 +05:30
Onkar Ravgan
ceaa9d3e8a
Fix #15611 Parse PowerBI Dax files for lineage (#15975) 2024-04-29 14:55:06 +05:30
harshsoni2024
68e036418c
Fix #15719: Improve unit test to increase coverage. (#15905)
* issue-15719: unit test for superset db source

* issue-15719: use testcontainers for superset_api client test

* issue-15719: superset-api yield data changes

* fix failed test cases due to testcontainer version

* issue-15719: postgres container version fix

* issue-15719: setup & teardown with testcontainers

* issue-15719: remove more patch code
2024-04-29 08:00:39 +02:00
harshsoni2024
d98a8c5cf1
issue-15858: reduce es call in ingestion for foreign key processing (#15988) 2024-04-29 11:07:32 +05:30
Fredrik Möllerstrand
21d7a2ffba
MINOR: Fix error message in S3Source (#16043)
* Fix error message in S3Source

* Fix checkstyle

---------

Co-authored-by: Pablo Takara <pjt1991@gmail.com>
2024-04-26 11:50:38 +02:00
Imri Paran
4ac5912d4c
MINOR: added TestCase inspection query to backend and sample data (#16003)
* added TestCase inspection query to backend and sample data

* format

* format
2024-04-26 11:49:08 +02:00
Teddy
4ed87a4d08
Fix #15341 - Test Case reference as inherited field for Test Case Incident (#16027)
* fix: unique test computation to scalar_subquery

* fix: make test case reference an inherited field

* style: ran java linting

* fix: added test case resolution migration

* style: ran java linting
2024-04-25 17:31:11 +02:00
Pere Miquel Brull
4b4a7d70aa
MINOR - Add default to ingestion dependency sh (#16021)
* MINOR - Add default to ingestion dependency sh

* MINOR - Add default to ingestion dependency sh
2024-04-25 14:18:41 +02:00
Onkar Ravgan
828e9abc97
Added enum support in custom prop python sdk (#16026) 2024-04-25 14:46:55 +05:30
Mayur Singal
976e4b304f
MINOR: Ignore lineage edge not found logs (#16019) 2024-04-25 14:16:51 +05:30
Ayush Shah
18ba585d2f
Revert from DBA_ to ALL_ (#16030) 2024-04-25 13:02:47 +05:30
Ayush Shah
3621407642
Fixes #15732: Modify Reference for Tags to EntityName (#15938) 2024-04-25 11:53:46 +05:30
Ayush Shah
a15da7ec98
Issue #14812: Add support for empty string as missing count (#16017) 2024-04-25 09:45:26 +05:30
Ayush Shah
595117f8b4
Minor: Fixes #15776 Numeric out of value error Trino (#15995) 2024-04-24 13:06:08 +05:30
Ayush Shah
8c26d2748d
Fix Import error (#16011) 2024-04-24 13:05:55 +05:30
Imri Paran
91a98a82ad
delete ingestion/src/metadata/__init__.py (#16004) 2024-04-24 09:30:54 +02:00
RabochiyYan
c90f1c0010
issue-15607 (#16002)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-04-24 08:08:22 +02:00
Ayush Shah
cf914dea94
Fix Inspector (#15997) 2024-04-23 16:32:30 +05:30
Teddy
a4b856956d
fix: compute unique and count at the same time (#15996) 2024-04-23 12:38:49 +02:00
Ayush Shah
0963a111fe
Fixes #12127: Add Support for Complex types of Databricks & UnityCatalog in profiler (#15976) 2024-04-23 15:54:36 +05:30
Pere Miquel Brull
df5d5e1866
MINOR - Fix datamodel lineage call (#15991)
* MINOR - Fix datamodel lineage call

* amend merge
2024-04-23 09:56:24 +02:00
Mayur Singal
85b6983eee
Fix #15062 & #14810: Fix Column level lineage overwrites pipeline Lineage & manual col lineage (#15897) 2024-04-23 09:37:43 +05:30
Teddy
449a5f2de3
FIX #11951 - ingestion logic for global profiler config (#15948)
* feat: add global metric configuration for the profiler

* style: ran java linting

* fix: renamed disable to disabled

* style: ran java linting

* feat: ometa sdk for profiler setting

* test: ingestion profiler global config tests

* fix: update metric name to use MetricType Enum

* fix: allow bot to retrieve settings

* fix: exclude GX artifacts

* feat: implement global profiler setting logic for ingestion side

* fix: exclude metrics if Metric is empty

* style: ran python linting

* style: ran python linting

* fix: skip empty metrics

* style: ran python linting

* fix: moved GET profiler config to seperate endpoint in system resource

* fix: moved compute metric filter to MetricFilter + renamed container

* fix: test failures

* fix: profiler test case
2024-04-22 22:35:37 +02:00
Imri Paran
93ec391f5c
MINOR: Dynamodb sample data (#15264)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* feat(profiler): add dynamodb row count

* feat(profiler): add dynamodb row count

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format

* feat(nosql-profiler): added sample data

1. Implemented the NoSQL sampler.
2. Some naming changes to the NoSQL adaptor to avoid fixing names with the profiler interface.
3. Tests.

* added default sample limit

* formatting

* fixed import

* feat(profiler): dynamodb sample data

* tests for dynamo db sample data

* format

* format

* use service connection for nosql adaptor factory

* fixed tests

* format

* fixed after merge
2024-04-22 17:46:40 +02:00
IceS2
cb801dedb4
FIXES 13209: Add Sagemaker Model Storage (#15986)
* Add Sagemaker Model Storage

* Fix checkstyle

* Sagemaker unittest

* Small refactor to be less verbose
2024-04-22 16:53:25 +02:00
IceS2
08c114c340
FIXES 15626: Fix issue with not url model store (#15974)
* Changed the MLModelStore storage type to string

* fix checkstyle

* remove unused files

* Update requirements

* fix checkstyle

* Skipping MLFlow intergration on python 3.8

* Hack to allow pytest to parse the mlflow integrations test on python 3.8

* Fix checkstyle
2024-04-22 15:50:44 +02:00
Mayur Singal
c04ec3e922
Fix #14973: Qliksense - Add option to disable hostname validation (#15954) 2024-04-22 18:49:59 +05:30
IceS2
19fa15f010
fix e2e (#15981) 2024-04-22 09:57:06 +02:00
Imri Paran
0a1018648c
Fixes #15566: add dynamodb row count (#15204)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* feat(profiler): add dynamodb row count

* feat(profiler): add dynamodb row count

* formatting

* fixed import

* format

* dded dynamodb row count

* format

* removed unused factory file

* removed "validate"

* migrations

* removed validations

* format

* linting

* fixed: test_amundsen.py

* Update schemaChanges.sql
2024-04-22 09:14:52 +02:00
Imri Paran
d8781bbef2
MINOR: postgres integration test (#15929)
* implemented postgres-integration-tests

* format

* format

* - disable ryuk
- disabled verbose sqlfluff logging

* query usage assertion
2024-04-19 10:00:37 -07:00