2958 Commits

Author SHA1 Message Date
Onkar Ravgan
38f3863002
Added limits to dagster graphql queries (#16202) 2024-05-10 09:31:49 +02:00
Mayur Singal
96668c881d
MINOR: Fix missing attribute get_table_owner (#16214) 2024-05-10 06:46:41 +02:00
harshsoni2024
f5a176c5f2
issue-9281: column lineage for powerbi (#16135)
* issue-9281: col. lineage for powerbi

* issue-9281: error handling for columns fqn
2024-05-09 19:15:26 +05:30
Ayush Shah
88f6ccd000
Fix Hive Profiler Interface (#16193) 2024-05-09 16:56:48 +05:30
Huanjie Guo
1a6f009757
Fixes 16185: try - catch the usage parsing error (#16188)
* try catch the exception in parsing usage

* Update usage-workflow-query-logs.md

* Update usage-workflow-query-logs.md

* Update usage-workflow-query-logs.md

* Update usage-workflow-query-logs.md
2024-05-09 12:19:07 +02:00
Fredrik Möllerstrand
5ce81b36ac
MINOR: Remove duplicated code in S3Source (#16153) 2024-05-08 12:00:37 +02:00
Pere Miquel Brull
39eed12f32
MINOR - Version match logic update & Airflow docs (#16157)
* airflow docs

* update version validation

* MINOR - docs and version match
2024-05-08 07:37:14 +02:00
Mayur Singal
cf9b1a55ec
MINOR: athena usage & lineage minor fix (#16156)
* MINOR: athena usage & lineage minor fix

* py format
2024-05-07 18:59:39 +02:00
Mayur Singal
5cae2137d3
MINOR: improve s3 test connection (#16158) 2024-05-07 18:59:00 +02:00
Onkar Ravgan
4a6849a05d
MINOR: Added custom property EntityReference support to python sdk (#16132)
* Added cust prop entityref to python sdk

* Added name and displayName fields to entityref
2024-05-07 17:35:39 +05:30
Antoine Balliet
2b7853581c
feat: compute dbt model lineage across databases (#16079)
* feat: compute dbt model lineage across databases

* fix lint

* try other way

* remove useless function

* Update metadata.py

* review

* chore: remove config.serviceName usage
2024-05-07 11:49:50 +05:30
Mayur Singal
e13a1af395
MINOR: S3 Add support for single bucket ingestion (#16096) 2024-05-07 11:33:57 +05:30
Pere Miquel Brull
1969906643
MINOR - Add missing ES mappings (#16137) 2024-05-06 16:46:40 +02:00
Mohit Yadav
0769d71ee7
Fixes Test Suite Reference in Table Schema (#16129)
* Fixes Test Suite Reference in Table Schema

* fix: fix test suite to interact with entity reference

---------

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2024-05-06 19:03:23 +05:30
Suman Maharana
488078da8a
Add DDL query ingest (#15860) 2024-05-06 18:03:50 +05:30
IceS2
795879d776
MINOR: Fix issue with SQLAlchemy types not being correctly mapped to OM Type on the profiler (#16122)
* Fix issue with SQLAlchemy types not being correctly mapped to OM Types on the profiler

* Fix checkstyle
2024-05-03 17:05:52 +02:00
Mayur Singal
215a274bd0
Fix #14130: Add support for azure auth in trino (#16117) 2024-05-03 15:17:25 +05:30
Onkar Ravgan
f1004e7960
Fix #16091: Fixed case issue in dbt (#16105)
* dbt col name case fix

* Update ingestion/src/metadata/ingestion/ometa/mixins/patch_mixin.py

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2024-05-03 14:29:33 +05:30
Pere Miquel Brull
a0db32c18a
MINOR - Docs and deprecations (#16118)
* MINOR - Docs and deprecations

* clean
2024-05-03 10:18:29 +02:00
Yuki Shiga
f2cfbd5982
Fixes #16075: Fix Incorrect Spark Property Name in DeltaLake Connection Configuration (#16097)
* fix typo (sparks.driver.extraClassPath -> spark.driver.extraClassPath)

* Fix incorrect Spark session username configuration in DeltaLake connection
2024-05-02 18:04:31 +02:00
Onkar Ravgan
87c8254c38
Fix #15454: Added protobuf parser complex schema support (#16071)
* Added protobuf parser complex schema support

* Added options keyword in proto testing
2024-04-30 17:59:27 +05:30
Pere Miquel Brull
78324bd17c
MINOR - Set pipeline status early (#16077) 2024-04-30 09:22:04 +02:00
Teddy
e1bda12b79
fix: cache constraint results (#16078) 2024-04-30 08:12:04 +02:00
Onkar Ravgan
c63d441aa1
Fix #12929: Added config to include/exclude undeployed pipelines while ingestion (#15797) 2024-04-30 10:54:06 +05:30
Pere Miquel Brull
a5e062cf30
MINOR - Metadata Actions - New Actions (#16037)
* MINOR - Metadata Actions - New Actions

* Add actions

* Add actions

* Add actions

* Add actions

* Add actions

* Add actions

* Add support for chart search in UI

* Refactor searchEntity function in AlertsUtil.tsx and getSearchIndexEntityTypeMapping function in SearchClassBase.ts

* add domain

* add domain

* format

* Update setSourceAsValue parameter in searchEntity function

* add missing fields in the advanced search field selection for all the assets

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2024-04-30 07:11:03 +02:00
Ayush Shah
6055955f4e
Fix DataLake Client (#16072) 2024-04-30 10:25:03 +05:30
Mayur Singal
eac0466b4d
Fix #16064: Add support for dynamic table type (#16069) 2024-04-29 18:56:52 +05:30
Onkar Ravgan
ceaa9d3e8a
Fix #15611 Parse PowerBI Dax files for lineage (#15975) 2024-04-29 14:55:06 +05:30
harshsoni2024
d98a8c5cf1
issue-15858: reduce es call in ingestion for foreign key processing (#15988) 2024-04-29 11:07:32 +05:30
Fredrik Möllerstrand
21d7a2ffba
MINOR: Fix error message in S3Source (#16043)
* Fix error message in S3Source

* Fix checkstyle

---------

Co-authored-by: Pablo Takara <pjt1991@gmail.com>
2024-04-26 11:50:38 +02:00
Imri Paran
4ac5912d4c
MINOR: added TestCase inspection query to backend and sample data (#16003)
* added TestCase inspection query to backend and sample data

* format

* format
2024-04-26 11:49:08 +02:00
Teddy
4ed87a4d08
Fix #15341 - Test Case reference as inherited field for Test Case Incident (#16027)
* fix: unique test computation to scalar_subquery

* fix: make test case reference an inherited field

* style: ran java linting

* fix: added test case resolution migration

* style: ran java linting
2024-04-25 17:31:11 +02:00
Onkar Ravgan
828e9abc97
Added enum support in custom prop python sdk (#16026) 2024-04-25 14:46:55 +05:30
Mayur Singal
976e4b304f
MINOR: Ignore lineage edge not found logs (#16019) 2024-04-25 14:16:51 +05:30
Ayush Shah
18ba585d2f
Revert from DBA_ to ALL_ (#16030) 2024-04-25 13:02:47 +05:30
Ayush Shah
a15da7ec98
Issue #14812: Add support for empty string as missing count (#16017) 2024-04-25 09:45:26 +05:30
Ayush Shah
595117f8b4
Minor: Fixes #15776 Numeric out of value error Trino (#15995) 2024-04-24 13:06:08 +05:30
Ayush Shah
8c26d2748d
Fix Import error (#16011) 2024-04-24 13:05:55 +05:30
Imri Paran
91a98a82ad
delete ingestion/src/metadata/__init__.py (#16004) 2024-04-24 09:30:54 +02:00
RabochiyYan
c90f1c0010
issue-15607 (#16002)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-04-24 08:08:22 +02:00
Ayush Shah
cf914dea94
Fix Inspector (#15997) 2024-04-23 16:32:30 +05:30
Teddy
a4b856956d
fix: compute unique and count at the same time (#15996) 2024-04-23 12:38:49 +02:00
Ayush Shah
0963a111fe
Fixes #12127: Add Support for Complex types of Databricks & UnityCatalog in profiler (#15976) 2024-04-23 15:54:36 +05:30
Pere Miquel Brull
df5d5e1866
MINOR - Fix datamodel lineage call (#15991)
* MINOR - Fix datamodel lineage call

* amend merge
2024-04-23 09:56:24 +02:00
Mayur Singal
85b6983eee
Fix #15062 & #14810: Fix Column level lineage overwrites pipeline Lineage & manual col lineage (#15897) 2024-04-23 09:37:43 +05:30
Teddy
449a5f2de3
FIX #11951 - ingestion logic for global profiler config (#15948)
* feat: add global metric configuration for the profiler

* style: ran java linting

* fix: renamed disable to disabled

* style: ran java linting

* feat: ometa sdk for profiler setting

* test: ingestion profiler global config tests

* fix: update metric name to use MetricType Enum

* fix: allow bot to retrieve settings

* fix: exclude GX artifacts

* feat: implement global profiler setting logic for ingestion side

* fix: exclude metrics if Metric is empty

* style: ran python linting

* style: ran python linting

* fix: skip empty metrics

* style: ran python linting

* fix: moved GET profiler config to seperate endpoint in system resource

* fix: moved compute metric filter to MetricFilter + renamed container

* fix: test failures

* fix: profiler test case
2024-04-22 22:35:37 +02:00
Imri Paran
93ec391f5c
MINOR: Dynamodb sample data (#15264)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* feat(profiler): add dynamodb row count

* feat(profiler): add dynamodb row count

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format

* feat(nosql-profiler): added sample data

1. Implemented the NoSQL sampler.
2. Some naming changes to the NoSQL adaptor to avoid fixing names with the profiler interface.
3. Tests.

* added default sample limit

* formatting

* fixed import

* feat(profiler): dynamodb sample data

* tests for dynamo db sample data

* format

* format

* use service connection for nosql adaptor factory

* fixed tests

* format

* fixed after merge
2024-04-22 17:46:40 +02:00
IceS2
cb801dedb4
FIXES 13209: Add Sagemaker Model Storage (#15986)
* Add Sagemaker Model Storage

* Fix checkstyle

* Sagemaker unittest

* Small refactor to be less verbose
2024-04-22 16:53:25 +02:00
Mayur Singal
c04ec3e922
Fix #14973: Qliksense - Add option to disable hostname validation (#15954) 2024-04-22 18:49:59 +05:30
Imri Paran
0a1018648c
Fixes #15566: add dynamodb row count (#15204)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* feat(profiler): add dynamodb row count

* feat(profiler): add dynamodb row count

* formatting

* fixed import

* format

* dded dynamodb row count

* format

* removed unused factory file

* removed "validate"

* migrations

* removed validations

* format

* linting

* fixed: test_amundsen.py

* Update schemaChanges.sql
2024-04-22 09:14:52 +02:00