2362 Commits

Author SHA1 Message Date
Onkar Ravgan
d9d3f6895b
Fix 6874: Added Support for lineage from dbt ephemeral nodes (#12101)
* fixed dbt ephemeral nodes lin

* fixed dbt tests
2023-06-23 10:01:22 +02:00
Mayur Singal
258a2e33e2
Fix #12078: Add clientSessionKeepAlive Field for snowflake (#12107) 2023-06-23 10:00:53 +02:00
Pere Miquel Brull
b5c4d5a9d3
Fix #10175 - Add Oracle lineage and usage (#12090)
* Add Oracle lineage and usage

* Add docs

* Add examples
2023-06-23 06:52:34 +02:00
Teddy
6f33465b7c
fix: move service retrival to workflow (#12109) 2023-06-23 06:40:32 +02:00
Ayush Shah
cb6e42941a
Fix 12025: Clickhouse NaN issue (#12079) 2023-06-22 12:51:56 +05:30
Onkar Ravgan
5197682921
Fixed dagster bugs and Added Pydantic Models (#12048) 2023-06-22 10:59:09 +05:30
Onkar Ravgan
78d7faf04a
Added Azure Blob Storage to dbt config (#11839) 2023-06-22 10:58:38 +05:30
Pere Miquel Brull
3b8e5e69fd
Fix #12056 - Clean verbose debug logs (#12060)
* Clean verbose debug logs

* Format

* Remove unnecessary test

* remove verbosity
2023-06-22 06:54:09 +02:00
Teddy
1e86b6533c
Fixes #11743 - Remove SQLParse dependency for System Metrics (#12072)
* fix: removed sqlparse dependency for system metrics

* fix: update sample query

* fix: move system test os retrieval to `.get()`

* fix: move os.environ to `get`
2023-06-22 06:51:24 +02:00
Onkar Ravgan
8fac9d8976
Added Powerbi Lineage across workspaces and datamodel projects (#12052)
* powerbi lin multiple workspace

* Addressed review comments
2023-06-21 17:16:09 +02:00
Onkar Ravgan
0cfd8d29c5
Added dbt test_metadata check (#12067) 2023-06-21 12:17:26 +02:00
Ayush Shah
9fa7b78d87
Issue 11686: Add IncludeOwner + scheduleInterval in Airflow (#12005) 2023-06-21 14:18:45 +05:30
Onkar Ravgan
47514f6c5f
Fixed dbt test col name (#12065) 2023-06-21 10:47:54 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Pere Miquel Brull
8751427679
Add project to Data Models JSON Schema & Implement in Looker (#12030)
* Add project to dashboards and models

* Add project to dashboards and models

* Update openmetadata-spec/src/main/resources/json/schema/entity/data/dashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Update openmetadata-spec/src/main/resources/json/schema/api/data/createDashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Fix tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-06-20 11:45:29 +02:00
Mayur Singal
2738310887
Fix #12012: Fix clickhouse usage schema_name error (#12032) 2023-06-20 11:41:28 +05:30
Ayush Shah
83e9b6c310
Fixes 10395: Validation of yaml workflow configs (#11985) 2023-06-20 11:20:59 +05:30
Onkar Ravgan
f07c421264
Removed Empty Description Assignment to entities and added database name logic to tableau (#12031) 2023-06-19 19:19:42 +05:30
Teddy
76f5d3d571
Fixes #11994 - Update dbt and GE integration with new DQ flow (#12018)
* feat: updated GE integration to match new test workflow

* feat: updated unit tests to match new signature

* feat: added GE integration tests

* feat: ran python linting

* feat: updated dbt ingestion to match new TestSuite workflow

* feat: ran python linting

* feat: remove testSuite from Elasticsearch event test case update

* feat: ran java linting
2023-06-19 15:05:51 +02:00
Pere Miquel Brull
7d39584bc9
Fix Airflow callback import (#12008) 2023-06-19 11:39:23 +02:00
Onkar Ravgan
d6d721225d
Fixed mark deleted all dbs (#11996) 2023-06-19 12:59:13 +05:30
Mayur Singal
8ecd88fd1b
Fix #11312: Deprecate ES Reindex on Ingestion and use reindex APIs instead (#11988) 2023-06-19 12:45:18 +05:30
Teddy
bc76515a6b
fix: release connection from engine pool (#12006) 2023-06-16 16:49:55 +02:00
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Teddy
3f01ee938f
Fixes #11895 - Add Indexing and Search logic for TestCases (#11989)
* feat: added logic to delete logical test suite + added check to prevent adding existing testCases to executable test suite

* feat: added elasticsearch index logic for testCases

* feat: added deletion logic from index logic when deleting test suites

* feat: added test case index search to  endpoint

* feat: add executable/logical filter in list testSuite + filterOut tables without tests in Table resource

* feat: added summary field to testSuite

* feat: added executionSummary endpoint for test cases

* feat: removed tick marks around timestamp

* feat: addressed test failures

* feat: ran python linting

* feat: add limit to fetch all tables in TableResource testSuite test

* feat: fix conflict

* feat: ran java checkstyle

* feat: fixed mongo linting + disabled mongo failing tests

* feat: removed mongo test skip

* feat: removed unsued pytest import
2023-06-15 21:27:54 +02:00
Onkar Ravgan
d08c928801
Added project property to dashboards (#11986)
* Added projects to dashboards

* Added powerbi proj

* merge conflicts after source url

* fixed mongo pytest
2023-06-15 21:23:43 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities (#11991) 2023-06-15 14:44:48 +05:30
Mayur Singal
7fa963eec3
Fix #1076: Add mongodb support (#11943) 2023-06-15 11:14:22 +05:30
YAMADA Yutaka
63e8aa57ce
Fixes #11955: Modify URL to register with Airflow Lineage Backend (#11956)
* fix: include host_port in url

* fix: run make py_format
2023-06-14 14:06:01 +02:00
Onkar Ravgan
d409c339e4
Fix 10689: Added Powerbi Datamodels (#11923) 2023-06-14 17:00:08 +05:30
Onkar Ravgan
81ec95ab27
dbt test value fixed (#11948) 2023-06-13 10:54:32 +05:30
Pere Miquel Brull
095f15064c
Fix #11752 - Multiple inlets in AirflowLineageOperator (#11945)
* Fix multiple inlets in AirflowLineageOperator

* Update dag file
2023-06-12 07:01:19 +02:00
Mayur Singal
05dc42bdb8
Fix #11808: Handle lineage for single db sources in superset (#11933) 2023-06-09 12:43:06 +05:30
Onkar Ravgan
caabe89f9c
Centralize tags ingestion logic (#11880) 2023-06-09 10:45:53 +05:30
Onkar Ravgan
66aa28a3ee
Added check for parent nodes (#11911) 2023-06-08 10:50:11 +05:30
Teddy
8dfa640a82
fix: check for empty description strings (#11914) 2023-06-07 12:49:29 +02:00
Mayur Singal
9e6e00caf9
Fix #9282: Support Databricks Unity Catalog (#11896) 2023-06-07 11:50:31 +05:30
Sriharsha Chintalapani
2b2602b76b
Fix column resize to limit to 128 chars (#11889)
* Fix column resize to limit to 128 chars

* Fix pyformat issue

* replace column_name with col_name

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-07 07:24:08 +02:00
Teddy
b0bfbad9da
fix: disable sys. metrics for snowflake (#11900) 2023-06-06 20:58:12 +00:00
Pere Miquel Brull
b68d063694
Update Test Connection Definition FQN (#11861)
* Update Test Connection Definition FQN

* chore: use object literal for string concatenation

* Fix test

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Fix test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-06 15:22:32 +02:00
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ (#11893)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting

* feat: fixed  update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results (#11881)" (#11892)
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results (#11881)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting
2023-06-06 09:45:49 +02:00
Ayush Shah
236141d9df
Fixes 11825 - Add dataset description on schema level - bigquery (#11878) 2023-06-06 07:28:47 +00:00
Ayush Shah
65f370e4aa
Rename GCS to GCP (#11812) 2023-06-06 11:57:00 +05:30
Onkar Ravgan
e21f38c5fd
handled null upstreamcol values (#11835) 2023-06-02 16:55:28 +05:30
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL (#11847)
* feat: Add threshold and strategy logic on the custom SQL object test

* feat: ran python linting

* feat: added safety checks for custom sql query

* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
c98a15ca19
Fixes #11705 - Update ingestion and backend to match new DQ flow (#11836)
* feat: refactor ingestion flow logic

* feat: ran python linting

* feat: update tests to match new workflow

* feat: ran python linting

* feat: update sample data test suite name

* feat: Added backend logic to support logical and executable test suites

* feat: clean up java and json code

* feat: added sample data for logical and executable test suites

* feat: remove executable from CreateTestSuite

* feat: ran python and java linting

* feat: added README info for data quality structure

* skipping cypress to keep main green

* fixed typescript type issue

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-06-01 23:19:13 -07:00
Pere Miquel Brull
fdeea71671
Fix Looker explore git link & Add BitBucket reader (#11837)
* Add looker test connection step

* Add looker test connection step

* Update Credentials

* Fix explore link and add bitbucket reader

* Format

* Fix test

* Fix spline linting

* Fix import
2023-06-02 07:19:32 +02:00
Mayur Singal
b7a1066f5b
Fix #11781: Fix atlas lineage issue (#11832) 2023-06-01 12:02:43 -07:00