1368 Commits

Author SHA1 Message Date
Shailesh Parmar
3bcef4f58c
UI: Added Data Quality test suite listing page and detail page (#7014)
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* added test suite listing page

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

* updated all API call for test suite ingestion page

* updated test suite view redirection

* updated ingestion infinite loader

* added edit action for test case

* added edit/add ingestion flow

* added after action for form

* remove duplicate close action

* fixed alignment issue

* added expand on row click functionality and change expand/collapse icon

* used antd component

* resolving conflict

* Fixed#7040: Test-suite count support in counts API (#7041)

* updated test suite count

* fixed failing unit test

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: Parth Panchal <83201188+parthp2107@users.noreply.github.com>
2022-08-30 19:00:51 +05:30
mohitdeuex
b7351c220a
Added Team Webhook (#6973)
* Added Team Webhook

* Ms Teams UI for setting Page

* Updated Review Comment Changes

* minor changes

* Fix webhook types

* Updated Parser for Teams

* Change MsTeams content and created Justify Card

* Fixed failing tests cases

Co-authored-by: Ashish gupta <ag939431@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-08-30 11:10:02 +05:30
Suresh Srinivas
1de569af61
Fixes #7013 Backend : Policy rule description is not getting updated (#7027)
* Fixing test failures

* Fixes #6978  Helper function for building ChangeDescription

* Fixes #7013 Backend : Policy rule description is not getting updated
2022-08-29 22:03:38 -07:00
Ashish Gupta
b59878e1b5
Fix (#6984) doc links in OpenMetaData (#6986)
* Fix doc links in OpenMetaData

* Link updated

* remove unwanted file

* changes as per comments

* Changes as per comments
2022-08-29 19:38:54 +05:30
Mayur Singal
58d27811d0
Bigquery Mark Paertitioned Tables (#7007) 2022-08-29 14:15:59 +02:00
Teddy
bcdb8891f8
Fixed datatype bug (#6987) 2022-08-27 15:56:08 +05:30
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
Suresh Srinivas
c2a70bd545
Fixes #6978 Helper function for building ChangeDescription (#6979)
* Fixing test failures

* Fixes #6978  Helper function for building ChangeDescription
2022-08-26 14:07:15 -07:00
Pere Miquel Brull
6cebf37b6c
Fix #6906 - Add Nifi connector (#6956)
* Add Nifi connector

* Mock token

* Format

* pass verify param
2022-08-26 21:50:01 +02:00
Onkar Ravgan
13d433b0d7
Add Owner for DBT (#6872)
* Added owner field in DataModel

* Added user in backend

* review comments addressed

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-26 16:22:27 +05:30
Suresh Srinivas
a8732c8e89
Fixes #6844 Ingestion causes Custom Property values to disappear (#6912) 2022-08-25 14:04:18 -07:00
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Vivek Ratnavel Subramanian
f23357ba68
Update slack chat to redirect to a URL (#6909) 2022-08-25 15:25:11 +05:30
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Pere Miquel Brull
667c837ae0
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
2022-08-24 16:27:31 +02:00
NiharDoshi99
f0f0476396
Added http & https scheme options in Hive connector (#6899)
* Added other scheme options in Hive connector

* Added other scheme options in Hive connector
2022-08-24 18:30:30 +05:30
Mayur Singal
404cc67911
WIP - Fix #6744: allow more than one metadata ingestion workflow (#6831)
* Fix #6744: allow more than one metadata ingestion workflow

* Rename to markDeletedTablesFromFilterOnly

* ui support for new field markDeletedTablesFromFilterOnly

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-08-24 14:21:52 +02:00
Ayush Shah
322f6445e1
Antlr entitylink split fix (#6901)
* Modified Antlr Split Listeners

* Modified EntityLink.g4

* Fix Grammar split
2022-08-24 14:05:21 +02:00
Suresh Srinivas
895a6e5551
Fixes #6860 Add policy support for policies based on resource ownership hierachy (#6861)
* WIP

* Fixes #6728 API to get detailed permissions for any user

* Fixes #6860 Add policy support for policies based on resource ownership hierachy

* Check if inheritedRoles is null

* Revert "Check if inheritedRoles is null"

This reverts commit 3dee922d5937c1f4ac9a298d1156e7adf15083a6.

* Initialize SubjectCache in NoOpAuthorizer

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2022-08-23 20:44:11 -07:00
Suresh Srinivas
605871db2e
Fixes #6851 Backend : Roles patch API is not working for policies (#6890) 2022-08-23 15:31:42 -07:00
Suresh Srinivas
ca39b9f603
Fixes #6886 Add API for validating expressions in Rule conditions (#6887) 2022-08-23 13:53:16 -07:00
Teddy
afd12cff04
added testResultValue to testCaseResult (#6881) 2022-08-23 18:32:15 +02:00
Nahuel
6db7eba16b
Fix table repository bug (#6871) 2022-08-23 15:13:11 +02:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean (#6866)
* Databricks Connection - User & Password clean

* Fix: Python checkstyle failed

* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00
Pere Miquel Brull
a975d5177d
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
2022-08-23 08:59:44 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832)
* Implementation of temp secret for testing connection

* Fix tests
2022-08-22 08:43:23 +02:00
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805)
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
2022-08-19 16:15:40 +02:00
Pere Miquel Brull
3f1fc5344a
Remove Athena hostPort (#6825) 2022-08-19 14:11:53 +02:00
Pere Miquel Brull
ed0a01edea
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
2022-08-19 11:19:20 +02:00
Pere Miquel Brull
6f3cd4cfed
Fix #6818 - Superset code cleanup and docs (#6821)
* code cleaning

* superset docs

* Fix metabase docs
2022-08-19 11:19:06 +02:00
Suresh Srinivas
7302c1658d
Upgrade dependencies for 0.12 release (#6815) 2022-08-18 16:13:04 -07:00
Suresh Srinivas
4b8af0a03a
Make WebhookType part of webhook.json and not the create request (#6812) 2022-08-18 12:28:38 -07:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Nahuel
89ec1f9c6d
AWS SSM secrets manager implementation on backend (#6771)
* SSM secrets manager implementation in backend

* Add test and refactor

* Use SecretString type for storing parameters
2022-08-18 11:06:14 +02:00
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-18 00:29:59 -07:00
Sriharsha Chintalapani
8943e3bb7d
Fix #5831: Add pipeline to testsuite (#6756)
* Fix #5831: Add pipeline to testsuite

* Fix #5831: Add pipeline to testsuite
2022-08-18 08:49:32 +02:00
Teddy
8e0f1274f2
Added timestamp with timezone data type (#6761) 2022-08-17 16:02:59 +02:00
Nahuel
00ce67c01e
Fix#6643: Secure in secrets manager the dbtConfigSource object (#6743)
* Secure in secrets manager the dbtConfigSource object

* Add missing tests

* Fix minor bug

* Minor code refactor
2022-08-17 10:32:43 +02:00
Onkar Ravgan
424239840f
Added result limits to usage queries (#6748)
* Added result limits to queries

* Fixed formatting

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-17 08:07:39 +02:00
Parth Panchal
28976583d5
Added SSL config for Kafka publisher (#6752)
* Added ssl config for kafka publisher

* Added ssl config for kafka publisher

* Added ssl config for kafka publisher
2022-08-17 11:28:55 +05:30
Nahuel
30e7312203
Fix#5920: Expose hostPort field for some services (#6678)
* Expose hostPort field for some services

* Fix failing tests after changes

* Remove added interface to json schemas

* Change implementation to use annotations for exposing fields of objects

* Update JsonUtils test

* Format Java code
2022-08-16 09:54:47 +02:00
Suresh Srinivas
b16a821f9c
Fixes #6728 API to get detailed permissions for any user (#6729)
* WIP

* Fixes #6728 API to get detailed permissions for any user
2022-08-15 13:40:49 -07:00
Vivek Ratnavel Subramanian
c968fc8912
Fix #6603 Backend: Add support for delete API for threads (#6655)
* Fix #6603 Backend: Add support for delete API for threads

* Fix compilation error
2022-08-14 16:10:31 -07:00
Suresh Srinivas
bd5ac5c67f
Fixes #6723 GET API to get a list of functions that can be used for authoring Policy Rule conditions (#6724)
* WIP

* Fixes #6723 GET API to get a list of functions that can be used for authoring Policy Rule conditions
2022-08-13 21:37:55 -07:00
Suresh Srinivas
9e4be2a48b
Fixes #6719 Validate rule conditions and return meaningful errors in API response (#6720) 2022-08-13 13:10:00 -07:00
Vivek Ratnavel Subramanian
d04a28e45b
Fix #6652 Backend: Sort Feed by created timestamp by default (#6653) 2022-08-13 11:41:38 -07:00
Abhishek Pandey
a6a7662517
dagster-connector-added (#6513)
* dagster-connector-added

* code-formatted

* dagster-comments-removed

* dagster-comments-removed

* dagster-unittest-added

* code-formatted-pr-comments-resolved

* improved-code-quality

* increased-py-test-time-out

* improved-dagster-tests

* improved-dagster-tests

* reverted-pytests-timeout

* dagster-pytest-improved

* added-schema-in-omdcore

* updated-schema-in-omd-core

* ui-schema-updated-omd-core

* updated-dagster-schema

* dagter-schema-test-updated
2022-08-13 15:18:14 +05:30
Nahuel
80ca224c39
Fix: Avoid store sensitive info in events repository using custom annotation (#6666)
* Avoid store sensitive infomation in created event when a Secrets Manager is configured

* Refactor code to start using custom annotations in the json schma deffinition

* Remove unused config object

* Add removed comment

* Add separated entity comparision for events in EntityResourceTest

* Add missing mask property in service connection schemas

* Update openmetadata-core pom.xml

* Missing import in openmetadata-core pom.xml
2022-08-12 15:56:37 -07:00
Parth Panchal
99453279c0
Sonar cloud findings fix (#6709)
* Sonar cloud findings fix

* Sonar cloud findings fix

* Sonar cloud findings fix

* Sonar cloud findings fix
2022-08-12 22:15:57 +05:30