1379 Commits

Author SHA1 Message Date
Pere Miquel Brull
613a329f40
Fix #7089 - Add Redpanda connector (#7092)
Fix #7089 - Add Redpanda connector (#7092)
2022-09-01 06:39:27 +02:00
Pere Miquel Brull
7d2527bbeb
Fix #6961 - Fallback to another IP service (#7019)
Fix #6961  - Fallback to another IP service (#7019)
2022-09-01 06:39:12 +02:00
Teddy
ef41382cb1
Fixes #7094 by fixing minior bugs in table tests (#7095) 2022-08-31 21:35:33 +02:00
Teddy
811f640a18
Fixes #6891 by implementing support for VARIANT DType (#7084) 2022-08-31 19:01:00 +02:00
Nahuel
0762e39172
Minor fixes related to Secrets Manager implementation (#7046)
* Minor fixes related to Secrets Manager implementation

* Fix failing test
2022-08-31 14:14:55 +02:00
Pere Miquel Brull
d791fe8289
Fix #6816 - Add supportsQueryComment & Fix #7078 - Athena Session Token (#7079)
* Athena with aws session token

* add supports query comment property to the required db services

* Hide query comment toggle
2022-08-31 14:03:54 +02:00
Suresh Srinivas
cffecc550a
Fixes #7072 - Remove life cycle policies (#7073)
* Fixes #7072 - Remove life cycle policies

* Remove policy sink

* Remove OMetaPolicy

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-08-31 12:47:03 +05:30
Mayur Singal
7caac4616f
Fix #6645: Remove required fields from mssql (#7065) 2022-08-31 06:52:37 +02:00
Suresh Srinivas
1dfcb45e86
Clean up lingering code related to defaultRole field in Role entity (#7064) 2022-08-30 11:25:22 -07:00
Damon-Zeng
7cd5579c84
add 'supportsDatabase' property in mssqlConnection.json (#7015)
* add 'supportsDatabase' property in mssqlConnection.json

* version uipdate with run 'make core_bump_version_dev'

Co-authored-by: Peng Zeng <pengzeng@PengdeMacBook-Pro.local>
2022-08-30 17:30:15 +02:00
Pere Miquel Brull
9841b0887f
Remove required fields from BQ for ADC (#7037) 2022-08-30 17:25:02 +02:00
Shailesh Parmar
3bcef4f58c
UI: Added Data Quality test suite listing page and detail page (#7014)
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* added test suite listing page

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

* updated all API call for test suite ingestion page

* updated test suite view redirection

* updated ingestion infinite loader

* added edit action for test case

* added edit/add ingestion flow

* added after action for form

* remove duplicate close action

* fixed alignment issue

* added expand on row click functionality and change expand/collapse icon

* used antd component

* resolving conflict

* Fixed#7040: Test-suite count support in counts API (#7041)

* updated test suite count

* fixed failing unit test

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: Parth Panchal <83201188+parthp2107@users.noreply.github.com>
2022-08-30 19:00:51 +05:30
mohitdeuex
b7351c220a
Added Team Webhook (#6973)
* Added Team Webhook

* Ms Teams UI for setting Page

* Updated Review Comment Changes

* minor changes

* Fix webhook types

* Updated Parser for Teams

* Change MsTeams content and created Justify Card

* Fixed failing tests cases

Co-authored-by: Ashish gupta <ag939431@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-08-30 11:10:02 +05:30
Suresh Srinivas
1de569af61
Fixes #7013 Backend : Policy rule description is not getting updated (#7027)
* Fixing test failures

* Fixes #6978  Helper function for building ChangeDescription

* Fixes #7013 Backend : Policy rule description is not getting updated
2022-08-29 22:03:38 -07:00
Ashish Gupta
b59878e1b5
Fix (#6984) doc links in OpenMetaData (#6986)
* Fix doc links in OpenMetaData

* Link updated

* remove unwanted file

* changes as per comments

* Changes as per comments
2022-08-29 19:38:54 +05:30
Mayur Singal
58d27811d0
Bigquery Mark Paertitioned Tables (#7007) 2022-08-29 14:15:59 +02:00
Teddy
bcdb8891f8
Fixed datatype bug (#6987) 2022-08-27 15:56:08 +05:30
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
Suresh Srinivas
c2a70bd545
Fixes #6978 Helper function for building ChangeDescription (#6979)
* Fixing test failures

* Fixes #6978  Helper function for building ChangeDescription
2022-08-26 14:07:15 -07:00
Pere Miquel Brull
6cebf37b6c
Fix #6906 - Add Nifi connector (#6956)
* Add Nifi connector

* Mock token

* Format

* pass verify param
2022-08-26 21:50:01 +02:00
Onkar Ravgan
13d433b0d7
Add Owner for DBT (#6872)
* Added owner field in DataModel

* Added user in backend

* review comments addressed

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-26 16:22:27 +05:30
Suresh Srinivas
a8732c8e89
Fixes #6844 Ingestion causes Custom Property values to disappear (#6912) 2022-08-25 14:04:18 -07:00
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Vivek Ratnavel Subramanian
f23357ba68
Update slack chat to redirect to a URL (#6909) 2022-08-25 15:25:11 +05:30
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Pere Miquel Brull
667c837ae0
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
2022-08-24 16:27:31 +02:00
NiharDoshi99
f0f0476396
Added http & https scheme options in Hive connector (#6899)
* Added other scheme options in Hive connector

* Added other scheme options in Hive connector
2022-08-24 18:30:30 +05:30
Mayur Singal
404cc67911
WIP - Fix #6744: allow more than one metadata ingestion workflow (#6831)
* Fix #6744: allow more than one metadata ingestion workflow

* Rename to markDeletedTablesFromFilterOnly

* ui support for new field markDeletedTablesFromFilterOnly

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-08-24 14:21:52 +02:00
Ayush Shah
322f6445e1
Antlr entitylink split fix (#6901)
* Modified Antlr Split Listeners

* Modified EntityLink.g4

* Fix Grammar split
2022-08-24 14:05:21 +02:00
Suresh Srinivas
895a6e5551
Fixes #6860 Add policy support for policies based on resource ownership hierachy (#6861)
* WIP

* Fixes #6728 API to get detailed permissions for any user

* Fixes #6860 Add policy support for policies based on resource ownership hierachy

* Check if inheritedRoles is null

* Revert "Check if inheritedRoles is null"

This reverts commit 3dee922d5937c1f4ac9a298d1156e7adf15083a6.

* Initialize SubjectCache in NoOpAuthorizer

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2022-08-23 20:44:11 -07:00
Suresh Srinivas
605871db2e
Fixes #6851 Backend : Roles patch API is not working for policies (#6890) 2022-08-23 15:31:42 -07:00
Suresh Srinivas
ca39b9f603
Fixes #6886 Add API for validating expressions in Rule conditions (#6887) 2022-08-23 13:53:16 -07:00
Teddy
afd12cff04
added testResultValue to testCaseResult (#6881) 2022-08-23 18:32:15 +02:00
Nahuel
6db7eba16b
Fix table repository bug (#6871) 2022-08-23 15:13:11 +02:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean (#6866)
* Databricks Connection - User & Password clean

* Fix: Python checkstyle failed

* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00
Pere Miquel Brull
a975d5177d
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
2022-08-23 08:59:44 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832)
* Implementation of temp secret for testing connection

* Fix tests
2022-08-22 08:43:23 +02:00
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805)
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
2022-08-19 16:15:40 +02:00
Pere Miquel Brull
3f1fc5344a
Remove Athena hostPort (#6825) 2022-08-19 14:11:53 +02:00
Pere Miquel Brull
ed0a01edea
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
2022-08-19 11:19:20 +02:00
Pere Miquel Brull
6f3cd4cfed
Fix #6818 - Superset code cleanup and docs (#6821)
* code cleaning

* superset docs

* Fix metabase docs
2022-08-19 11:19:06 +02:00
Suresh Srinivas
7302c1658d
Upgrade dependencies for 0.12 release (#6815) 2022-08-18 16:13:04 -07:00
Suresh Srinivas
4b8af0a03a
Make WebhookType part of webhook.json and not the create request (#6812) 2022-08-18 12:28:38 -07:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Nahuel
89ec1f9c6d
AWS SSM secrets manager implementation on backend (#6771)
* SSM secrets manager implementation in backend

* Add test and refactor

* Use SecretString type for storing parameters
2022-08-18 11:06:14 +02:00
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-18 00:29:59 -07:00
Sriharsha Chintalapani
8943e3bb7d
Fix #5831: Add pipeline to testsuite (#6756)
* Fix #5831: Add pipeline to testsuite

* Fix #5831: Add pipeline to testsuite
2022-08-18 08:49:32 +02:00
Teddy
8e0f1274f2
Added timestamp with timezone data type (#6761) 2022-08-17 16:02:59 +02:00
Nahuel
00ce67c01e
Fix#6643: Secure in secrets manager the dbtConfigSource object (#6743)
* Secure in secrets manager the dbtConfigSource object

* Add missing tests

* Fix minor bug

* Minor code refactor
2022-08-17 10:32:43 +02:00