393 Commits

Author SHA1 Message Date
Teddy
90abd73394
Fixes #7105 -- Add support for dataType filter for TestDefinition request (#7640)
* Updated json schema to support dataType filter

* Added migration for test definition support of data type filtering

* Added backend logic to support filter on data type for testdefinitions

* Added migration for:
- supported test types
- renaming of certain tests from starting uppercase to lowercase

* removed openmetadata_db from migration

* added datatype filter in column test

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-09-23 10:31:00 +02:00
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection (#7520)
* Fix #7121 - Support Spark metastore DB connection

* appname

* Update docs

* test validation

* Address PR comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Teddy
e22036cc09
Fixes #7504 Added migration steps to delete profiler pipeline from entity relationship (#7511)
* Added migration steps to delete profiler pipeline from entity relationship

* Added migration to remove rows from entity_relationship with profiler pipelines

* Fixed sample data TestDefinition name after renaming testDefinition

* Added user_tokens table to v005

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-20 08:14:29 +02:00
mohitdeuex
4c6932e172
[Basicauth] update table name (#7538)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts

* [BasicAuth] Update table name
2022-09-19 12:03:12 +05:30
mohitdeuex
4b594d0d6e
[BasicAuth] Added Basic Authentication (#7390)
* Fixed issue with testCases not posted in slack

* Return empty string in case of faling to build url

* [BasicAuth] Added Basic Authentication APIs

* [BasicAuth] Checkstyle

* [BasicAuth] Test yaml updated

* [BasicAuth] Update review comments

* [BasicAuth] Refresh logic added

* [BasicAuth] Add sql updates

* [BasicAuth] Maintain dependency version in main pom.xml

* [BasicAuth] Updated as per new module

* [BasicAuth] Fixes for refresh and missing edge scenario handled

* [BasicAuth] remove quotes

* [BasicAuth] Fix merge conflicts
2022-09-19 11:38:45 +05:30
Vivek Ratnavel Subramanian
2f84bfbfa8
Fix #7100 Upgrade issues for 0.12 release (#7101) 2022-09-01 22:22:42 -07:00
Teddy
704385da0a
Clean up profile and test data when migrating from 0.11 to 0.12 (#7039)
* Added SQL to clean up profile and test data when migrating from 0.11 to 0.12

* Added deletion query for ingestion pipeline mysql

* Added logic to delete profiler ingestion pipeline from DB

* Update v004__create_db_connection_info.sql

added semicolon
2022-08-30 18:43:12 +02:00
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
mohitdeuex
c8ab6fa59b
[Backend][Settings] Settings through UI (#6514) 2022-08-25 06:37:44 -07:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean (#6866)
* Databricks Connection - User & Password clean

* Fix: Python checkstyle failed

* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Pere Miquel Brull
3f1fc5344a
Remove Athena hostPort (#6825) 2022-08-19 14:11:53 +02:00
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-18 00:29:59 -07:00
Parth Panchal
e451a9ca5f
Fixed#5400: Added Kafka Event Publisher (#6697)
* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher

* Fixed#5400: Added Kafka Event Publisher
2022-08-11 20:27:15 +05:30
Suresh Srinivas
7825986105
Fixes #6554 Add support for team of type Group (#6588)
* Fixes #6554 Add support for team of type Group

* Fixing test failure
2022-08-05 10:49:42 -07:00
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579)
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
2022-08-05 10:28:04 +05:30
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Pere Miquel Brull
157fcd8dcf
DeltaLake does not support profiler (#6495) 2022-08-02 08:45:43 +02:00
Vivek Ratnavel Subramanian
62877c2e5d
Fix #6333 Backend: Add API support to create and list announcements (#6446) 2022-07-31 21:02:49 -07:00
Sriharsha Chintalapani
5f8e2a7f4f
Generic Test Case definition to capture different testcases (#5647) 2022-07-28 13:58:41 -07:00
mohitdeuex
160806c013
[Backend][Slack] Slack integration with UI (#6261)
* [Backend][Slack] Slack integration with UI

* [Backend][Slack] Slack integration with UI - removed unused jdbi

* [Backend][Slack] Slack integration with UI - URL fix + init webhook url during initialization

* [Backend][Slack] Slack integration with UI - remove slack

* [Backend][Slack] add Slack generic to exisiting jsons

* [Backend][Slack] PG fix and Java CheckStyle
2022-07-28 23:45:12 +05:30
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
2022-07-24 18:49:15 +02:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props (#5700)
Fix #5822 - Remove looker env and rename props (#5700)
2022-07-24 16:02:26 +02:00
Onkar Ravgan
1673b1aa30
DynamoDB Topology refactor (#5967)
DynamoDB Topology refactor (#5967)
2022-07-20 10:42:40 +02:00
Suresh Srinivas
4c62237bcb
Fixes #6133 - Add support for Team Hiearchy (#6134)
* Adding different team types for supporting organization team hierarchy

* Add support for team hierarchy

* Create organization during initalization

* Team hierachy test for POST methods

* Add PUT and PATCH support for changing team hiearchy

* Added tests for the team hiearchy feature

* Restored v003 sql files and added v004

* Merge conflicts with main and adopt new JSON based configuration

* Remove configuration Organization name to simplify user MUST do configurations

* Snowflake Make Query tag Optional (#6114)

* Snowflake make query tag optional

* Modified Description

* [Backend][UsageResource] Add PUT to enable updating Usage #6117 :::::: and allow adding space to tagCAtegory and Tags (#6119)

* [Backend][UsageResource] Add PUT to enable updating Usage #6117

* [Backend][UsageResource] Allow space in tags category and primary tags #6121

* [Backend][UsageResource] Update in more genric way to make use of space in resourcePath

* ui changes for allowing space in tag and tag category name

* [Backend][Improvement]add test for put and post

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>

* Adding different team types for supporting organization team hierarchy

* Add support for team hierarchy

* Create organization during initalization

* Team hierachy test for POST methods

* Add PUT and PATCH support for changing team hiearchy

* Added tests for the team hiearchy feature

* Restored v003 sql files and added v004

* Merge conflicts with main and adopt new JSON based configuration

* Remove configuration Organization name to simplify user MUST do configurations

* Python jsonSchema code generation not respecting  defaults

* fixed failing cypress

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: mohitdeuex <105265192+mohitdeuex@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2022-07-19 11:10:52 -07:00
Pere Miquel Brull
7e0a46394a
Update missing FQN (#5966) 2022-07-08 16:27:49 +02:00
Mayur Singal
dd97e849a5
Fix #5905: Bootsrap Migration Fix (#5930)
* Fix #5905: Bootsrap Migration Fix

* Delete from ingestion_pipeline_entity
2022-07-08 06:57:51 +02:00
Pere Miquel Brull
97761b97a6
Prepare postgres migrations (#5814) 2022-07-01 10:18:24 +02:00
Sriharsha Chintalapani
d42783fbe0
Fix #5415 - Add migrations (#5767)
* Add migrations

* Migration Extension

* removed repeated code

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-06-30 18:40:26 +05:30
Vivek Ratnavel Subramanian
e826c98acf
Fix migration script for migrations to 0.11 (#5773) 2022-06-30 00:38:24 -07:00
Vivek Ratnavel Subramanian
296f580304
Fix #4679 Backend: Support assigning a task to user/team and resolving the task (#5501) 2022-06-22 16:17:29 -07:00
Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices (#5446)
Fix #5367 - MlFlow connector & MLServices (#5446)
2022-06-21 14:54:36 +02:00
Parth Panchal
f8f79ebb2d
Fixed#2730: added status filter support for webhooks (#5214)
* Fixed#2730: added status filter support for webhooks

* added status param

* addressing comments

* postgres query change
2022-05-30 15:53:23 +05:30
Suresh Srinivas
9d25596227
Fixes #5122 Add backend and API support for column-level lineage (#5123)
* Fixes #5122 Add backend and API support for column-level lineage

* amend

* Fix test failures
2022-05-24 16:00:12 -07:00
Suresh Srinivas
0ba22c1d2f
Fixes #4977 Add APIs for adding custom fields to an existing entity (#4978) 2022-05-16 15:26:20 -07:00
Suresh Srinivas
6ce528d70a
Fixes #4921 Add support for type system (#4922) 2022-05-14 10:57:19 -07:00
mosiac1
bb131710e3
Add IF EXISTS checks to postgres migration (#4573) 2022-04-29 09:20:52 -07:00
Suresh Srinivas
f0139c3ea5
Make bootstrap sql files for mysql and postgres consistent with each other (#4541)
* Make mysql bootstrap file consistent with postgres

* Adding a java code change to trigger all the precommit checks

* Set test container reuse flag to true

* Make Postgres bootstrap files consistent with MySQL

* Test pagination using random jump of limit between 1-5 instead of 1 to reduce test time

* Starting the test container outside @BeforeAll in a static block for container reuse

* Comment printing out list output in pagination tests

* Move starting the test container to @BeforeAll setup method instead of static initialization

* Remove unnecessary tests in PolicyResourceTest to speed up tests

* Add test container reuse for github workflows

* Add test container reuse for github workflows

* Add test container reuse for github workflows

* Change timeouts for the TestContainer

* Add test container reuse for github workflows

* Add test container reuse for github workflows (#4563)

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2022-04-28 07:30:56 -07:00
mosiac1
c6e8181fa9
Add postgres support (#3952)
* Add postgres support

* Add testing for postgresql databases

Split surefire tests into: unit-tests, postgres-integration-tests, mysql-integration-tests
Integration tests get db information from system properties
All tests ending in ResourceTests.java are integration tests, all other are unit tests
Move to using testscointainers, remove embeded mysql

* Fix some tests

* Fix webhook and table tests

* Fix random name generation in tests, remove parallelism

* Fix style

* Add headers, fix config files

* Apply formatting

* Fix tag ussage query for postgres

* Replace references to mysql with generic database

* Fix postgres migration file

* Fix postgres migration script
2022-04-27 10:58:44 -07:00
Suresh Srinivas
1986423f9d
Remove JSON_EXTRACT calls to make it MySQL closer to Postgres bootstrap sql files (#4508) 2022-04-26 19:14:51 -07:00
Suresh Srinivas
e880224da8
Fixes #4448 Consolidate different versions of bootstrap sql files (#4450) 2022-04-24 13:16:24 -07:00
Suresh Srinivas
200e4f723f
Remove indexing updatedAt and updatedBy columns for database_schema_entity table (#4407) 2022-04-23 09:11:42 -07:00
Suresh Srinivas
a0b50d328a
Fixes #3807 Add DELETE api support for tag and tagCategory (#3898) 2022-04-06 17:10:29 -07:00
Suresh Srinivas
79535033ca
Fixes #3895 Refactor Tag and TagCategory repositories to use EntityRepository (#3896)
* Move TagCategory to use EntityRepository

* Handle names with '.' in tests and other clean ups
2022-04-06 11:30:08 -07:00
Suresh Srinivas
3b50eb9e52
Fixes #3602 Add support for database schema name (#3789)
* Fixes #3602 Add support for database schema name

* Fix package name issues with SourceConfig and other ingestion related types
2022-04-04 22:06:41 -07:00
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service (#3804)
* Fix #3659 Refactor Service Connection String to be specific to per service

* Simplify and centralize Airflow Pipeline info for REST (#3740)

* Remove code

* Modified Configs based on refactoring schema (#3816)

* Clean WorkflowContext


Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Suresh Srinivas
b89a9b40cb
Fixes #3769 Drop unnecessary column indexes (#3770) 2022-03-30 13:40:06 -07:00
Sriharsha Chintalapani
65c4030fb1
Issue-3291: Fix upgrade from 0.8.4 to 0.9.0 (#3292)
* Issue-3291: Fix upgrade from 0.8.4 to 0.9.0
2022-03-08 23:07:24 -08:00
Sriharsha Chintalapani
b35d1116df
Issue-3234: Rename default in role_entity to defaultRole (#3235) 2022-03-08 14:34:32 +05:30
Suresh Srinivas
f540981de1
Fixes #3239 Add support for using glossary terms as tag labels (#3240) 2022-03-07 19:35:48 -08:00