71 Commits

Author SHA1 Message Date
Parth Panchal
f8f79ebb2d
Fixed#2730: added status filter support for webhooks (#5214)
* Fixed#2730: added status filter support for webhooks

* added status param

* addressing comments

* postgres query change
2022-05-30 15:53:23 +05:30
Suresh Srinivas
9d25596227
Fixes #5122 Add backend and API support for column-level lineage (#5123)
* Fixes #5122 Add backend and API support for column-level lineage

* amend

* Fix test failures
2022-05-24 16:00:12 -07:00
Suresh Srinivas
0ba22c1d2f
Fixes #4977 Add APIs for adding custom fields to an existing entity (#4978) 2022-05-16 15:26:20 -07:00
Suresh Srinivas
6ce528d70a
Fixes #4921 Add support for type system (#4922) 2022-05-14 10:57:19 -07:00
mosiac1
bb131710e3
Add IF EXISTS checks to postgres migration (#4573) 2022-04-29 09:20:52 -07:00
Suresh Srinivas
f0139c3ea5
Make bootstrap sql files for mysql and postgres consistent with each other (#4541)
* Make mysql bootstrap file consistent with postgres

* Adding a java code change to trigger all the precommit checks

* Set test container reuse flag to true

* Make Postgres bootstrap files consistent with MySQL

* Test pagination using random jump of limit between 1-5 instead of 1 to reduce test time

* Starting the test container outside @BeforeAll in a static block for container reuse

* Comment printing out list output in pagination tests

* Move starting the test container to @BeforeAll setup method instead of static initialization

* Remove unnecessary tests in PolicyResourceTest to speed up tests

* Add test container reuse for github workflows

* Add test container reuse for github workflows

* Add test container reuse for github workflows

* Change timeouts for the TestContainer

* Add test container reuse for github workflows

* Add test container reuse for github workflows (#4563)

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2022-04-28 07:30:56 -07:00
mosiac1
c6e8181fa9
Add postgres support (#3952)
* Add postgres support

* Add testing for postgresql databases

Split surefire tests into: unit-tests, postgres-integration-tests, mysql-integration-tests
Integration tests get db information from system properties
All tests ending in ResourceTests.java are integration tests, all other are unit tests
Move to using testscointainers, remove embeded mysql

* Fix some tests

* Fix webhook and table tests

* Fix random name generation in tests, remove parallelism

* Fix style

* Add headers, fix config files

* Apply formatting

* Fix tag ussage query for postgres

* Replace references to mysql with generic database

* Fix postgres migration file

* Fix postgres migration script
2022-04-27 10:58:44 -07:00
Suresh Srinivas
1986423f9d
Remove JSON_EXTRACT calls to make it MySQL closer to Postgres bootstrap sql files (#4508) 2022-04-26 19:14:51 -07:00
Suresh Srinivas
e880224da8
Fixes #4448 Consolidate different versions of bootstrap sql files (#4450) 2022-04-24 13:16:24 -07:00
Suresh Srinivas
200e4f723f
Remove indexing updatedAt and updatedBy columns for database_schema_entity table (#4407) 2022-04-23 09:11:42 -07:00
Suresh Srinivas
a0b50d328a
Fixes #3807 Add DELETE api support for tag and tagCategory (#3898) 2022-04-06 17:10:29 -07:00
Suresh Srinivas
79535033ca
Fixes #3895 Refactor Tag and TagCategory repositories to use EntityRepository (#3896)
* Move TagCategory to use EntityRepository

* Handle names with '.' in tests and other clean ups
2022-04-06 11:30:08 -07:00
Suresh Srinivas
3b50eb9e52
Fixes #3602 Add support for database schema name (#3789)
* Fixes #3602 Add support for database schema name

* Fix package name issues with SourceConfig and other ingestion related types
2022-04-04 22:06:41 -07:00
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service (#3804)
* Fix #3659 Refactor Service Connection String to be specific to per service

* Simplify and centralize Airflow Pipeline info for REST (#3740)

* Remove code

* Modified Configs based on refactoring schema (#3816)

* Clean WorkflowContext


Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Suresh Srinivas
b89a9b40cb
Fixes #3769 Drop unnecessary column indexes (#3770) 2022-03-30 13:40:06 -07:00
Sriharsha Chintalapani
65c4030fb1
Issue-3291: Fix upgrade from 0.8.4 to 0.9.0 (#3292)
* Issue-3291: Fix upgrade from 0.8.4 to 0.9.0
2022-03-08 23:07:24 -08:00
Sriharsha Chintalapani
b35d1116df
Issue-3234: Rename default in role_entity to defaultRole (#3235) 2022-03-08 14:34:32 +05:30
Suresh Srinivas
f540981de1
Fixes #3239 Add support for using glossary terms as tag labels (#3240) 2022-03-07 19:35:48 -08:00
Vivek Ratnavel Subramanian
3d1c56511f
Fix #3028: Add filter support for Feed API (#3093) 2022-03-03 07:59:46 -08:00
Matt
9e4d8d709d
Support swapping default roles (#2868)
* Support swapping default roles

Pseudocode:

```
A. patchRole(role1, default=True):
B.   set role1.default = True
     for all users:
       add role1 to user.roles
C.   for role in roles if role != role1:
       set role.default = False
       for all users:
         delete role from user.roles
```

This ensures that changeDescription for the role(s) and user(s) are updated accordingly.

Potential optimization:
Adding role1 and removing role from user.roles could be considered/implemented as one change.
However, increases code complexity.

* Set DataConsumer as default role

* Fix tests

* Fix code smell
2022-02-20 11:27:32 -08:00
Matt
f0804816ab
Add support for default role (#2631) (#2676)
- Add default field to role
- Amend GET /roles to query default roles
- Assign default role to new users being created
2022-02-19 11:02:21 -08:00
Suresh Srinivas
28ba1a3c04
Fixes #2760 - Add entities for Glossary and initial API (#2761)
* Fixes #2760 - Add entities for Glossary and initial API

* Fixing merge issues
2022-02-15 20:54:46 -08:00
Vivek Ratnavel Subramanian
6447e34bff
Fix #2718: Update entity_thread schema to store activity feeds (#2726) 2022-02-10 20:47:31 -08:00
Alberto Miorin
0e73de4658
See #2486. Implement encryption at rest (#2488)
* See #2486. Add fernet Java library to encrypt DatabaseService.DatabaseConnection.password

* Add support for rotation.

* Add support for CLI to rotate the key.

* The fernet key can be null. No encryption will be done.

* Add FernetConfiguration

* Add env variable for Fernet.

* Convert base64 to base64url

* Decrypt (admin or bot) or nullify tokens for all methods including ListVersions and GetVersion

* Fix test with owner USER to handle null in DatabaseConnection
2022-02-08 09:50:39 +01:00
Matt
d2132551b1
Revert "Add support for default role (#2631)" (#2651)
This reverts commit bfcd97ed0bae1770deb992c6e4ec518b1ea0b663.
2022-02-07 10:54:26 +01:00
Matt
bfcd97ed0b
Add support for default role (#2631)
- Add default field to role
- Amend GET /roles to query default roles
- Assign default role to new users being created
2022-02-06 15:28:15 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357)
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
2022-01-22 21:12:25 -08:00
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341)
* Fix #2340: Ingestion pipeline deployment refactor
2022-01-21 22:06:14 -08:00
Sriharsha Chintalapani
138fd0747d Fix #2320: Fix DB migration scripts to upgrade to 0.8 from 0.7 2022-01-20 15:08:03 -08:00
Sriharsha Chintalapani
20e6c4bc86 fix upgrade 2022-01-20 15:08:03 -08:00
Matt
7181141803
Initialize roles and policies with seed data (#2236)
- Set up initial roles and policies seed data
- Extend EntityRepository to support initSeedData()
2022-01-17 00:31:12 -08:00
Suresh Srinivas
5a8e2ee31f
Issue2194 - Change dateTime field to unix epoch timestamp milliseconds and add support for webhook secret key (#2195)
* Add support for webhook secret key

* Fixes #2194 - Change dateTime field to unix epoch timestamp milliseconds and add support for webhook secret key

* Fixes #2194 - Added support for upgrading existing metadata to move dateTime attributes to timestamp
2022-01-13 12:08:42 -08:00
Suresh Srinivas
fb78944633
Merge v003 and v004 versions of bootstramp sql for 0.8 release (#2055) 2022-01-05 10:36:17 -08:00
Suresh Srinivas
7b96ccb827
Issue1977 - Add support for recursive deletion of entities (#1978)
* Recursive delete

* Fixes #1977 - Add support for recursive deletion of entities
2021-12-31 09:30:27 -08:00
Matt
485e661430
Add Role entity (#1970) 2021-12-30 08:30:12 -08:00
Suresh Srinivas
6d7a496aa4
Fixes #1961 - Delete API soft deletes the entities (#1968) 2021-12-29 14:20:31 -08:00
Suresh Srinivas
1b44e0480a
Fixes #1919 Fix flaky tests related to webhook (#1920) 2021-12-24 17:35:42 -08:00
Suresh Srinivas
5892595b1f
Fixes #1900 - Initial implementation of webhook API (#1905)
Co-authored-by: Suresh Srinivas <sureshsrinivas@Suresh-Collate.local>
2021-12-23 11:26:00 -08:00
Akash Jain
ed2a57fab2
Added usage info for #1814 (#1816) 2021-12-17 10:53:32 +05:30
Sriharsha Chintalapani
39e4163d6f
Fix #1814: Add migrate-all option to update ElasticSearch index mappings (#1815)
* Fix #1814: Add migrate-all option to update ElasticSearch index mappings

* Fix #1814: Add migrate-all option to update ElasticSearch index mappings
2021-12-16 20:56:58 -08:00
Suresh Srinivas
82f0557af5
Fixes #1678 - Cleanup mysql schema to make all entities consistent (#1679)
Co-authored-by: sureshms <suresh@getcollate.io>
2021-12-10 10:34:08 -08:00
Sriharsha Chintalapani
2ef91fff27
Fix #1593: bootstrap_storage.sh not working (#1594) 2021-12-07 10:31:30 +05:30
Alberto Miorin
4d20e187a3
Fix #1579: Drop and recreate MySQL and ES (#1589) 2021-12-06 23:28:35 +01:00
Sriharsha Chintalapani
2d348e9704
Add SSL options to ElasticSearch client - Ingestion & Server (#1548) (#1557)
* Add SSL options to ElasticSearch client - Ingestion & Server (#1548)

* Add SSL options to ElasticSearch client - Ingestion & Server

* Add SSL options to ElasticSearch client - Ingestion & Server

* Issue-1556: Support for ElasticSearch SSL , Bootstrap command create and delete, ingestion updates for change descriptions

* Issue-1556: Fix checkstyle issues

* Issue-1556: Fix checkstyle issues
2021-12-04 18:54:36 -08:00
parthp2107
a9e70e6a89
License header update (#1498)
* updated liscense header

* updated license header

* updated license header

* updated license header

* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Suresh Srinivas
9af02c8ab3
Issue1482 (#1483)
* Fixes #1447 Fix TableResourceTest.get_tableListWithDifferentFields_200_OK test failure

* Fixes #1482 - Change entityName limit to 256 characters

Co-authored-by: sureshms <suresh@getcollate.io>
2021-11-30 11:49:06 -08:00
Sriharsha Chintalapani
5a3bdf2f70
Fix #1327: rename Model to DBTModel and provide indexing support (#1330)
* Fix #1327: rename Model to DBTModel and provide indexing support

* Fix #1327: update dbt connector

* Fix #1327: add search index for dbt models

* Fix #1327: Address checkstyle issues
2021-11-23 13:37:52 -08:00
Sriharsha Chintalapani
eea60a8696
Fix #1264: Pipelines: remove 64 character constraint in names (#1265)
* Fix #1264: Pipelines: remove 64 character constraint in names
2021-11-18 16:27:43 -08:00
Sriharsha Chintalapani
345ad9d413
Fix #1159: Add Model as Entity (#1160)
* Fix #1159: Add Model as Entity

* Fix #1159: Add Model as Entity

* Fix #1159: Add Model as Entity; Address check style issues
2021-11-15 12:51:33 -08:00
Sriharsha Chintalapani
ab836ac26f
Fix #1151: Refactor Model to MLModel (#1152)
* Fix #1151: Refactor Model to MLModel

* Fix #1151: Refactor Model to MLModel

* Fix #1151: Refactor Model to MLModel

* Fix #1151: Refactor Model to MLModel
2021-11-11 12:44:03 -08:00