80 Commits

Author SHA1 Message Date
codingwithabhi
c09dc3bff7
athena-source-connection-fixed (#4278)
* athena-source-connection-fixed

* ran-core_bump_version_dev

* aws-cred-json-schema-support-added-in-athena

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-04-21 16:07:29 +02:00
Suresh Srinivas
14fce51749
Fixes #4255 Move associated tags functionality from Tags to GlossaryT… (#4256) 2022-04-19 15:29:28 -07:00
Akash Jain
540f5ca3b5
Fix: Prepare 0.10.0 snapshot (#4190)
* update pom versions to 0.10.0-SNAPSHOT

* fix version to latest release 0.9.1

* fix ingestion core version to 0.10.0
2022-04-19 17:40:42 +05:30
codingwithabhi
8115c2ece4
issue-4135: updated-mysql-connection-json (#4136) 2022-04-14 14:16:09 -07:00
Suresh Srinivas
339968c71e
Issue4113 (#4114) 2022-04-13 19:51:04 -07:00
Sriharsha Chintalapani
be836e5404
Fix #4071: PUT IngestionPipeline missing property & error message (#4085)
Fix #4071: PUT IngestionPipeline missing property & error message (#4085)
2022-04-13 08:40:21 +02:00
Sriharsha Chintalapani
966cedd17f Fix #4042: Ingestion: Sample data ingestion is failing 2022-04-11 16:23:00 -07:00
Sriharsha Chintalapani
2e870669e3 Fix #4042: Ingestion: Sample data ingestion is failing 2022-04-11 16:12:01 -07:00
Sriharsha Chintalapani
bd4bcaf876 Fix #4044: Service connection supportedPipelineType to be a common type 2022-04-11 14:22:58 -07:00
Sriharsha Chintalapani
8067103a1d Fix #4044: Service connection supportedPipelineType to be a common type 2022-04-11 14:22:30 -07:00
Sriharsha Chintalapani
5f47b66742 Fix #4044: Service connection supportedPipelineType to be a common type 2022-04-11 14:22:30 -07:00
codingwithabhi
01336853ef
Snowflake private key fix (#4031)
* snowflake-private-key-fixed

* ran-core_bump_version_dev
2022-04-11 19:27:50 +05:30
Sriharsha Chintalapani
afd7d1b9a7 Fix #3721: Typo in table.json schema 2022-04-10 17:04:12 -07:00
codingwithabhi
c6bd06944b
glue-connector-fixed (#3980)
* glue-connector-fixed

* code-formatted

* ran-make-core_bump_version_dev

* config-stored-in-variable
2022-04-09 18:28:53 +05:30
codingwithabhi
a92dff15e6
multiple-database-support-added (#3777)
* multiple-database-support-added

* FQDN_SEPARATOR-added

* code-formatted

* code-formatted
2022-04-09 15:11:11 +05:30
codingwithabhi
0e0c3459a8
postgres-connector-fixed (#3970) 2022-04-08 13:28:00 -07:00
codingwithabhi
bb76a2f11b
salesforce-connector-fixed (#3923)
* salesforce-connector-fixed

* salesforce-connection-updated

* ran-make-core_bump_version_dev

* code-formatted

* removed-get_connection_url

* code-formatted

* code-formatted

* code-formatted
2022-04-08 16:08:31 +05:30
codingwithabhi
300e8fec55
snowflake-connector-fix (#3885)
* snowflake-connector-fixed

* schema-filter-fixed

* updated-connection_arguments

* ran-py_format

* removed-FQN-computation

* ran-make-core_bump_version_dev-and-py_format

* ranpy_format
2022-04-08 13:28:54 +05:30
Mayur Singal
3651efd7f5
Databricks Source Fix (#3917)
Databricks Source Fix (#3917)
2022-04-07 16:40:50 +02:00
Mayur Singal
b709a8055e
Fixed MSSQL Source (#3877)
Fixed MSSQL Source (#3877)
2022-04-06 14:21:46 +02:00
codingwithabhi
dbfe488a9e
Schema support fix (#3856)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-04-05 10:51:42 -07:00
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service (#3804)
* Fix #3659 Refactor Service Connection String to be specific to per service

* Simplify and centralize Airflow Pipeline info for REST (#3740)

* Remove code

* Modified Configs based on refactoring schema (#3816)

* Clean WorkflowContext


Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Sriharsha Chintalapani
f79b251509
Fix #3771: Main branch schemas and ingestion is failing; Fix #2992: Remove ChangeEvents log from ElasticSearch Indexes (#3772) 2022-03-30 18:26:57 -07:00
Pere Miquel Brull
e830f1787f
Fix #3726 - Remove mlmodel custom defs (#3728) 2022-03-29 10:01:36 +02:00
Suresh Srinivas
de6fb11927
Fixes #3653 - Variable based separator used for fullyQualifiedName instead of hardcoded "." (#3674)
* Use common entityName and fullyQualifiedEntityName type definition with DOT allowed

* Make EntityInterface abstract class and move building entity reference to the base class

* Entity separator is not longer hardcoded but uses a variable
2022-03-25 14:48:20 -07:00
Ayush Shah
07266b58f7
Fix: Added DeltaLake Service type (#3639)
* fix-deltalake

* added deltalake.json

* Bumped up version
2022-03-24 18:42:00 +05:30
Sriharsha Chintalapani
50ee11e6ca
Issue-3577: Add partitioning details to Table Entity (#3579) 2022-03-22 18:37:56 -07:00
Pere Miquel Brull
dfd6286cb4
Fix #3463 - Prepare Custom Metric definition (#3467)
Fix #3463 - Prepare Custom Metric definition (#3467)
2022-03-22 15:29:41 +01:00
Suresh Srinivas
eb74487985
Issue3536 (#3537) 2022-03-20 18:01:51 -07:00
Sriharsha Chintalapani
0b30c9f08d
Fix #3056: Introduce team owner and let them manage their own team; Fix #3516: Authorizer should expect an owner field to be populated before granting permissions. (#3517) 2022-03-18 18:33:24 -07:00
Pere Miquel Brull
16e82d45de
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
2022-03-16 06:05:59 +01:00
Ayush Shah
7c84f063e6
Minor Fixes - Ingestion - Datetime format, Looker fix (#3365)
* Minor Fixes - UTC timezone for usage, tag method modification, looker fix

* Modified core version

* Snowflake Usage Result limit and Makefile ignore env fix
2022-03-11 13:31:42 +05:30
Pere Miquel Brull
f0dd85b9f7
Fix #3356 - Vertica concat and len (#3358) 2022-03-10 08:01:47 -08:00
parthp2107
794cf64787
updated schema docs (#3329)
* updated schema docs

* addressing ingestion-core version bump
2022-03-10 12:55:35 +05:30
Sriharsha Chintalapani
65c4030fb1
Issue-3291: Fix upgrade from 0.8.4 to 0.9.0 (#3292)
* Issue-3291: Fix upgrade from 0.8.4 to 0.9.0
2022-03-08 23:07:24 -08:00
Akash Jain
211a8a55e8
feat: Prepare 0.9.1-SNAPSHOT (#3297)
* prepare snapshot version for 0.9.1

* prepare dev version for ingestion 0.9.1

* prepare dev version for ingestion-core 0.9.1

* prepare Docker CI workflows for 0.9.1
2022-03-09 12:08:42 +05:30
Pere Miquel Brull
94d7500216
Fix #3248 & #3251 - Update metrics and column profile (#3262)
Fix #3248 & #3251 - Update metrics and column profile (#3262)
2022-03-08 11:44:39 +01:00
Pere Miquel Brull
434aa30a92
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
2022-03-08 09:15:20 +01:00
Pere Miquel Brull
954908efec
Fix #3217 - Test case props should be unique (#3220)
Fix #3217 - Test case props should be unique (#3220)
2022-03-07 14:43:42 +01:00
Pere Miquel Brull
4116233697
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
2022-03-04 18:11:49 +01:00
Pere Miquel Brull
bd7b91b448
Fix #3112 - col profile safety & sample data (#3142)
Fix #3112 - col profile safety & sample data (#3142)
2022-03-04 13:14:11 +01:00
Pere Miquel Brull
e96ac838ff
Fix #3084 - Implement missing tests (#3117)
Fix #3084 - Implement missing tests
2022-03-04 06:59:47 +01:00
Matt
6c4bb1f401
Clean up System.out.println and bump up ingestion-core version (#3072) 2022-03-02 01:35:24 -08:00
Ayush Shah
568845b65c
Power BI connector (#3019)
* Power BI Dashboard and Tiles added

* Added Powerbi dependency

* Modified Powerbi debug logs

* Bump up _version.py

* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Pere Miquel Brull
990608522a
Fix #2981 - Update Profile to match TableProfile (#2982) 2022-02-25 09:26:30 -08:00
Pere Miquel Brull
4d928bab80
Fix #2965 - Additional Properties set to False (#2976) 2022-02-24 08:38:56 -08:00
Matt
a1257bdcc9
Support returning teams for GET /roles (#2958) 2022-02-23 19:29:20 -08:00
Matt
0791c8c266
Extend team to support defaultRoles (#2885)
* Extend team to support defaultRoles

* Add support for team defaultRoles change description

* Add tests
2022-02-20 21:21:08 -08:00
Matt
9e4d8d709d
Support swapping default roles (#2868)
* Support swapping default roles

Pseudocode:

```
A. patchRole(role1, default=True):
B.   set role1.default = True
     for all users:
       add role1 to user.roles
C.   for role in roles if role != role1:
       set role.default = False
       for all users:
         delete role from user.roles
```

This ensures that changeDescription for the role(s) and user(s) are updated accordingly.

Potential optimization:
Adding role1 and removing role from user.roles could be considered/implemented as one change.
However, increases code complexity.

* Set DataConsumer as default role

* Fix tests

* Fix code smell
2022-02-20 11:27:32 -08:00
Pere Miquel Brull
e55579aaa8
Fix #2845 - Init Profiler Workflow (#2862)
* Fix list typing

* Add sqlite service

* Add sqlite service

* Add sqlite service

* Refactor validation into class

* Refactor validation into class

* Prepare table simple profiler

* Add note

* test ORM conversion

* Prepare workflow config utilities

* Prepare workflow skeleton

* Use new core Validation

* Refactor workflow config parsing

* Add comment

* Simplify workflow validations

* Fix table metric check

* Add init for convenience, otherwise interpreter cries when trying to __call__ the get result

* Fix table metric check

* Format

* Format

* Fix table list and metrics init

* Prepare profiler workflow integration tests

* Bump version

* Fix pycharm imports

* format
2022-02-20 17:55:12 +01:00