120 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
5f47b66742 Fix #4044: Service connection supportedPipelineType to be a common type 2022-04-11 14:22:30 -07:00
codingwithabhi
01336853ef
Snowflake private key fix (#4031)
* snowflake-private-key-fixed

* ran-core_bump_version_dev
2022-04-11 19:27:50 +05:30
Sriharsha Chintalapani
afd7d1b9a7 Fix #3721: Typo in table.json schema 2022-04-10 17:04:12 -07:00
codingwithabhi
c6bd06944b
glue-connector-fixed (#3980)
* glue-connector-fixed

* code-formatted

* ran-make-core_bump_version_dev

* config-stored-in-variable
2022-04-09 18:28:53 +05:30
codingwithabhi
a92dff15e6
multiple-database-support-added (#3777)
* multiple-database-support-added

* FQDN_SEPARATOR-added

* code-formatted

* code-formatted
2022-04-09 15:11:11 +05:30
codingwithabhi
0e0c3459a8
postgres-connector-fixed (#3970) 2022-04-08 13:28:00 -07:00
codingwithabhi
bb76a2f11b
salesforce-connector-fixed (#3923)
* salesforce-connector-fixed

* salesforce-connection-updated

* ran-make-core_bump_version_dev

* code-formatted

* removed-get_connection_url

* code-formatted

* code-formatted

* code-formatted
2022-04-08 16:08:31 +05:30
codingwithabhi
300e8fec55
snowflake-connector-fix (#3885)
* snowflake-connector-fixed

* schema-filter-fixed

* updated-connection_arguments

* ran-py_format

* removed-FQN-computation

* ran-make-core_bump_version_dev-and-py_format

* ranpy_format
2022-04-08 13:28:54 +05:30
Mayur Singal
3651efd7f5
Databricks Source Fix (#3917)
Databricks Source Fix (#3917)
2022-04-07 16:40:50 +02:00
Mayur Singal
b709a8055e
Fixed MSSQL Source (#3877)
Fixed MSSQL Source (#3877)
2022-04-06 14:21:46 +02:00
codingwithabhi
dbfe488a9e
Schema support fix (#3856)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-04-05 10:51:42 -07:00
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service (#3804)
* Fix #3659 Refactor Service Connection String to be specific to per service

* Simplify and centralize Airflow Pipeline info for REST (#3740)

* Remove code

* Modified Configs based on refactoring schema (#3816)

* Clean WorkflowContext


Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Sriharsha Chintalapani
f79b251509
Fix #3771: Main branch schemas and ingestion is failing; Fix #2992: Remove ChangeEvents log from ElasticSearch Indexes (#3772) 2022-03-30 18:26:57 -07:00
Pere Miquel Brull
e830f1787f
Fix #3726 - Remove mlmodel custom defs (#3728) 2022-03-29 10:01:36 +02:00
Suresh Srinivas
de6fb11927
Fixes #3653 - Variable based separator used for fullyQualifiedName instead of hardcoded "." (#3674)
* Use common entityName and fullyQualifiedEntityName type definition with DOT allowed

* Make EntityInterface abstract class and move building entity reference to the base class

* Entity separator is not longer hardcoded but uses a variable
2022-03-25 14:48:20 -07:00
Ayush Shah
07266b58f7
Fix: Added DeltaLake Service type (#3639)
* fix-deltalake

* added deltalake.json

* Bumped up version
2022-03-24 18:42:00 +05:30
Sriharsha Chintalapani
50ee11e6ca
Issue-3577: Add partitioning details to Table Entity (#3579) 2022-03-22 18:37:56 -07:00
Pere Miquel Brull
dfd6286cb4
Fix #3463 - Prepare Custom Metric definition (#3467)
Fix #3463 - Prepare Custom Metric definition (#3467)
2022-03-22 15:29:41 +01:00
Suresh Srinivas
eb74487985
Issue3536 (#3537) 2022-03-20 18:01:51 -07:00
Sriharsha Chintalapani
0b30c9f08d
Fix #3056: Introduce team owner and let them manage their own team; Fix #3516: Authorizer should expect an owner field to be populated before granting permissions. (#3517) 2022-03-18 18:33:24 -07:00
Pere Miquel Brull
16e82d45de
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
Fix #3371 - Run Profiler and Tests on a % of the data (#3424)
2022-03-16 06:05:59 +01:00
Ayush Shah
7c84f063e6
Minor Fixes - Ingestion - Datetime format, Looker fix (#3365)
* Minor Fixes - UTC timezone for usage, tag method modification, looker fix

* Modified core version

* Snowflake Usage Result limit and Makefile ignore env fix
2022-03-11 13:31:42 +05:30
Pere Miquel Brull
f0dd85b9f7
Fix #3356 - Vertica concat and len (#3358) 2022-03-10 08:01:47 -08:00
parthp2107
794cf64787
updated schema docs (#3329)
* updated schema docs

* addressing ingestion-core version bump
2022-03-10 12:55:35 +05:30
Sriharsha Chintalapani
65c4030fb1
Issue-3291: Fix upgrade from 0.8.4 to 0.9.0 (#3292)
* Issue-3291: Fix upgrade from 0.8.4 to 0.9.0
2022-03-08 23:07:24 -08:00
Akash Jain
211a8a55e8
feat: Prepare 0.9.1-SNAPSHOT (#3297)
* prepare snapshot version for 0.9.1

* prepare dev version for ingestion 0.9.1

* prepare dev version for ingestion-core 0.9.1

* prepare Docker CI workflows for 0.9.1
2022-03-09 12:08:42 +05:30
Pere Miquel Brull
94d7500216
Fix #3248 & #3251 - Update metrics and column profile (#3262)
Fix #3248 & #3251 - Update metrics and column profile (#3262)
2022-03-08 11:44:39 +01:00
Pere Miquel Brull
434aa30a92
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
2022-03-08 09:15:20 +01:00
Pere Miquel Brull
954908efec
Fix #3217 - Test case props should be unique (#3220)
Fix #3217 - Test case props should be unique (#3220)
2022-03-07 14:43:42 +01:00
Pere Miquel Brull
4116233697
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
2022-03-04 18:11:49 +01:00
Pere Miquel Brull
bd7b91b448
Fix #3112 - col profile safety & sample data (#3142)
Fix #3112 - col profile safety & sample data (#3142)
2022-03-04 13:14:11 +01:00
Pere Miquel Brull
e96ac838ff
Fix #3084 - Implement missing tests (#3117)
Fix #3084 - Implement missing tests
2022-03-04 06:59:47 +01:00
Matt
6c4bb1f401
Clean up System.out.println and bump up ingestion-core version (#3072) 2022-03-02 01:35:24 -08:00
Ayush Shah
568845b65c
Power BI connector (#3019)
* Power BI Dashboard and Tiles added

* Added Powerbi dependency

* Modified Powerbi debug logs

* Bump up _version.py

* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Pere Miquel Brull
990608522a
Fix #2981 - Update Profile to match TableProfile (#2982) 2022-02-25 09:26:30 -08:00
Pere Miquel Brull
4d928bab80
Fix #2965 - Additional Properties set to False (#2976) 2022-02-24 08:38:56 -08:00
Matt
a1257bdcc9
Support returning teams for GET /roles (#2958) 2022-02-23 19:29:20 -08:00
Matt
0791c8c266
Extend team to support defaultRoles (#2885)
* Extend team to support defaultRoles

* Add support for team defaultRoles change description

* Add tests
2022-02-20 21:21:08 -08:00
Matt
9e4d8d709d
Support swapping default roles (#2868)
* Support swapping default roles

Pseudocode:

```
A. patchRole(role1, default=True):
B.   set role1.default = True
     for all users:
       add role1 to user.roles
C.   for role in roles if role != role1:
       set role.default = False
       for all users:
         delete role from user.roles
```

This ensures that changeDescription for the role(s) and user(s) are updated accordingly.

Potential optimization:
Adding role1 and removing role from user.roles could be considered/implemented as one change.
However, increases code complexity.

* Set DataConsumer as default role

* Fix tests

* Fix code smell
2022-02-20 11:27:32 -08:00
Pere Miquel Brull
e55579aaa8
Fix #2845 - Init Profiler Workflow (#2862)
* Fix list typing

* Add sqlite service

* Add sqlite service

* Add sqlite service

* Refactor validation into class

* Refactor validation into class

* Prepare table simple profiler

* Add note

* test ORM conversion

* Prepare workflow config utilities

* Prepare workflow skeleton

* Use new core Validation

* Refactor workflow config parsing

* Add comment

* Simplify workflow validations

* Fix table metric check

* Add init for convenience, otherwise interpreter cries when trying to __call__ the get result

* Fix table metric check

* Format

* Format

* Fix table list and metrics init

* Prepare profiler workflow integration tests

* Bump version

* Fix pycharm imports

* format
2022-02-20 17:55:12 +01:00
Matt
f0804816ab
Add support for default role (#2631) (#2676)
- Add default field to role
- Amend GET /roles to query default roles
- Assign default role to new users being created
2022-02-19 11:02:21 -08:00
Sriharsha Chintalapani
165f5bb0e9
Fix #2819: Kafka connector security settings along with all configs to be retrieved (#2823)
* Fix #2811: Log level option for running metadata

* Fix #2819: Kafka connector security settings along with all configs to be retrieved

* Fix #2819: Kafka connector security settings along with all configs to be retrieved
2022-02-17 15:17:43 +05:30
Alberto Miorin
b3b5f44c23
Fix #1876: Allow dots in entity names (#2700)
Add getter and setter for `name` into EntityInterface

Refactor code in utility method.

Rename the escape method.

Fix webhook.

Fix pipeline tests

Fix airflowPipeline tests

Fix role, team, and user tests

Replace . with _DOT_ to keep compatibility with the past and avoid a migration.

Remove the Python code that was replacing the . with _DOT_.

Restore "no dots" for services and implement the JSON Schema rule.

** Entites escaping dots
1. AirflowPipeline
2. Chart
3. Dashboard
4. Database
5. Location
6. Metrics
7. MlModel
8. Pipeline
9. Table
10. Topic

** Entities rejecting dots
1. Database Service
2. Dashboard Service
3. Messaging Service
4. Pipeline Service
5. Storage Service

** Entities accepting dots
1. Bots
2. Policy
3. Report
4. Role
5. Team
6. User
7. Webhook

** Fields escaping dots
1. Column
2. Task

Add escape for Column and TableConstraints

Add escape to Task

Remove the Python code. It will be added in a follow-up PR.

Fix Pipeline entity tests
2022-02-17 08:21:53 +01:00
Suresh Srinivas
4eee1e93cb
Fixes #2812 - Add support for querying glossary term filtered by glossary or parent term (#2813) 2022-02-16 19:31:13 -08:00
Suresh Srinivas
ac59a092ea
Fixes #2779 Change the webhook state names (#2782) 2022-02-16 08:59:37 -08:00
Mayur Singal
5e4cd516bf
issue #2712: Added SingleStore Connector (#2747) 2022-02-15 21:11:46 -08:00
Pere Miquel Brull
53aeddf2c6
Fix #2226 - Add pipeline & task status (#2692)
* Add pipeline & task status

* Bump core version

* Simplify validateColumn

* Add apiResponse

* Use enum and timestamp

* Simplify validate and use timestamp for executionDate

* mvn format
2022-02-12 16:21:31 +01:00
Alberto Miorin
c9b78e9e65
Bump openmetadata-core (#2673)
* Bump openmetadata-core

* To dev=2
2022-02-08 16:15:20 +05:30
Matt
d2132551b1
Revert "Add support for default role (#2631)" (#2651)
This reverts commit bfcd97ed0bae1770deb992c6e4ec518b1ea0b663.
2022-02-07 10:54:26 +01:00
Matt
bfcd97ed0b
Add support for default role (#2631)
- Add default field to role
- Amend GET /roles to query default roles
- Assign default role to new users being created
2022-02-06 15:28:15 -08:00