95 Commits

Author SHA1 Message Date
Akash Jain
211a8a55e8
feat: Prepare 0.9.1-SNAPSHOT (#3297)
* prepare snapshot version for 0.9.1

* prepare dev version for ingestion 0.9.1

* prepare dev version for ingestion-core 0.9.1

* prepare Docker CI workflows for 0.9.1
2022-03-09 12:08:42 +05:30
Pere Miquel Brull
94d7500216
Fix #3248 & #3251 - Update metrics and column profile (#3262)
Fix #3248 & #3251 - Update metrics and column profile (#3262)
2022-03-08 11:44:39 +01:00
Pere Miquel Brull
434aa30a92
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
Fix #3221 - columnValuesToBeUnique vs. columnValuesToBeNotNull JSONs (#3255)
2022-03-08 09:15:20 +01:00
Pere Miquel Brull
954908efec
Fix #3217 - Test case props should be unique (#3220)
Fix #3217 - Test case props should be unique (#3220)
2022-03-07 14:43:42 +01:00
Pere Miquel Brull
4116233697
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
Fix #3105 - ColumnValuesToMatchRegex & other fixes (#3149)
2022-03-04 18:11:49 +01:00
Pere Miquel Brull
bd7b91b448
Fix #3112 - col profile safety & sample data (#3142)
Fix #3112 - col profile safety & sample data (#3142)
2022-03-04 13:14:11 +01:00
Pere Miquel Brull
e96ac838ff
Fix #3084 - Implement missing tests (#3117)
Fix #3084 - Implement missing tests
2022-03-04 06:59:47 +01:00
Matt
6c4bb1f401
Clean up System.out.println and bump up ingestion-core version (#3072) 2022-03-02 01:35:24 -08:00
Ayush Shah
568845b65c
Power BI connector (#3019)
* Power BI Dashboard and Tiles added

* Added Powerbi dependency

* Modified Powerbi debug logs

* Bump up _version.py

* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Pere Miquel Brull
990608522a
Fix #2981 - Update Profile to match TableProfile (#2982) 2022-02-25 09:26:30 -08:00
Pere Miquel Brull
4d928bab80
Fix #2965 - Additional Properties set to False (#2976) 2022-02-24 08:38:56 -08:00
Matt
a1257bdcc9
Support returning teams for GET /roles (#2958) 2022-02-23 19:29:20 -08:00
Matt
0791c8c266
Extend team to support defaultRoles (#2885)
* Extend team to support defaultRoles

* Add support for team defaultRoles change description

* Add tests
2022-02-20 21:21:08 -08:00
Matt
9e4d8d709d
Support swapping default roles (#2868)
* Support swapping default roles

Pseudocode:

```
A. patchRole(role1, default=True):
B.   set role1.default = True
     for all users:
       add role1 to user.roles
C.   for role in roles if role != role1:
       set role.default = False
       for all users:
         delete role from user.roles
```

This ensures that changeDescription for the role(s) and user(s) are updated accordingly.

Potential optimization:
Adding role1 and removing role from user.roles could be considered/implemented as one change.
However, increases code complexity.

* Set DataConsumer as default role

* Fix tests

* Fix code smell
2022-02-20 11:27:32 -08:00
Pere Miquel Brull
e55579aaa8
Fix #2845 - Init Profiler Workflow (#2862)
* Fix list typing

* Add sqlite service

* Add sqlite service

* Add sqlite service

* Refactor validation into class

* Refactor validation into class

* Prepare table simple profiler

* Add note

* test ORM conversion

* Prepare workflow config utilities

* Prepare workflow skeleton

* Use new core Validation

* Refactor workflow config parsing

* Add comment

* Simplify workflow validations

* Fix table metric check

* Add init for convenience, otherwise interpreter cries when trying to __call__ the get result

* Fix table metric check

* Format

* Format

* Fix table list and metrics init

* Prepare profiler workflow integration tests

* Bump version

* Fix pycharm imports

* format
2022-02-20 17:55:12 +01:00
Matt
f0804816ab
Add support for default role (#2631) (#2676)
- Add default field to role
- Amend GET /roles to query default roles
- Assign default role to new users being created
2022-02-19 11:02:21 -08:00
Sriharsha Chintalapani
165f5bb0e9
Fix #2819: Kafka connector security settings along with all configs to be retrieved (#2823)
* Fix #2811: Log level option for running metadata

* Fix #2819: Kafka connector security settings along with all configs to be retrieved

* Fix #2819: Kafka connector security settings along with all configs to be retrieved
2022-02-17 15:17:43 +05:30
Alberto Miorin
b3b5f44c23
Fix #1876: Allow dots in entity names (#2700)
Add getter and setter for `name` into EntityInterface

Refactor code in utility method.

Rename the escape method.

Fix webhook.

Fix pipeline tests

Fix airflowPipeline tests

Fix role, team, and user tests

Replace . with _DOT_ to keep compatibility with the past and avoid a migration.

Remove the Python code that was replacing the . with _DOT_.

Restore "no dots" for services and implement the JSON Schema rule.

** Entites escaping dots
1. AirflowPipeline
2. Chart
3. Dashboard
4. Database
5. Location
6. Metrics
7. MlModel
8. Pipeline
9. Table
10. Topic

** Entities rejecting dots
1. Database Service
2. Dashboard Service
3. Messaging Service
4. Pipeline Service
5. Storage Service

** Entities accepting dots
1. Bots
2. Policy
3. Report
4. Role
5. Team
6. User
7. Webhook

** Fields escaping dots
1. Column
2. Task

Add escape for Column and TableConstraints

Add escape to Task

Remove the Python code. It will be added in a follow-up PR.

Fix Pipeline entity tests
2022-02-17 08:21:53 +01:00
Suresh Srinivas
4eee1e93cb
Fixes #2812 - Add support for querying glossary term filtered by glossary or parent term (#2813) 2022-02-16 19:31:13 -08:00
Suresh Srinivas
ac59a092ea
Fixes #2779 Change the webhook state names (#2782) 2022-02-16 08:59:37 -08:00
Mayur Singal
5e4cd516bf
issue #2712: Added SingleStore Connector (#2747) 2022-02-15 21:11:46 -08:00
Pere Miquel Brull
53aeddf2c6
Fix #2226 - Add pipeline & task status (#2692)
* Add pipeline & task status

* Bump core version

* Simplify validateColumn

* Add apiResponse

* Use enum and timestamp

* Simplify validate and use timestamp for executionDate

* mvn format
2022-02-12 16:21:31 +01:00
Alberto Miorin
c9b78e9e65
Bump openmetadata-core (#2673)
* Bump openmetadata-core

* To dev=2
2022-02-08 16:15:20 +05:30
Matt
d2132551b1
Revert "Add support for default role (#2631)" (#2651)
This reverts commit bfcd97ed0bae1770deb992c6e4ec518b1ea0b663.
2022-02-07 10:54:26 +01:00
Matt
bfcd97ed0b
Add support for default role (#2631)
- Add default field to role
- Amend GET /roles to query default roles
- Assign default role to new users being created
2022-02-06 15:28:15 -08:00
Akash Jain
80a7a2fc3d
update ingestion-core version for #2348 (#2523) 2022-01-31 22:07:17 -08:00
Pere Miquel Brull
6ea6f0c43d
Fix #2383 - Multiple validation workflows (#2500)
* Clean duplicated validation

* Move black dep to dev

* Fix black version for ingestion core

* Validate core generated in the same workflow
2022-01-30 12:26:41 -08:00
Akash Jain
20309cde64
fix:cherry pick from 0.8.0 and miscellaneous fixes (#2445)
* updating ingestion-core version to 0.8.0

* updating ingestion version to 0.8.0

* fixing file copy path

* update docker images to point to 0.8.0

* set up new release version

* reset ingestion version to 0.9.0.dev0

* reset ingestion-core version to 0.9.0
2022-01-26 14:00:27 +05:30
Matt
c092b03171
Add RBAC for UpdateLineage MetadataOperation (#2406) 2022-01-25 02:11:37 -08:00
Akash Jain
71a2b0145c
bump version ingestion-core (#2374) 2022-01-24 14:10:31 +05:30
Akash Jain
c3ba6d6efd
Update _version.py (#2371) 2022-01-24 12:26:54 +05:30
Suresh Srinivas
8bb50d1a24
Fixes #2364 Use terms entityType and entityName consistently (#2365) 2022-01-23 15:23:19 -08:00
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359)
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
2022-01-23 09:47:06 -08:00
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357)
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
2022-01-22 21:12:25 -08:00
Matt
8e5a9c1f0e
Set Access Control Rule default priority to 25000 (#2346) 2022-01-22 02:14:56 -08:00
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341)
* Fix #2340: Ingestion pipeline deployment refactor
2022-01-21 22:06:14 -08:00
Matt
845b62d48d
Add Access Control Policy for DataConsumer role (#2328) 2022-01-21 00:26:06 -08:00
Matt
065d051dc1
GET /v1/roles?:roleId?fields=users should return users (#2308) 2022-01-20 08:10:10 -08:00
Pere Miquel Brull
a9c840778a
[issue-2116] - Add target property to MlModel (#2134)
* Add target property to MlModel

* Use pull_request_target for forks

* Simplify str

* Update ingestion-core setup

* Bump ingestion-core version

* Update install
2022-01-18 16:27:35 +01:00
Pere Miquel Brull
de3b3e4abc
[issue-2239] - Update core publish to push (#2240)
* Update core publish to push

* Rename jobs to be py explicit

* Upload to testpypi
2022-01-17 10:29:55 -08:00
Matt
7181141803
Initialize roles and policies with seed data (#2236)
- Set up initial roles and policies seed data
- Extend EntityRepository to support initSeedData()
2022-01-17 00:31:12 -08:00
Akash Jain
00ccd612ee
fix: Ingestion Core CI to Publish PyPi packages (#2187)
* Remove gitignore on _versions.py

CI commits and pushes the new version in _versions.py and since the folder is marked under gitignore, CI fails to commit and push changes to the branch.

* bump version to 0.8.0.dev0 as already published
2022-01-13 12:40:47 +05:30
Alberto Miorin
c8de136cca
First part of #2091: Python code in branch main is always formatted correctly (#2093) 2022-01-07 06:17:48 +01:00
Akash Jain
5685efd651
feat: openmetadata-ingestion-core pypi publish dev packages (#2023)
* update requirements-dev.txt

* integrate pypi incremental

* Add CI for #1875

* gitignore generated dir in ingestion-core

* update version to match ingestion package
2022-01-04 21:11:33 -08:00
Alberto Miorin
c31ba9cf5b
Fix #1861: Rename metadata-ingestion-generated to metadata-ingestion-core (#1862) 2021-12-20 22:29:07 -08:00