250 Commits

Author SHA1 Message Date
Suresh Srinivas
d758a542aa
Fix upgrade issues with new bot role feature (#8895) 2022-11-18 11:32:26 -08:00
Mohit Yadav
884b747485
Added Task Types for DataInsight and ES Reindex (#8882) 2022-11-18 15:07:13 +00:00
Mohit Yadav
e966bb343b
Refresh token Changes (#8878) 2022-11-18 17:59:32 +05:30
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Suresh Srinivas
b3472b03ea
Fixes #2254 Tag usage count is incorrect when a tag name is a prefix of another tag name (#8826) 2022-11-16 17:57:54 -08:00
Sriharsha Chintalapani
422eccf6e7
Fix #8738: Improve Search Relevancy (#8739) 2022-11-16 15:43:41 -08:00
Suresh Srinivas
45d45aab70
Fixes #8821 Inherit ownership from parent container when creating children (#8824) 2022-11-16 13:59:45 -08:00
Nahuel
1bde821e9a
On init the auth provider 'openmetadata' must generate a JWT token if not present on config (#8815)
* On init the auth provider 'openmetadata' must generate a JWT token if not present on config

* Address PR comments
2022-11-16 16:53:01 +01:00
Suresh Srinivas
6a0b89c27e
Fixes #8783 Support moving glossary terms in a nested fashion (#8784) 2022-11-15 19:39:25 -08:00
Suresh Srinivas
ea8cf999ad
Fixes #8701 Unable to add condition for matchAnyTags condition policy rule expression (#8782) 2022-11-15 17:20:18 -08:00
Suresh Srinivas
2edaec4155
Fixes #7569 - Adding Reviewers at the root of the Glossary should be applied to all the children of Glossary (#8740) 2022-11-15 12:52:15 -08:00
Mohit Yadav
771348a518
Fixes Backend : KPI patch API is not working #8774 (#8776)
* Endpoint should be more specific, causes issue in case we have enitities containing login, config etc in name

* Endpoint should be more specific, causes issue in case we have enitities containing login, config etc in name

* Change GET APIs to PUT apis #8687

* change request type for registration

* Data Insight Index should be created on the server start, if not exist

* checkstyle fix

* Fixes Backend : KPI patch API is not working #8774

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-11-15 20:14:36 +05:30
Mohit Yadav
2e04ff24f9
Fixes the index not found issue on data insight page in case workflow is not run or created (#8770)
* Endpoint should be more specific, causes issue in case we have enitities containing login, config etc in name

* Endpoint should be more specific, causes issue in case we have enitities containing login, config etc in name

* Change GET APIs to PUT apis #8687

* change request type for registration

* Data Insight Index should be created on the server start, if not exist

* checkstyle fix

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-11-15 18:02:03 +05:30
Nahuel
f57fe1d507
Fix#8577: Ingestion Pipelines sensitive passwords fields are stored in secrets store (#8725)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test

* Bots sensitive passwords fields are stored in secrets store

* Ingestion pipeline sensitive passwords fields are stored in secrets store

* Fix Java style

* Add update for ingestion pipelines on start up

* Add pylint exception

* Fix broken test

* Update UI references
2022-11-15 06:34:49 +01:00
Teddy
3dbaa69978
Data insight workflow (#8729) 2022-11-15 05:44:25 +01:00
Suresh Srinivas
6a0e326f59
Fixes #7524 Backend: Allow users to rename the Glossary name and Glossary term names (#8734) 2022-11-14 14:09:00 -08:00
Suresh Srinivas
9571326d92
Fixes #8730 Backend - Support renaming tag category and tag names (#8731) 2022-11-14 10:15:55 -08:00
Nahuel
83003a42da
Fix#8577: Bots sensitive passwords fields are stored in secrets store (#8720)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test

* Bots sensitive passwords fields are stored in secrets store

* Fix Java style
2022-11-14 16:46:05 +01:00
Mohit Yadav
db0215f964
Mapping from kpi to dichart(One to one) (#8707)
* Mapping from kpi to dichart(One to one)

* Mapping from kpi to dichart(One to one)

* Removed second tests creating KPI

* Fix some ui improve as well

* review comment changed function name

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-11-14 19:44:01 +05:30
Mohit Yadav
c88aac896b
Change GET APIs to PUT apis (#8693)
* Endpoint should be more specific, causes issue in case we have enitities containing login, config etc in name

* Endpoint should be more specific, causes issue in case we have enitities containing login, config etc in name

* Change GET APIs to PUT apis #8687

* change request type for registration

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-11-14 19:25:26 +05:30
Nahuel
e5abdc8c5c
Fix#8577: Services sensitive passwords fields are stored in secrets store (#8671)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test
2022-11-14 13:48:50 +01:00
Sriharsha Chintalapani
895a123f81
Add restore method for metadata resource (#8695) 2022-11-13 23:18:28 -08:00
Sriharsha Chintalapani
9e34cb8440
Fix #8416: Restore Entities after soft delete with all it's old data (#8635) 2022-11-13 10:16:01 -08:00
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Suresh Srinivas
6306c9ac8d
Fixes #8688 Enforce mutuallyExclusive tag labels (#8689) 2022-11-12 18:44:17 -08:00
Suresh Srinivas
1db6f1c8a0
Simplify EntityRepository store method (#8690) 2022-11-12 14:11:30 -08:00
Suresh Srinivas
4d15696d71
Fixes #8506 Build tag and glossary term cache and use it instead of database lookups in list operations (#8686) 2022-11-11 14:51:41 -08:00
Mohit Yadav
4dc24b904f
python fix circular import (#8670)
python fix circular import (#8670)
2022-11-11 17:03:43 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mohit Yadav
2dbd43a589
Added MetadataServices for creating DataInsight and ElasticSearch Reindex (#8565)
* Added SystemServices for creating DataInsight and ElasticSearch Reindexing Pipeling with Sink config

* Add Amundsen config

* Changed SystemService to MetadataService

* Change to MetadataConnection

* Fixed failing test

* Fixed Failing test

* Added Metadata Service Test

* Bootstrapped Openmetadat service

* checkstyle fix
2022-11-11 13:16:54 +05:30
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Mohit Yadav
b4e5f6ec13
Emailing Task and Test Notification (#8626)
* Emailing Task Notification [WIP]

* Emailing Test Result to Owners of table in case it is enabled
2022-11-10 14:22:44 +05:30
Mohit Yadav
5bd4e56b3f
Feat kpis (#8584)
* Added Kpi Resource

* Fix typo

* Fixed failing test

* Review Suggestions
2022-11-09 13:06:13 +05:30
Teddy
dcd0bbb566
Fixes 8470 -- Implements ES aggregation for web analytic data (#8566)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Added dailyActiveUsers aggregation

* Added page views entities aggregation and active users aggregation
2022-11-08 17:25:31 +01:00
Suresh Srinivas
292672a970
Clean up code warnings (#8532) 2022-11-04 11:24:46 -07:00
Mayur Singal
a49c261cbe
Add Queued Status in Ingestion Pipelines (#8529) 2022-11-04 11:19:50 +05:30
Shailesh Parmar
0f0489a7d4
Feat: Added pipeline as edge from ui (#8503)
* Feat: Added pipeline as edge from ui

* fixed failing unit test

* removed pipeline

* added unit test for util function
2022-11-03 17:46:49 +05:30
Teddy
f4cc2b57a7
Fixes #8353 - Implement Aggregation Endpoint for dataInsight (#8464)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* Fix pageViewEvent import issue

* Changed dataType to PERCENTAGE and NUMBER

* Cleaned up PR by removing metadata resource and ingestionPipeline sink logic

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByTier.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

* Update openmetadata-service/src/main/resources/json/data/dataInsight/totalEntitiesByType.json

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-11-03 14:51:14 +05:30
Mayur Singal
1386b43607
Fix #6141: Ingestion Pipeline Status Updates (#8216)
* Fix #6141: Ingestion Pipeline Status Updates

* List Pipeline Status API & Improvements

* Rename State field to PipelineState in UI

* Convert Pipeline Status array to single object

* fix braking UI

* Rebase Fixes

* Profiler, TestStuite & DataInsigts Pipeline

* py_format

* fix logs page not loading
add pipelineStatus endpoint

* fix recent run changes

* Fix Tests

* address review comments for ui

* fix failing checks

* fix unit tests

* fix cypress test

* remove loader test as not using it

* wait for API when we click on the tab

* fix cypress waitFor api

* fix failed cypress tests

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-03 14:37:26 +05:30
Suresh Srinivas
d47f5738cb
Add support for system provided Tags, Glossaries, Policies, and Roles #8484 (#8508) 2022-11-02 14:56:31 -07:00
Mayur Singal
fa68a1f18f
Fix #6754: Added User Details for Usage (#8449) 2022-11-02 14:07:41 +05:30
Suresh Srinivas
8a008022c8
Fixes #7004 Backend : conditionAllow should not be applicable to create operation (#8483) 2022-11-01 12:46:18 -07:00
Parth Panchal
af85b9629d
Fixed#8239: Query mismatch while searching for - and _ makes data overlap (#8476)
* Fixed#8239: Query mismatch while searching for - and _ makes data overlap

* Fixed#8239: Query mismatch while searching for - and _ makes data overlap
2022-11-01 18:20:03 +05:30
Suresh Srinivas
e7a4a7a3e5
Fixes #6681 Add support for policy evaluation API (#8466) 2022-10-31 15:54:32 -07:00
Parth Panchal
ca9c571179
Fixed#8354: Support Login For User from multiple devices with different refresh tokens (#8454)
* Fixed#8354: Support Login For User from multiple devices with different refresh tokens

* Fixed#8354: Support Login For User from multiple devices with different refresh tokens
2022-10-31 19:17:26 +05:30
Sajitha
a731b3fdfd
Fix sonar issues (#8421)
* Fix sonar issue

* Remove duplicate constant
2022-10-30 12:32:45 +01:00
Teddy
f027c26cc1
Remove authorizer for the collect endpoint (#8398) 2022-10-28 12:45:55 +02:00
Sajitha
debba08190
Fix sonar: Use try-with-resources and handle exception (#8414)
* Fix sonar: Use try-with-resources and handle exception

* Fix formatting
2022-10-28 13:04:54 +05:30
Suresh Srinivas
a6438c6347
Fixes #8401 Remove lingering authorizeAdmin in TagResource (#8403) 2022-10-27 15:37:44 -07:00
Mohit Yadav
6efe6df96b
[LDAP] Added Ldap-Authentication (#8256)
* [LDAP] Add Ldap-Authentication

* Refactor Basic to Service Layer

* added read login config

* pom updates

* review comments

* review comments

* move authenticator after authorizer

* merge conflicts resolve

* add init method , for initializing authenticator

* sonar fix
2022-10-27 22:56:20 +05:30