1565 Commits

Author SHA1 Message Date
07Himank
29f49ee783
Ingestion pipeline by service type task (#10914)
* working on ingestion getList Issue

* filter ingestion pipeline by pipelineType

* revoked docker conf

* removed old changes and formatted

* .

* addressing comments

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
2023-04-04 17:03:57 +05:30
07Himank
4eb1ec81e1
Update query task for activity feed (#10895)
* solved change event query update for activity feed

* working on url config

* testing

* fix bug in slack alerts

* fixing alerts url

* fixed space issue url issue and edited the testcases for changeeventparser

* fixed email markers bug

* default value added

* removed my gmail config

* fix

* addressing comments

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2023-04-04 17:02:08 +05:30
Suresh Srinivas
c8b640674b
10041 part2 - Refactor and cleanup APIs (#10900)
* Use @Tag annotation to group APIs in the swagger documentation.

* Hide internal APIs

* Change API path events/subscription to events/subscriptions

* Change API path from automations/workflow to automations/workflows

* Change API path v1/testCase to v1/testCases

* Change API path v1/testDefinition to v1/testDefinitions

* Change API path v1/testSuite to v1/testSuites

* Rename Kpi and kpi in the documentation to KPI

* Change API path v1/testConnectionDefinition to v1/testConnectionDefinitions

* Update API section in the API documentation

* Fix test failures

* Correctly capitalize Test Cases and Test Suites in API docs
2023-04-03 13:03:48 -07:00
Teddy
2ed2e9f20a
Fix #8726 - Add permission to data insight (#10694)
* feat: added auth logic for reportData endpoint

* feat: added permission logic for dataInsightChart and dataReport

* feat: ran mvn clean for code linting

* feat: changed endpoint name to `/reports`

* fix: renamed analytics API endpoints

* fix: eslint

* fix: updated data insight test collection path

* fix: description and changed dataInsight to dataInsights
2023-04-03 14:16:27 +02:00
Suresh Srinivas
cdce2ca08c
Fixes #10241 - Include Status, Owner and Reviewer in Bulk Glossary Upload (#10881) 2023-04-02 15:04:00 -07:00
Suresh Srinivas
2759a70379
Fixes #10711 Cannot assign a policy to a team with a soft-deleted user (#10880) 2023-04-02 11:07:38 -07:00
Mayur Singal
752163ac71
Fix #10814: Improve parsing logic for union fields in topic (#10836) 2023-04-01 11:10:05 +05:30
Nahuel
0ca8c12596
Fix#10584: Update dashboard ES indexes for dataModels (#10851)
* Fix#10584: Update dashboard ES indexes for dataModels

* Address pylint error
2023-03-31 11:50:43 +02:00
Nahuel
6fa6b95f74
Fix: Remove bot configuration with SSO service account (#10847)
* Add deprecation warning for SSO config for bots

* Fix logging
2023-03-30 17:37:45 +00:00
Sriharsha Chintalapani
2febfe8b06
Add ngram to topic display name indexing (#10837) 2023-03-30 12:22:30 +02:00
Mayur Singal
54b635dd60
1.0.0-alpha version upgrade (#10826) 2023-03-29 23:24:39 +05:30
Sriharsha Chintalapani
744cac26c4
Fix #10174: Display Synonym in the Glossary Term List and include Syn…onyms in Glossary Search; Fix #10738: Search Filters needs to be consistent across entities. Example Topic and Container; Fix #10789: Search experience: improve weight on direct match for tableName and columnName (#10807)
* Fix #10174: Display Synonym in the Glossary Term List and include Synonyms in Glossary Search; Fix #10738: Search Filters needs to be consistent across entities. Example Topic and Container; Fix #10789: Search experience: improve weight on direct match for tableName and columnName

* Fix #10174: Display Synonym in the Glossary Term List and include Synonyms in Glossary Search; Fix #10738: Search Filters needs to be consistent across entities. Example Topic and Container; Fix #10789: Search experience: improve weight on direct match for tableName and columnName

* Fix #10174: Display Synonym in the Glossary Term List and include Synonyms in Glossary Search; Fix #10738: Search Filters needs to be consistent across entities. Example Topic and Container; Fix #10789: Search experience: improve weight on direct match for tableName and columnName

* feat: implement filters for topic containers and glossary

* fix: jest tests

* fix: cypress tests

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
2023-03-29 17:48:34 +05:30
Shailesh Parmar
e53a8c5859
feat #8096 integrated filter & full-screen mode for query (#10761)
* integreted filter for query

* refactor query filters

* fixed no placeholder issue

* add URL support for query entity

* translation-sync

* added query fullscreen mode page structure

* added fullscreen support

* added profiler picture for owner filter

* miner UI changes

* Edit query support and fix ES votes not appearing

* added unit test for new component

* updated owner style

* type fixed

* fix test

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-29 16:46:14 +05:30
Teddy
e1b193a719
Fixes #6795 - Implement profiler support for struct types (#10817)
* fix: raise more informative error message when service is not found

* fix: profiling for struct table

* fix: linting

* fix: added tests for struct and nestedt struct for get_columns
2023-03-29 10:06:34 +00:00
Suresh Srinivas
263da23220
Fixes #10791 Add tests for various Resource & Policy scenarios (#10792) 2023-03-28 18:30:53 -07:00
Pere Miquel Brull
4dad80e2fe
S3 Improvements for test connection and connector status (#10795)
* Handle test connection for S3 and new status

* Remove exception

* Update endpoint

* Update endpoint

* Update endpoint

* chore: remove toast messages for test connection

* chore: change objectstoreService to objectStoreService

* minor fix

* Use the default service param for listing filters

* Format

* Format

* address comments

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-03-28 17:52:47 +02:00
Mayur Singal
cc35749454
Minor Fixes for test connection (#10800) 2023-03-28 15:25:33 +00:00
Nahuel
bea38d7200
Fix#10584: Add Data Model as an entity (#10636)
* Add Data Model as entity

* Add sample_data + update dashboard resource and repository with data models

* Fix Java style

* Addess PR comments

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v009__create_db_connection_info.sql

* Pylint error

* Address PR comments

* Address PR comments

* Address PR comments

* Minor change

* Fix error in sample_data

* Fix failing test

* Add missing resource and event sub descriptors
2023-03-28 15:07:38 +00:00
Pere Miquel Brull
4853dfc6c8
Deprecate test_connection endpoint (#10798) 2023-03-28 15:52:36 +02:00
Parth Panchal
8f424e1787
Fixed#10565: Make Prometheus Mappings Configurable (#10671)
* Fixed#10565: Make Prometheus Mappings Configurable

* Fixed#10565: Make Prometheus Mappings Configurable

* Fixed#10565: Make Prometheus Mappings Configurable

* Addressing comments and fixed nullpointer in tests

* Fixed#10565: Make Prometheus Mappings Configurable

* Addressing comments

* Fixing tests
2023-03-28 12:51:52 +05:30
Pere Miquel Brull
78d7dd8789
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660)
* Prepare the new test connection ingestion logic

* Update test assert

* Update Test Connection for SQA Sources

* Correct return type and method doc

* Handle decryption

* Non SQA Database Sources

* Add the run_automation script in ingestion-base

* Dashboard Test Connection Changes

* Pipeline, Messagin, MlModel & Metadata Sources

* ui: test connect flow-1

* Unmask connection parameters before sending to Ariflow

* ui: test connect flow-2

* Address review comments and pylint

* pytest fix

* ui: test connect flow-3 (refactoring and style fix)

* ui: test connect flow-4 (fix test connection status logic)

* sync local file

* ui: test connect flow-5 (fix lowercase issue and styling)

* ui: test connect flow-5 (show toast notifications)

* test: add unit test

* ui: test connect flow-5 (update service page test connection button)

* Databrick fix & pytest fix

* pylint

* Update test

* Fix merge

* S3 Test connection

* add style for mandatory step

* sync locales

* chore: add service name in workflow request

* Unmask using original service connection parameters

* Fix test connection unmasking

* Wrap inspector function to eliminate error outside test conn

* Fix linting

* fix:cy test

* Fix linting

* address comment

* refactor and fix connection type casing issue

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
Mohit Yadav
13edb4ccdf
Return proper message in case invalid or missing token (#10788) 2023-03-27 21:56:39 +00:00
Mohit Yadav
bdb6a16430
Fix Query Used In (#10784) 2023-03-27 23:03:33 +05:30
Cristian Calugaru
3ffde9a293
Fixes #9064 - Ingestion for S3 bucket containers (#10515)
* Prep skeleton for object store connector

* First of of ingestion piepline for bucket containers.

* addressed PR review comments.

* swaped random with secrets due to security warning.

* linter issues.

* added prefix population for containers.

* more linting

* removed temp file.

* object store unit tests, PR comments and minor refactorings

* docs update and new TestConnection approach.

* linting

* removed isStructured field from containermeta schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-26 11:35:34 +02:00
Suresh Srinivas
ea8e1fda46
Fixes #10759 A Non-Privileged user can add new 'Roles' to Teams (#10760) 2023-03-24 19:23:22 -07:00
Mohit Yadav
11032a2d48
SAML feature (#10669)
* Added Saml Feature

* add localization

* fix tests

* added collate copyright

* remove thread.getType

* fixed keystore issues and updated review comments

* remove private key getter

* update test saml config

* update test saml config

* Update openmetadata-service/src/main/java/org/openmetadata/service/security/saml/OMMicrometerHttpFilter.java

Co-authored-by: Nahuel <nahuel@getcollate.io>

* review comments

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-24 21:56:49 +05:30
Mohit Yadav
fb02cbfeed
fix failing Py test due to backend change (#10749) 2023-03-24 17:41:52 +05:30
Sriharsha Chintalapani
d911e0caba
Add query index (#10682)
* Add query index

* add QueryUsage ChangeEvent

* fix test

* Add query index

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-03-23 21:26:27 -07:00
Suresh Srinivas
b80fc39632
Fixes #10740 Inherited ownership must not overwrite manually entered ownership (#10741) 2023-03-23 17:15:55 -07:00
Mohit Yadav
acb0e71a3e
Fix Email sent everytime a admin owns an Entity during startup (#10697)
* Fix Email sent everytime a admin owns an Entity during startup

* fix
2023-03-23 11:58:54 +05:30
Mayur Singal
bbce9c5aa4
Fix #10489: Handle unknown data types & store raw data type (#10563) 2023-03-23 11:41:29 +05:30
Pere Miquel Brull
e2a2bcc8da
Fix search by email index keyword (#10698)
* Fix search by email index keyword

* Fix search by email index keyword
2023-03-21 20:50:47 -07:00
Suresh Srinivas
b8e0ae489a
Fixes #10679 - Add policy functions inAnyTeam and hasAnyRole (#10680) 2023-03-21 07:24:41 -07:00
Teddy
64205cf231
feat: Added custom event event type for web analytics (#10554)
* feat: Added custom event event type for web analytics

* feat: renamed  to

* Update openmetadata-spec/src/main/resources/json/schema/analytics/webAnalyticEventType/customEvent.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* Update openmetadata-spec/src/main/resources/json/schema/analytics/webAnalyticEventType/customEvent.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* feat: move common type to basic.json

* feat: ran linting

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-21 10:12:15 +01:00
Mohit Yadav
06498bc011
Update alerts api (#10265)
* Refactored Alerts Api

* missing semicolon

* fixed Test Cases

* missing files

* Randomly run webhook in tests

* - update EventResource to ChangeEventResource

* - remove trigger config

* - missing ResourceDescriptors.json entity entry

* remove alertAction

* fix typos

* fix typos

* updated ui as per new alert API

* updated delete functionality and cypress

* fixed failing cypress test

* review comments

* Revert "- update EventResource to ChangeEventResource"

This reverts commit 90290cebd6c9572c021af3bde75eaf63c289b749.

* ChangeEventList -> EventList

* Change SQL Query

* add Subscription Status Builder

* fix java style

* update name back to EventSubscription

* Fixed Tests

* Fix import

* missing .

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-21 10:12:49 +05:30
Mohit Yadav
d5afb864ab
Fix Import errors (#10674) 2023-03-20 20:10:16 +05:30
Suresh Srinivas
6d717c6c33
Fixes #10656 Fix permissions API bug (#10657)
* Fixes #10656 Fix permissions API bug

* Remove parseTags method and add getAllTags to ContainerRepository

* Implement getAllTags for Topic
2023-03-20 04:42:03 -07:00
Mohit Yadav
55d5b4ed7c
Added Vote for Queries (#10655)
* added Vote for entities

* fix failing test

* fixed failing unit test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-20 13:58:20 +05:30
Suresh Srinivas
fc5cad915d
Move common functionality to EntityRepository store method (#10659) 2023-03-19 08:46:31 -07:00
Mohit Yadav
2e48ef0107
Personal Access Token Feature (#10555)
* Personal Access Token Feature

* migration swap

* Remove postgres

* remove unnecessary enum

* review comments updated

* Add back setting JWTTokenExpiry when generating a JWTAuthMechanism

* added test around Personal Access Token

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-03-17 16:22:41 +05:30
Parth Panchal
796fb8e768
Added ldap docker documentation (#10641) 2023-03-17 14:23:02 +05:30
Pere Miquel Brull
9b9cbd2d92
Update Storage Container list filter, add missing counts & rename workflow operations to automations (#10601)
* Rename Workflow package to automations

* Add list filter by root container & counts api

* Rename path

* Add glossary and glossaryTerm to count API

* Unify SqlQuery when there is no diff in mysql or psql

* chore: address feedback for object store service page

* Update glossary tests

* chore: add containers count on my data page

* fix: tag removal issue

* test: add unit test

* address comments

* address comment

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-03-16 23:34:53 +05:30
Sriharsha Chintalapani
b153b79723
Add users.isBot flag to index and add defaults to the user spec (#10558)
* Add users.isBot flag to index and add defaults to the user spec

* Added bot flag in user search api

* change as per comments

* fix cypress

* revert user.json changes

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-03-16 14:49:05 +05:30
Sriharsha Chintalapani
3c4b423f36
Fix patching tags for tasks (#10561)
* Fix patching tags for tasks

* Fix stylecheck
2023-03-15 23:44:47 -07:00
Sriharsha Chintalapani
0e9930c0ed
Add search api integration for object store (#10562)
* Add search api integration for object store

* Add search api integration for object store

* Addressing comments

* Fix stylecheck
2023-03-15 22:05:37 -07:00
Mohit Yadav
b982d3fe2b
Query as entity (#10449)
* added query as an entity

* changed name of the variables and methods

* Added Resource Descriptors

* testcase bug fix

* addressing comments

* added script for table query migration

* added script for table query migration postgresql

* bug fix

* db change for script test

* added current timestamp

* change db config from postgresql to mysql

* added extension to use fucntion gen_random_uuid()

* solving maven ci

* added queryUsage and change is migration script

* addressing comments

* addressing comments

* added queryUsage relation and testcase

* added api to insert queries in bulk

* .

* fix a test case which was failing due to latest changes

* Ingestion Changes for Query as Entity

* move query changes to latest sqls

* added tags and owner

* update PR for Query as Entity

* update type

* fixed pagination

* fix path param

* fix TestCases

* add validation criteria

* removed exisitng query apis

* checkstyle fix

* remove vote from put

* remove vote from put

* Query As Entity Ingestion Changes

* Remove unused func

* update Review Comments

* update Review Comments

* remove previous changes for Query and Update Tests

* moved Checksum to Query Util Class

* update python api

* fix python checkstyle

* Fixed Tests

* Fix pytest

* remove space changes

* remove space changes

* Fixed put_addFollowerDeleteEntity_200

* Fix usage ingestion

* Update Python SDK and tests

* pylint fix

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-15 20:55:30 -07:00
Pere Miquel Brull
d84a2b3ed7
Handle object store tags & Create abstract EntityUpdaterWithColumn (#10548)
* Handle object store tags

* Set major update in test

* Handle datamodel updates

* Rename column updater
2023-03-15 09:21:44 +01:00
Teddy
8acd077c23
Fix test suite pipeline and slack alerts (#10552)
* fix: change TestSuite to implement ServiceEntityInterface

* fix: updated slack alert URL for test suites

* added  as default service type
2023-03-15 07:52:32 +01:00
Megumi AIKAWA
48512ae334
add bindDN and bindPassword to args of LDAPConnection() (#10581) 2023-03-14 17:27:14 +01:00
Suresh Srinivas
ef97e04318
Fixes #10559 Remove individual resources & operations when All is specified in a policy rule (#10560) 2023-03-14 06:12:31 -07:00