2200 Commits

Author SHA1 Message Date
Pere Miquel Brull
4dad80e2fe
S3 Improvements for test connection and connector status (#10795)
* Handle test connection for S3 and new status

* Remove exception

* Update endpoint

* Update endpoint

* Update endpoint

* chore: remove toast messages for test connection

* chore: change objectstoreService to objectStoreService

* minor fix

* Use the default service param for listing filters

* Format

* Format

* address comments

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-03-28 17:52:47 +02:00
Mayur Singal
cc35749454
Minor Fixes for test connection (#10800) 2023-03-28 15:25:33 +00:00
Nahuel
bea38d7200
Fix#10584: Add Data Model as an entity (#10636)
* Add Data Model as entity

* Add sample_data + update dashboard resource and repository with data models

* Fix Java style

* Addess PR comments

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v009__create_db_connection_info.sql

* Pylint error

* Address PR comments

* Address PR comments

* Address PR comments

* Minor change

* Fix error in sample_data

* Fix failing test

* Add missing resource and event sub descriptors
2023-03-28 15:07:38 +00:00
Pere Miquel Brull
4853dfc6c8
Deprecate test_connection endpoint (#10798) 2023-03-28 15:52:36 +02:00
Parth Panchal
8f424e1787
Fixed#10565: Make Prometheus Mappings Configurable (#10671)
* Fixed#10565: Make Prometheus Mappings Configurable

* Fixed#10565: Make Prometheus Mappings Configurable

* Fixed#10565: Make Prometheus Mappings Configurable

* Addressing comments and fixed nullpointer in tests

* Fixed#10565: Make Prometheus Mappings Configurable

* Addressing comments

* Fixing tests
2023-03-28 12:51:52 +05:30
Pere Miquel Brull
78d7dd8789
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660)
* Prepare the new test connection ingestion logic

* Update test assert

* Update Test Connection for SQA Sources

* Correct return type and method doc

* Handle decryption

* Non SQA Database Sources

* Add the run_automation script in ingestion-base

* Dashboard Test Connection Changes

* Pipeline, Messagin, MlModel & Metadata Sources

* ui: test connect flow-1

* Unmask connection parameters before sending to Ariflow

* ui: test connect flow-2

* Address review comments and pylint

* pytest fix

* ui: test connect flow-3 (refactoring and style fix)

* ui: test connect flow-4 (fix test connection status logic)

* sync local file

* ui: test connect flow-5 (fix lowercase issue and styling)

* ui: test connect flow-5 (show toast notifications)

* test: add unit test

* ui: test connect flow-5 (update service page test connection button)

* Databrick fix & pytest fix

* pylint

* Update test

* Fix merge

* S3 Test connection

* add style for mandatory step

* sync locales

* chore: add service name in workflow request

* Unmask using original service connection parameters

* Fix test connection unmasking

* Wrap inspector function to eliminate error outside test conn

* Fix linting

* fix:cy test

* Fix linting

* address comment

* refactor and fix connection type casing issue

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
Mohit Yadav
13edb4ccdf
Return proper message in case invalid or missing token (#10788) 2023-03-27 21:56:39 +00:00
Mohit Yadav
bdb6a16430
Fix Query Used In (#10784) 2023-03-27 23:03:33 +05:30
Cristian Calugaru
3ffde9a293
Fixes #9064 - Ingestion for S3 bucket containers (#10515)
* Prep skeleton for object store connector

* First of of ingestion piepline for bucket containers.

* addressed PR review comments.

* swaped random with secrets due to security warning.

* linter issues.

* added prefix population for containers.

* more linting

* removed temp file.

* object store unit tests, PR comments and minor refactorings

* docs update and new TestConnection approach.

* linting

* removed isStructured field from containermeta schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-26 11:35:34 +02:00
Suresh Srinivas
ea8e1fda46
Fixes #10759 A Non-Privileged user can add new 'Roles' to Teams (#10760) 2023-03-24 19:23:22 -07:00
Mohit Yadav
11032a2d48
SAML feature (#10669)
* Added Saml Feature

* add localization

* fix tests

* added collate copyright

* remove thread.getType

* fixed keystore issues and updated review comments

* remove private key getter

* update test saml config

* update test saml config

* Update openmetadata-service/src/main/java/org/openmetadata/service/security/saml/OMMicrometerHttpFilter.java

Co-authored-by: Nahuel <nahuel@getcollate.io>

* review comments

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-24 21:56:49 +05:30
Mohit Yadav
fb02cbfeed
fix failing Py test due to backend change (#10749) 2023-03-24 17:41:52 +05:30
Sriharsha Chintalapani
d911e0caba
Add query index (#10682)
* Add query index

* add QueryUsage ChangeEvent

* fix test

* Add query index

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-03-23 21:26:27 -07:00
Suresh Srinivas
b80fc39632
Fixes #10740 Inherited ownership must not overwrite manually entered ownership (#10741) 2023-03-23 17:15:55 -07:00
Mohit Yadav
acb0e71a3e
Fix Email sent everytime a admin owns an Entity during startup (#10697)
* Fix Email sent everytime a admin owns an Entity during startup

* fix
2023-03-23 11:58:54 +05:30
Mayur Singal
bbce9c5aa4
Fix #10489: Handle unknown data types & store raw data type (#10563) 2023-03-23 11:41:29 +05:30
Pere Miquel Brull
e2a2bcc8da
Fix search by email index keyword (#10698)
* Fix search by email index keyword

* Fix search by email index keyword
2023-03-21 20:50:47 -07:00
Suresh Srinivas
b8e0ae489a
Fixes #10679 - Add policy functions inAnyTeam and hasAnyRole (#10680) 2023-03-21 07:24:41 -07:00
Teddy
64205cf231
feat: Added custom event event type for web analytics (#10554)
* feat: Added custom event event type for web analytics

* feat: renamed  to

* Update openmetadata-spec/src/main/resources/json/schema/analytics/webAnalyticEventType/customEvent.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* Update openmetadata-spec/src/main/resources/json/schema/analytics/webAnalyticEventType/customEvent.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* feat: move common type to basic.json

* feat: ran linting

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-21 10:12:15 +01:00
Mohit Yadav
06498bc011
Update alerts api (#10265)
* Refactored Alerts Api

* missing semicolon

* fixed Test Cases

* missing files

* Randomly run webhook in tests

* - update EventResource to ChangeEventResource

* - remove trigger config

* - missing ResourceDescriptors.json entity entry

* remove alertAction

* fix typos

* fix typos

* updated ui as per new alert API

* updated delete functionality and cypress

* fixed failing cypress test

* review comments

* Revert "- update EventResource to ChangeEventResource"

This reverts commit 90290cebd6c9572c021af3bde75eaf63c289b749.

* ChangeEventList -> EventList

* Change SQL Query

* add Subscription Status Builder

* fix java style

* update name back to EventSubscription

* Fixed Tests

* Fix import

* missing .

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-21 10:12:49 +05:30
Mohit Yadav
d5afb864ab
Fix Import errors (#10674) 2023-03-20 20:10:16 +05:30
Suresh Srinivas
6d717c6c33
Fixes #10656 Fix permissions API bug (#10657)
* Fixes #10656 Fix permissions API bug

* Remove parseTags method and add getAllTags to ContainerRepository

* Implement getAllTags for Topic
2023-03-20 04:42:03 -07:00
Mohit Yadav
55d5b4ed7c
Added Vote for Queries (#10655)
* added Vote for entities

* fix failing test

* fixed failing unit test

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-03-20 13:58:20 +05:30
Suresh Srinivas
fc5cad915d
Move common functionality to EntityRepository store method (#10659) 2023-03-19 08:46:31 -07:00
Mohit Yadav
2e48ef0107
Personal Access Token Feature (#10555)
* Personal Access Token Feature

* migration swap

* Remove postgres

* remove unnecessary enum

* review comments updated

* Add back setting JWTTokenExpiry when generating a JWTAuthMechanism

* added test around Personal Access Token

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-03-17 16:22:41 +05:30
Parth Panchal
796fb8e768
Added ldap docker documentation (#10641) 2023-03-17 14:23:02 +05:30
Pere Miquel Brull
9b9cbd2d92
Update Storage Container list filter, add missing counts & rename workflow operations to automations (#10601)
* Rename Workflow package to automations

* Add list filter by root container & counts api

* Rename path

* Add glossary and glossaryTerm to count API

* Unify SqlQuery when there is no diff in mysql or psql

* chore: address feedback for object store service page

* Update glossary tests

* chore: add containers count on my data page

* fix: tag removal issue

* test: add unit test

* address comments

* address comment

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-03-16 23:34:53 +05:30
Sriharsha Chintalapani
b153b79723
Add users.isBot flag to index and add defaults to the user spec (#10558)
* Add users.isBot flag to index and add defaults to the user spec

* Added bot flag in user search api

* change as per comments

* fix cypress

* revert user.json changes

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-03-16 14:49:05 +05:30
Sriharsha Chintalapani
3c4b423f36
Fix patching tags for tasks (#10561)
* Fix patching tags for tasks

* Fix stylecheck
2023-03-15 23:44:47 -07:00
Sriharsha Chintalapani
0e9930c0ed
Add search api integration for object store (#10562)
* Add search api integration for object store

* Add search api integration for object store

* Addressing comments

* Fix stylecheck
2023-03-15 22:05:37 -07:00
Mohit Yadav
b982d3fe2b
Query as entity (#10449)
* added query as an entity

* changed name of the variables and methods

* Added Resource Descriptors

* testcase bug fix

* addressing comments

* added script for table query migration

* added script for table query migration postgresql

* bug fix

* db change for script test

* added current timestamp

* change db config from postgresql to mysql

* added extension to use fucntion gen_random_uuid()

* solving maven ci

* added queryUsage and change is migration script

* addressing comments

* addressing comments

* added queryUsage relation and testcase

* added api to insert queries in bulk

* .

* fix a test case which was failing due to latest changes

* Ingestion Changes for Query as Entity

* move query changes to latest sqls

* added tags and owner

* update PR for Query as Entity

* update type

* fixed pagination

* fix path param

* fix TestCases

* add validation criteria

* removed exisitng query apis

* checkstyle fix

* remove vote from put

* remove vote from put

* Query As Entity Ingestion Changes

* Remove unused func

* update Review Comments

* update Review Comments

* remove previous changes for Query and Update Tests

* moved Checksum to Query Util Class

* update python api

* fix python checkstyle

* Fixed Tests

* Fix pytest

* remove space changes

* remove space changes

* Fixed put_addFollowerDeleteEntity_200

* Fix usage ingestion

* Update Python SDK and tests

* pylint fix

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-15 20:55:30 -07:00
Pere Miquel Brull
d84a2b3ed7
Handle object store tags & Create abstract EntityUpdaterWithColumn (#10548)
* Handle object store tags

* Set major update in test

* Handle datamodel updates

* Rename column updater
2023-03-15 09:21:44 +01:00
Teddy
8acd077c23
Fix test suite pipeline and slack alerts (#10552)
* fix: change TestSuite to implement ServiceEntityInterface

* fix: updated slack alert URL for test suites

* added  as default service type
2023-03-15 07:52:32 +01:00
Megumi AIKAWA
48512ae334
add bindDN and bindPassword to args of LDAPConnection() (#10581) 2023-03-14 17:27:14 +01:00
Suresh Srinivas
ef97e04318
Fixes #10559 Remove individual resources & operations when All is specified in a policy rule (#10560) 2023-03-14 06:12:31 -07:00
Mohit Yadav
26efcb1197
Add Indexing Mapping Language in Reindexing (#10537) 2023-03-14 10:28:56 +05:30
Ashish Gupta
c0fe8ec871
UI : Added tags at schema Level (#10471)
* Added tags at schema Level

* Add task handling for Database Schema

* added cypress and unit test

* Changes as per comments

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-03-13 16:54:35 +00:00
Ayush Shah
0f9c2c2164
Add Labels on DatabaseSchema Level (#10547) 2023-03-13 14:13:51 +00:00
Milan Bariya
bc4abc44ed
Add steps in test_connection function (#10408)
* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Add steps in test_connection funstion

* Fix: Pylint issue

* Status response changefor statuscode 200

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-03-10 10:02:26 +05:30
Pere Miquel Brull
81dec813a0
Don't store the OM connection in the Ingestion Pipeline or Workflow (#10448)
* Do not store OM connection

* Migration to remove the server connection

* Update tests

* Add workflow masking and secrets manager

* Fix failing test

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-03-09 17:32:40 +01:00
Sriharsha Chintalapani
d9e4fbdebb
Fix #10454: Improve Search Relevancy, by adding functional scoring an… (#10455)
* Fix #10454: Improve Search Relevancy, by adding functional scoring and add ngram analyzer; Fix #10452: Enable Table and Column search by BOTH name and displayName

* fix stylecheck

* Undo changes in table example names

* remove ngram from teams & users

* Fix topic tags

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf (#10430)

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Fix #10429: Kafka Sample data improvements and adding support for JSONSchema and Protobuf

* Added top level parsing and unit tests

* fix(ui): show schemaText and fields both

* fix no data placeholder for fields & schema text

* addressing comments

* fixed py checkstyle

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>

* revert common_broker_source changes

* revert common_broker_source changes

* remove changes to user & team indexes

* fix team index

* fix glossary & tag index

* Fix to TopicIndex

* fix advance search pre-requisites cypress failure

* fix group advance search cy failures

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-09 21:37:08 +05:30
NiharDoshi99
ddbc0311e8
fix: mark table deleted dependencies (#10476) 2023-03-09 09:15:05 +00:00
Suresh Srinivas
4c6d184ef5
Fixes #10480 Glossary rename results in rename of Classification with… (#10486)
* Fixes #10480 Glossary rename results in rename of Classification with the same name

* Rename TagSource Tag to Classification
2023-03-09 00:30:36 -08:00
Sriharsha Chintalapani
b42f457d88
Fix error code if else in webhooks (#10386) 2023-03-07 12:58:19 -08:00
Pere Miquel Brull
050da1e2d1
Add service type to container (#10441) 2023-03-06 14:44:30 +01:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Mohit Yadav
df2001418e
add tags to schema Level (#10381)
=
2023-03-06 12:45:07 +05:30
Pere Miquel Brull
1e3652d35e
Fix #10382 - Test Connection Definitions, Results and Resource (#10383)
* Prep the skeleton

* Add DAO and tables

* Add Test Connection Definition Resource

* Format

* Add ResourceDescriptors & missing props

* Simplify APIs to scope for GET and LIST
2023-03-05 10:17:08 +01:00
Nahuel
247016307d
Fix#8648: Mask sensitive info from API responses (#10307)
* Mask sensitive info from API responses

* Rename converter classes

* Add missing Java classes from JSON schemas and class converters

* Update test service connection schema

* Update datalakeConnection JSON schema and fix some tests

* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh

* Fix Pipeline and Database service tests

* Minor refactor

* Fix CsvUtilTest

* Fix EventMonitorFactoryTest

* Fix CloudWatchEventMonitorTest

* Update datalake metadata

* Update bigquery metadata

* Fix test connection functionality

* Fix OMeta service api test

* Update gcsValues title and revert changes in GH actions

* Mask sensitive enabled by default for local docker

* Add missing tests

* Address PR comments

* Address PR comments

* fix ui breaks on gcsValues.json

* Address PR comments

* Minor refactor

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-03-03 18:10:01 +00:00
Sachin Chaurasiya
4ce038557e
feat(ui)#10274: UI - allow to configure OM logo (#10316)
* feat(ui)#10274: UI - allow to configure OM logo

* chore: use logo config to render the brand and login page logo

* fix : cy tests

* doc: update the doc

* chore: remove check for filePath logo location type

* test: add unit test

* - removed FilePath
- Added configs to docker

* address comment

* remove config from test yaml

* update logo aspect ratio

* remove type

* chore: update ui logic as per backend config

* update doc

* address comments

* Revert "address comments"

This reverts commit c4f9da88918343cab08cd5e603cba29bd05b2ec9.

* address comments

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-03-01 14:09:44 +05:30