156 Commits

Author SHA1 Message Date
Suresh Srinivas
89bd68cdf6
Fixes #9829 Add owner field to GlossaryTerm (#9833) 2023-01-20 12:20:25 -08:00
Mayur Singal
939adf887b
Fix #8973: Superset fetch metadata from DB (#9645)
* Docs #8973: Superset with SSO docs

* Superset support for mysql & postgres db

* remove unwanted field from yaml

* Fix pylint

* Fix superset tests

* Fix sample data connection

* ci fix & review comments

* cypress & defualt provider fix
2023-01-19 16:39:43 +01:00
Pere Miquel Brull
b0058bdb65
Fix #9707 - Install instantclient and enable thick mode (#9761)
* Install instantclient

* Prep instant client

* Update docs

* Amend comments
2023-01-19 08:08:56 +01:00
Ayush Shah
02147b6502
Fix: Quicksight Dashboard and chart Metadata Ingestion (#9786)
* Fix Quicksight json schema and errors

* Fix Quicksight lineage

* add pagination

* Generalize Pagination

* Fix Pylints

* update max results

* fix tests

* Add Pydantic Models for QuickSight resp

* Add fqn builder

* Fix Py lInt

* Fix Tests
2023-01-18 20:37:41 +05:30
NiharDoshi99
f2649041f2
Fix: atlas test connection (#9791)
* Fix: atlas test connection

* Fix: docs changes

* Fix: docs changes

* Fix: python checkstyle

* Fix: python test
2023-01-18 20:16:07 +05:30
Abhishek Pandey
6d7acdc140
Issue 9542 - Replace the default name for MySQL db (#9687)
* database-field-added-for-custom-database-name

* database-field-added-for-custom-database-name

* database field description-added

* databaseName-field-updated-in-connection-json

* database-name-field-updated-in-salesforce

* databaseName-description-updated

* database-name-logic-updated-in-common-db-source
2023-01-18 14:28:40 +01:00
Onkar Ravgan
08a4d33c5f
Powerbi pagination fix (#9784)
* Fixed powerbi pagination logic

* Added yaml changes
2023-01-18 17:17:13 +05:30
Suresh Srinivas
ac157a8f0b
Fixes #9618 Allow apostrophe in entity names (#9642)
* Fixes #9618 Allow apostrophe in entity names

* Add support for space in entity names (with exception) and column names

* Add regex checks to entity names, table column names to allow only required characters

* fix: allowed characters in entityName

* Add all the allowed special characters in the entity name in tests

* Allow '&', '/', ':' only for the table

* Override basic entityName by defining entity specific entityName for Python code gen

* fix: dagster migration + json schema path

* fix: python style check

* Fix test failure

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-01-17 11:34:29 -08:00
Suresh Srinivas
b6e7bcfee5
Fixes #9721 Add support for count of terms in Glossary and Classification (#9733) 2023-01-13 11:40:24 -08:00
Corey Zuares
c3197beaf4
Extra headers for metadata API (#9321)
* Configurable Extra Authorization Header

- Adds two new header that can be set via env-var
  (`OMETA_HEADER_EXTRA_AUTH_NAME` and `OMETA_HEADER_EXTRA_AUTH_VALUE`)
- By default, it copies the Authorization header value to the extra auth header
- If a value is specified in the corresponding env-var, that value is
  used instead of the existing token

The primary use case is for Google Cloud's Identity Aware Proxy (IAP) which allows secure access to private services but removes the Authorization header in the process. The `Proxy-Authorization` header gets forwarded along to the service as the `Authorization` header for the internal service.

* Use openMetadataConnection schema instead of env vars

* Disable too-many-locals for _request function

* Fix dict merge
2023-01-13 09:59:44 +01:00
Milan Bariya
bb41fec08f
Add: Function for Postgres Tags (#9683)
* Add: Function for Postgres Tags

* Update: Query for generate fqn
2023-01-12 09:46:08 +01:00
Teddy
41ce2406a4
fixes #9673 - Oracle casting error (#9682)
* fix (profiler): oracle profiler datatype error

* fix (profiler): linting

* fix: reorder connection type to default to db schema
2023-01-11 19:46:10 +01:00
Mohit Yadav
ea0e0e03e4
Alert Test Fix (#9640)
* Alert Test Fix

* Removed Db calls in case local publishers available

* Fix test

* Removed unrequired code

* Maintaing entityType to class map

* checkstyle
2023-01-10 12:56:10 +05:30
Sriharsha Chintalapani
3c9d39e9af
Fix #9158: OpenSearch/ElasticSearch - Language based mappings (#9163)
* Add language for Search mappings

* Fix #9158: OpenSearch/ElasticSearch - Language based mappings

* Add analyzer to search in Japanese

* Adding JP es analyzer for all the indexes

* Fix review comments

* rebase with main

Co-authored-by: Megumi AIKAWA <megumi.aikawa@gmail.com>
2023-01-09 21:42:01 -08:00
Pere Miquel Brull
84348d4748
Fix #8866 - bump datamodel-codegen (#9623)
* Fix #8866 - bump datamodel-codegen

* Update connection options and arguments structure

* Add builders test

* Format

* Allow Any values in componentConfig

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-01-09 13:20:32 +01:00
Onkar Ravgan
e9ac8ecb85
Fix 7972: Added dbt tag classification name (#9575)
* Added dbt tag classification name

* resolved merge conflicts

* handle ui side change for dbt schema change

* Fixed migration scripts

* fixed failing cypress for dbt

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-01-06 11:08:21 +05:30
Mayur Singal
9ef32eef99
Fix #6829: Improve usage filtering (#9613) 2023-01-05 15:07:03 +00:00
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector (#9554)
* feat: update databricks_client

* feat: create pipelineconnection & connection

* test: add unit test

* feat: add examples file

* fix: remove unused imports

* style: fix python style

* fix: code review adjustments

* fix: remove unneeded schema ref

* fix: format requirements

* fix: change context calls

* test: add pipeline_status test

* format: pylint format change

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
Mayur Singal
0a34e18e26
Fix #9562: Add duration in usage ingestion (#9564) 2023-01-05 06:33:07 +00:00
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588)
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588)
2023-01-04 08:50:53 +01:00
Parth Panchal
dbf0152c85
Added maven publish git action workflow (#9568) 2023-01-03 11:18:29 +05:30
Suresh Srinivas
240920fa9f
Fixes #9118 - Glossary bulk upload support (#9524)
* Add CSV support for bulk edits

* Add CSV support for bulk edits - updates to the previous patch

* APIs for Glossary bulk export/import support
2022-12-29 09:58:06 -08:00
Chirag Madlani
bf6fc5f93a
prepare(release) next release (#9479)
* prepare(release) next release

* airflow typo

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-27 20:15:46 +05:30
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package (#9521)
Fix #6562 - Sources have their own package (#9521)
2022-12-27 15:00:22 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs (#9260) 2022-12-26 12:32:17 -08:00
Suresh Srinivas
c931ee3447
Fixes #9510 Enable authorization tests for all the entities (#9511) 2022-12-23 07:47:09 -08:00
Mohit Yadav
70d5536745
Added Test cases for Alerts + bug Fixes (#9421)
* Added Test cases for Alerts + bug Fixes

* removed commented code

* random bool

* added anno

* Disabling test for AlertAction and Alert + alertAction random name

* Bulk Create AlertAction Api
2022-12-21 15:34:18 +05:30
NiharDoshi99
9a3d599f30
Azure datalake minor changes (#9407) 2022-12-21 10:28:41 +05:30
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number (#9104) 2022-12-20 14:55:11 +05:30
Mohit Yadav
fa65480864
Alert Fixes, Removed Alert and AlertAction from Entity List, Get latest status from the data base, removed lastSuccesfulAt from failuredetails (#9386) 2022-12-18 23:39:27 +05:30
Mayur Singal
c06dbfd22e
Fix caching issue for BQ (#9352) 2022-12-17 09:16:16 -08:00
Sriharsha Chintalapani
0055ca2d1c
Fix #2435: Add tagging support for services and database (#9330)
* Fix #2435: Add tagging support for services and database

* Fix check style

* Fix check style

* Fix check style
2022-12-16 08:34:26 -08:00
Mohit Yadav
08f6536558
Alert Fixes : removed redundant files, updated Triggwe (#9346)
* Removed Redundant Code + Removed Unnecessary Triggers, Activity Feed Filterin

* all should have intersecting filters

* updated by should have entities index
2022-12-16 21:59:09 +05:30
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler (#9340)
* feat(Profiler): Added timeout logic in the profiler workflow

* feat(Profiler): Added additional input in workflow UI

* fix(Profiler): timeout text blob in UI

* fix(Profiler): Return profile_results outside the context manager

* Fix(Profiler): Added localizer for timeout label and message

* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py

Co-authored-by: Nahuel <nahuel@getcollate.io>

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Ashish Gupta
7c181a5a89
Dbt worflow (#9302) 2022-12-16 17:00:28 +05:30
Mohit Yadav
b762593652
Created ActivityFeed Alert (#9339) 2022-12-16 16:04:55 +05:30
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Suresh Srinivas
8c08cf73b5
Fixes #9290 Policies Don't Prevent Custom Properties Modification by Object Owner (#9327) 2022-12-15 20:37:51 -08:00
Mohit Yadav
ce51009eb7
Fixes for Alerts (#9316)
* Fixes for Alerts

* Fix addTestResult and Delete to updatedBy
2022-12-15 16:43:44 +00:00
Mohit Yadav
bfdc32c8c1
Alerts (#9267)
* Schema for Alerts #9248

* 1. Api for alerts
2. Api for alertAction

API for Alert and AlertAction #9266

* missing file

* fix ui failures

* Added Api to give function parameter type and additional Context for UI

* Added default Triggers in Alerts

* checkstyle fix

* docs update

* function names updated

* added timestamp for alert status update

* receiver should be unique in json schema and alerts publishers

* updated bootstrap filters

* updatated review comments

* added functions to match eventType for entities

* added migrations for alerts

* added alert config in alertACtionConfig

* Fix issue in sending mails

* removed commented part

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-12-15 00:14:10 +05:30
Sriharsha Chintalapani
38074f763b
Fix #8509: Add schema fields (#9209)
* Fix #8509: Add schema fields

* Fix #4675: Ingestion deployment from UI is broken

* Added sample data for topics

* Fixed pytests

* Address comments

* Refactored sampledata according to new schema

* Added return type

* Feat(ui)  : Add support for Avro editor (#9224)

* Feat(ui)  : Add support for Avro editor

* chore : add mock data for nested fields

* style: add group and opacity css

* add schema fields component to topic details and unit test

* Add locale keys

* Add support for edit description.

* refactor: expandableConfig

* test: add unit test for util method

* chore : make changes according to the schema change

* test : fix unit test and add new mock data

* chore : rename files

* Add row key

* chore : add default value for tags

* chore : update util method

* chore : add support for editing field tags

* chore : rename util files

* test : add unit test

* add comments

* addressing comments

* Address comments

* Added avro requirments

* Added requirement versions

* fixed versions

* protobuf version fix

* chore : rename util test file

* Fixed Dataype

* test: add unit test for schema component

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-14 19:56:37 +05:30
NiharDoshi99
b959709275
Fix: added datalake gen2 (#9245)
* Fix: added datalake gen2

* Fix: pytest

* Fix: pytest

* Fix: made changes to incoporate azure in datalake

* Fix: adlfs version

* Fix: adlfs version

* Fix: add tsv file type

* Fix: refractor for reading tsv file

* Fix: refractor for reading tsv file
2022-12-14 17:05:59 +05:30
Teddy
387cd570f7
Fixes 9192 - Add entity link in response (#9239)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch

* feat: Added entity URL for most viewed entity

* feat(dataInsight): Added href to entity

* feat(DataInsight): Added backend logic for entity href

* Fix conflict resolving typos

* Update openmetadata-service/src/main/resources/json/data/dataInsight/mostViewedEntities.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* fix: addressed comments from PR review

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-13 10:39:09 +00:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
2022-12-13 06:36:55 +01:00
Pere Miquel Brull
f48ede31f8
Fix #9205 - Fix CustomPipelineConnection java class (#9243)
* Fix CustomPipelineConnection java class

* Remove extra log
2022-12-12 16:59:12 +01:00
Mohit Yadav
775016488d
Schema for Alerts #9248 (#9249) 2022-12-12 16:25:35 +05:30
Teddy
4a5cb870ec
Fixes #9192 - Update ES response for DI (#9221)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch
2022-12-09 18:53:17 -08:00
Pere Miquel Brull
ba9109d318
Fix #9044 - Add prometheus event monitor (#9176)
* Add prometheus event monitor

* Format

* Format
2022-12-09 11:51:39 +01:00
Milan Bariya
6f12c971e1
Add: function to get different catalogs (#9179)
* Add: function to get different catalogs

* Change Based on comments

* Change Based on comments

* Change Based on comments
2022-12-08 15:42:58 +05:30