743 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
3c9d39e9af
Fix #9158: OpenSearch/ElasticSearch - Language based mappings (#9163)
* Add language for Search mappings

* Fix #9158: OpenSearch/ElasticSearch - Language based mappings

* Add analyzer to search in Japanese

* Adding JP es analyzer for all the indexes

* Fix review comments

* rebase with main

Co-authored-by: Megumi AIKAWA <megumi.aikawa@gmail.com>
2023-01-09 21:42:01 -08:00
Pere Miquel Brull
84348d4748
Fix #8866 - bump datamodel-codegen (#9623)
* Fix #8866 - bump datamodel-codegen

* Update connection options and arguments structure

* Add builders test

* Format

* Allow Any values in componentConfig

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-01-09 13:20:32 +01:00
Onkar Ravgan
e9ac8ecb85
Fix 7972: Added dbt tag classification name (#9575)
* Added dbt tag classification name

* resolved merge conflicts

* handle ui side change for dbt schema change

* Fixed migration scripts

* fixed failing cypress for dbt

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-01-06 11:08:21 +05:30
Mayur Singal
9ef32eef99
Fix #6829: Improve usage filtering (#9613) 2023-01-05 15:07:03 +00:00
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector (#9554)
* feat: update databricks_client

* feat: create pipelineconnection & connection

* test: add unit test

* feat: add examples file

* fix: remove unused imports

* style: fix python style

* fix: code review adjustments

* fix: remove unneeded schema ref

* fix: format requirements

* fix: change context calls

* test: add pipeline_status test

* format: pylint format change

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
Mayur Singal
0a34e18e26
Fix #9562: Add duration in usage ingestion (#9564) 2023-01-05 06:33:07 +00:00
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588)
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588)
2023-01-04 08:50:53 +01:00
Parth Panchal
dbf0152c85
Added maven publish git action workflow (#9568) 2023-01-03 11:18:29 +05:30
Suresh Srinivas
240920fa9f
Fixes #9118 - Glossary bulk upload support (#9524)
* Add CSV support for bulk edits

* Add CSV support for bulk edits - updates to the previous patch

* APIs for Glossary bulk export/import support
2022-12-29 09:58:06 -08:00
Chirag Madlani
bf6fc5f93a
prepare(release) next release (#9479)
* prepare(release) next release

* airflow typo

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-27 20:15:46 +05:30
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package (#9521)
Fix #6562 - Sources have their own package (#9521)
2022-12-27 15:00:22 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs (#9260) 2022-12-26 12:32:17 -08:00
Suresh Srinivas
c931ee3447
Fixes #9510 Enable authorization tests for all the entities (#9511) 2022-12-23 07:47:09 -08:00
Mohit Yadav
70d5536745
Added Test cases for Alerts + bug Fixes (#9421)
* Added Test cases for Alerts + bug Fixes

* removed commented code

* random bool

* added anno

* Disabling test for AlertAction and Alert + alertAction random name

* Bulk Create AlertAction Api
2022-12-21 15:34:18 +05:30
NiharDoshi99
9a3d599f30
Azure datalake minor changes (#9407) 2022-12-21 10:28:41 +05:30
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number (#9104) 2022-12-20 14:55:11 +05:30
Mohit Yadav
fa65480864
Alert Fixes, Removed Alert and AlertAction from Entity List, Get latest status from the data base, removed lastSuccesfulAt from failuredetails (#9386) 2022-12-18 23:39:27 +05:30
Mayur Singal
c06dbfd22e
Fix caching issue for BQ (#9352) 2022-12-17 09:16:16 -08:00
Sriharsha Chintalapani
0055ca2d1c
Fix #2435: Add tagging support for services and database (#9330)
* Fix #2435: Add tagging support for services and database

* Fix check style

* Fix check style

* Fix check style
2022-12-16 08:34:26 -08:00
Mohit Yadav
08f6536558
Alert Fixes : removed redundant files, updated Triggwe (#9346)
* Removed Redundant Code + Removed Unnecessary Triggers, Activity Feed Filterin

* all should have intersecting filters

* updated by should have entities index
2022-12-16 21:59:09 +05:30
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler (#9340)
* feat(Profiler): Added timeout logic in the profiler workflow

* feat(Profiler): Added additional input in workflow UI

* fix(Profiler): timeout text blob in UI

* fix(Profiler): Return profile_results outside the context manager

* Fix(Profiler): Added localizer for timeout label and message

* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py

Co-authored-by: Nahuel <nahuel@getcollate.io>

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Ashish Gupta
7c181a5a89
Dbt worflow (#9302) 2022-12-16 17:00:28 +05:30
Mohit Yadav
b762593652
Created ActivityFeed Alert (#9339) 2022-12-16 16:04:55 +05:30
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Suresh Srinivas
8c08cf73b5
Fixes #9290 Policies Don't Prevent Custom Properties Modification by Object Owner (#9327) 2022-12-15 20:37:51 -08:00
Mohit Yadav
ce51009eb7
Fixes for Alerts (#9316)
* Fixes for Alerts

* Fix addTestResult and Delete to updatedBy
2022-12-15 16:43:44 +00:00
Mohit Yadav
bfdc32c8c1
Alerts (#9267)
* Schema for Alerts #9248

* 1. Api for alerts
2. Api for alertAction

API for Alert and AlertAction #9266

* missing file

* fix ui failures

* Added Api to give function parameter type and additional Context for UI

* Added default Triggers in Alerts

* checkstyle fix

* docs update

* function names updated

* added timestamp for alert status update

* receiver should be unique in json schema and alerts publishers

* updated bootstrap filters

* updatated review comments

* added functions to match eventType for entities

* added migrations for alerts

* added alert config in alertACtionConfig

* Fix issue in sending mails

* removed commented part

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-12-15 00:14:10 +05:30
Sriharsha Chintalapani
38074f763b
Fix #8509: Add schema fields (#9209)
* Fix #8509: Add schema fields

* Fix #4675: Ingestion deployment from UI is broken

* Added sample data for topics

* Fixed pytests

* Address comments

* Refactored sampledata according to new schema

* Added return type

* Feat(ui)  : Add support for Avro editor (#9224)

* Feat(ui)  : Add support for Avro editor

* chore : add mock data for nested fields

* style: add group and opacity css

* add schema fields component to topic details and unit test

* Add locale keys

* Add support for edit description.

* refactor: expandableConfig

* test: add unit test for util method

* chore : make changes according to the schema change

* test : fix unit test and add new mock data

* chore : rename files

* Add row key

* chore : add default value for tags

* chore : update util method

* chore : add support for editing field tags

* chore : rename util files

* test : add unit test

* add comments

* addressing comments

* Address comments

* Added avro requirments

* Added requirement versions

* fixed versions

* protobuf version fix

* chore : rename util test file

* Fixed Dataype

* test: add unit test for schema component

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-14 19:56:37 +05:30
NiharDoshi99
b959709275
Fix: added datalake gen2 (#9245)
* Fix: added datalake gen2

* Fix: pytest

* Fix: pytest

* Fix: made changes to incoporate azure in datalake

* Fix: adlfs version

* Fix: adlfs version

* Fix: add tsv file type

* Fix: refractor for reading tsv file

* Fix: refractor for reading tsv file
2022-12-14 17:05:59 +05:30
Teddy
387cd570f7
Fixes 9192 - Add entity link in response (#9239)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch

* feat: Added entity URL for most viewed entity

* feat(dataInsight): Added href to entity

* feat(DataInsight): Added backend logic for entity href

* Fix conflict resolving typos

* Update openmetadata-service/src/main/resources/json/data/dataInsight/mostViewedEntities.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* fix: addressed comments from PR review

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-13 10:39:09 +00:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
2022-12-13 06:36:55 +01:00
Pere Miquel Brull
f48ede31f8
Fix #9205 - Fix CustomPipelineConnection java class (#9243)
* Fix CustomPipelineConnection java class

* Remove extra log
2022-12-12 16:59:12 +01:00
Mohit Yadav
775016488d
Schema for Alerts #9248 (#9249) 2022-12-12 16:25:35 +05:30
Teddy
4a5cb870ec
Fixes #9192 - Update ES response for DI (#9221)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch
2022-12-09 18:53:17 -08:00
Pere Miquel Brull
ba9109d318
Fix #9044 - Add prometheus event monitor (#9176)
* Add prometheus event monitor

* Format

* Format
2022-12-09 11:51:39 +01:00
Milan Bariya
6f12c971e1
Add: function to get different catalogs (#9179)
* Add: function to get different catalogs

* Change Based on comments

* Change Based on comments

* Change Based on comments
2022-12-08 15:42:58 +05:30
NiharDoshi99
96e77af8a4
Fix: #1308 Docs and Atlas Connector Improvement (#9072)
* Fix: docs changes

* Fix: changes as per comment

* Fix: atlas connector

* Fix: as per comments

* Fix: as per comments
2022-12-08 12:14:15 +05:30
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159)
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Suresh Srinivas
16d97d6f96
Fix typos in JSON schemas and follow standard naming conventions (#9183) 2022-12-06 20:50:34 -08:00
Teddy
f86186d5b0
Fixed time field type for profiler (#9177) 2022-12-06 18:30:48 +01:00
Aniket Katkar
1fd1862ce6
Fix(UI)#8970: Fixed issue with user created Metadata Services not able to be deleted (#9140)
* Fixed issue with user created Metadata Services not able to be deleted

* Fixed default values for Amundsen connection configs

* Added missing break statement in switch block
2022-12-05 20:06:51 +05:30
NiharDoshi99
e0a3f3481b
Fix: aggregate type (#9138) 2022-12-05 14:50:37 +05:30
Milan Bariya
2f9f169de9
Fix: Safer default Mark Deleted Tables (#9065)
* Fix: Safer default Mark Deleted Tables

* Fix: Change Based On Comments

* Fix: Change Based On Comments

* Add: Migration Files

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-01 16:26:43 +05:30
Sriharsha Chintalapani
80a4399c1a
add required fields for amundsen connection (#9039)
* add required fields for amundsen connection

* Remove unused modelClass field

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-11-30 13:09:57 -08:00
NiharDoshi99
34aacb20d6
Fix: atlas connector (#9034) 2022-11-29 23:55:05 +05:30
Pere Miquel Brull
dda8f8054e
Fix #8571 - Log IngestionPipeline status change to CloudWatch (#9013)
* Prep monitoring skeleton

* Fix naming

* Push Ingestion Pipeline status to cloudwatch

* Review comments and tests

* import

* Remove singleton object

* Format
2022-11-28 17:45:18 +01:00
Ashish Gupta
36f27e947d
UI : Mask the JWT token in Metadata service (#8842)
* Mask the Jwt token in Metadata service

* minor fix

* fix the icon alignment in Add Ingestion button

* disable the test connection for metadata service OpenMetadata type

* change the css name

* fix unit test issue

* Fix Auth Provider

* Fix add ingestion dropdown icon alignment

* Fix formatting

* Do not encryt JWT auth mechanism with secrets manager

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2022-11-22 14:17:40 +00:00
Pere Miquel Brull
f3ac6f53e6
Rename datatype (#8947) 2022-11-22 10:47:17 +00:00
Pere Miquel Brull
83ef3315c3
Fix query log parser (#8920) 2022-11-21 12:55:55 +00:00