618 Commits

Author SHA1 Message Date
Mohit Yadav
b762593652
Created ActivityFeed Alert (#9339) 2022-12-16 16:04:55 +05:30
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Suresh Srinivas
8c08cf73b5
Fixes #9290 Policies Don't Prevent Custom Properties Modification by Object Owner (#9327) 2022-12-15 20:37:51 -08:00
Mohit Yadav
ce51009eb7
Fixes for Alerts (#9316)
* Fixes for Alerts

* Fix addTestResult and Delete to updatedBy
2022-12-15 16:43:44 +00:00
Mohit Yadav
bfdc32c8c1
Alerts (#9267)
* Schema for Alerts #9248

* 1. Api for alerts
2. Api for alertAction

API for Alert and AlertAction #9266

* missing file

* fix ui failures

* Added Api to give function parameter type and additional Context for UI

* Added default Triggers in Alerts

* checkstyle fix

* docs update

* function names updated

* added timestamp for alert status update

* receiver should be unique in json schema and alerts publishers

* updated bootstrap filters

* updatated review comments

* added functions to match eventType for entities

* added migrations for alerts

* added alert config in alertACtionConfig

* Fix issue in sending mails

* removed commented part

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-12-15 00:14:10 +05:30
Sriharsha Chintalapani
38074f763b
Fix #8509: Add schema fields (#9209)
* Fix #8509: Add schema fields

* Fix #4675: Ingestion deployment from UI is broken

* Added sample data for topics

* Fixed pytests

* Address comments

* Refactored sampledata according to new schema

* Added return type

* Feat(ui)  : Add support for Avro editor (#9224)

* Feat(ui)  : Add support for Avro editor

* chore : add mock data for nested fields

* style: add group and opacity css

* add schema fields component to topic details and unit test

* Add locale keys

* Add support for edit description.

* refactor: expandableConfig

* test: add unit test for util method

* chore : make changes according to the schema change

* test : fix unit test and add new mock data

* chore : rename files

* Add row key

* chore : add default value for tags

* chore : update util method

* chore : add support for editing field tags

* chore : rename util files

* test : add unit test

* add comments

* addressing comments

* Address comments

* Added avro requirments

* Added requirement versions

* fixed versions

* protobuf version fix

* chore : rename util test file

* Fixed Dataype

* test: add unit test for schema component

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-14 19:56:37 +05:30
NiharDoshi99
b959709275
Fix: added datalake gen2 (#9245)
* Fix: added datalake gen2

* Fix: pytest

* Fix: pytest

* Fix: made changes to incoporate azure in datalake

* Fix: adlfs version

* Fix: adlfs version

* Fix: add tsv file type

* Fix: refractor for reading tsv file

* Fix: refractor for reading tsv file
2022-12-14 17:05:59 +05:30
Teddy
387cd570f7
Fixes 9192 - Add entity link in response (#9239)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch

* feat: Added entity URL for most viewed entity

* feat(dataInsight): Added href to entity

* feat(DataInsight): Added backend logic for entity href

* Fix conflict resolving typos

* Update openmetadata-service/src/main/resources/json/data/dataInsight/mostViewedEntities.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* fix: addressed comments from PR review

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-13 10:39:09 +00:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
2022-12-13 06:36:55 +01:00
Pere Miquel Brull
f48ede31f8
Fix #9205 - Fix CustomPipelineConnection java class (#9243)
* Fix CustomPipelineConnection java class

* Remove extra log
2022-12-12 16:59:12 +01:00
Mohit Yadav
775016488d
Schema for Alerts #9248 (#9249) 2022-12-12 16:25:35 +05:30
Teddy
4a5cb870ec
Fixes #9192 - Update ES response for DI (#9221)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch
2022-12-09 18:53:17 -08:00
Pere Miquel Brull
ba9109d318
Fix #9044 - Add prometheus event monitor (#9176)
* Add prometheus event monitor

* Format

* Format
2022-12-09 11:51:39 +01:00
Milan Bariya
6f12c971e1
Add: function to get different catalogs (#9179)
* Add: function to get different catalogs

* Change Based on comments

* Change Based on comments

* Change Based on comments
2022-12-08 15:42:58 +05:30
NiharDoshi99
96e77af8a4
Fix: #1308 Docs and Atlas Connector Improvement (#9072)
* Fix: docs changes

* Fix: changes as per comment

* Fix: atlas connector

* Fix: as per comments

* Fix: as per comments
2022-12-08 12:14:15 +05:30
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159)
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Suresh Srinivas
16d97d6f96
Fix typos in JSON schemas and follow standard naming conventions (#9183) 2022-12-06 20:50:34 -08:00
Teddy
f86186d5b0
Fixed time field type for profiler (#9177) 2022-12-06 18:30:48 +01:00
Aniket Katkar
1fd1862ce6
Fix(UI)#8970: Fixed issue with user created Metadata Services not able to be deleted (#9140)
* Fixed issue with user created Metadata Services not able to be deleted

* Fixed default values for Amundsen connection configs

* Added missing break statement in switch block
2022-12-05 20:06:51 +05:30
NiharDoshi99
e0a3f3481b
Fix: aggregate type (#9138) 2022-12-05 14:50:37 +05:30
Milan Bariya
2f9f169de9
Fix: Safer default Mark Deleted Tables (#9065)
* Fix: Safer default Mark Deleted Tables

* Fix: Change Based On Comments

* Fix: Change Based On Comments

* Add: Migration Files

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-01 16:26:43 +05:30
Sriharsha Chintalapani
80a4399c1a
add required fields for amundsen connection (#9039)
* add required fields for amundsen connection

* Remove unused modelClass field

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-11-30 13:09:57 -08:00
NiharDoshi99
34aacb20d6
Fix: atlas connector (#9034) 2022-11-29 23:55:05 +05:30
Pere Miquel Brull
dda8f8054e
Fix #8571 - Log IngestionPipeline status change to CloudWatch (#9013)
* Prep monitoring skeleton

* Fix naming

* Push Ingestion Pipeline status to cloudwatch

* Review comments and tests

* import

* Remove singleton object

* Format
2022-11-28 17:45:18 +01:00
Ashish Gupta
36f27e947d
UI : Mask the JWT token in Metadata service (#8842)
* Mask the Jwt token in Metadata service

* minor fix

* fix the icon alignment in Add Ingestion button

* disable the test connection for metadata service OpenMetadata type

* change the css name

* fix unit test issue

* Fix Auth Provider

* Fix add ingestion dropdown icon alignment

* Fix formatting

* Do not encryt JWT auth mechanism with secrets manager

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2022-11-22 14:17:40 +00:00
Pere Miquel Brull
f3ac6f53e6
Rename datatype (#8947) 2022-11-22 10:47:17 +00:00
Pere Miquel Brull
83ef3315c3
Fix query log parser (#8920) 2022-11-21 12:55:55 +00:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change (#8909) 2022-11-21 12:35:23 +01:00
Mohit Yadav
8e57787ad4
Fix Url For Email Verification etc (#8904)
* Fix Url For Email Verification etc

* yaml fix
2022-11-21 09:01:05 +00:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Pere Miquel Brull
a23196d2fa
Centralise Component Config definition (#8863)
* Centralise Component Config def

* Move def to basic
2022-11-18 10:12:46 +01:00
Nahuel
2a77dd0ceb
Add secrets manager documentation back (#8771)
* Add secrets manager documentation back

* Change OM version
2022-11-17 15:38:06 +01:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Miguel Sempere
8c67e0f627
Include links to consumer and registry configurations (#8107) 2022-11-16 19:57:02 -08:00
NiharDoshi99
d783869ed6
Fix: Added metadata service (#8759)
* Fix: Added metadata service

* Fix as per comment

* Fix changes for py-test

* Fix changes for py-test

* Fix py-checkstyle
2022-11-16 06:41:54 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler (#8694)
* DQ commit

* Add DL Profiler

* Fix Ingestion and Profliing pylint checks

* Fix Tests

* PyFormat files

* Fix Tests

* Resolve Comments

* Fix Tests and Format Files

* Resolve Comments

* Fix Pylint and Code smells

* Resolve Comments

* Fix S3 parquet

* Fix Metrics Code Smell
2022-11-15 16:01:10 +01:00
Nahuel
f57fe1d507
Fix#8577: Ingestion Pipelines sensitive passwords fields are stored in secrets store (#8725)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test

* Bots sensitive passwords fields are stored in secrets store

* Ingestion pipeline sensitive passwords fields are stored in secrets store

* Fix Java style

* Add update for ingestion pipelines on start up

* Add pylint exception

* Fix broken test

* Update UI references
2022-11-15 06:34:49 +01:00
Suresh Srinivas
6a0e326f59
Fixes #7524 Backend: Allow users to rename the Glossary name and Glossary term names (#8734) 2022-11-14 14:09:00 -08:00
Pere Miquel Brull
f33003485e
ES reindex Airflow workflow (#8723)
* Added support for 'Metadata Service' connectors

* Fixed failing unit tests

* Removed delete button for 'Metadata Service' connector of type OpenMetadataServer

* Code optimization for SelectServiceType component

* Fixed errors arrived due to backend changes related to OpenMetadata Services

* init reindex

* Added additional step for adding Metadata to ES configs while creating ingestion pipeline for Metadata Service

* Add ES reindex

* Add ES reindex

* Format

* Format

* Rename service

* Pick service name from IngestionPipeline

* fix ui conflicts

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-14 18:59:56 +01:00
Sriharsha Chintalapani
895a123f81
Add restore method for metadata resource (#8695) 2022-11-13 23:18:28 -08:00
Sriharsha Chintalapani
9e34cb8440
Fix #8416: Restore Entities after soft delete with all it's old data (#8635) 2022-11-13 10:16:01 -08:00
Mohit Yadav
a20945e13f
Added pipelineTypes (#8692)
* Added SystemServices for creating DataInsight and ElasticSearch Reindexing Pipeling with Sink config

* Add Amundsen config

* Changed SystemService to MetadataService

* Change to MetadataConnection

* Fixed failing test

* Fixed Failing test

* Added Metadata Service Test

* Bootstrapped Openmetadat service

* checkstyle fix

* circular import fix

* circular import fix

* Fix FileNames Issue causing pytest failures and mismatch in connection name due to Java Class

* add dataInsight Pipeline

* Add DataInsight + elasticSearchReindexing Pipeline Types
2022-11-13 21:41:41 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Suresh Srinivas
6306c9ac8d
Fixes #8688 Enforce mutuallyExclusive tag labels (#8689) 2022-11-12 18:44:17 -08:00
Mohit Yadav
4dc24b904f
python fix circular import (#8670)
python fix circular import (#8670)
2022-11-11 17:03:43 +01:00
NiharDoshi99
6da01b13a9
Changed format for s3 staging (#8632) 2022-11-11 12:15:37 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mohit Yadav
2dbd43a589
Added MetadataServices for creating DataInsight and ElasticSearch Reindex (#8565)
* Added SystemServices for creating DataInsight and ElasticSearch Reindexing Pipeling with Sink config

* Add Amundsen config

* Changed SystemService to MetadataService

* Change to MetadataConnection

* Fixed failing test

* Fixed Failing test

* Added Metadata Service Test

* Bootstrapped Openmetadat service

* checkstyle fix
2022-11-11 13:16:54 +05:30
Teddy
9d594efcb5
Fixes #8473 - Implements KPI Workflow (#8657)
* Added KPI workflow logic

* Update ingestion/src/metadata/data_insight/runner/run_result_registry.py

* ran py format

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-11 06:35:33 +01:00
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00