618 Commits

Author SHA1 Message Date
Chirag Madlani
c936ac3a41
feat(ui): Data observability and notifications (#14806)
* Centralize EventType

* Format better for readability

* Add Handling for Thread as Change Event

* Centralize logic for Entity Message Creation

* Add Thread Parent Message in Conversation

* Add Task and Announcement

* Support for multiple destinations

* Delete existing alerts for schema changes

* Delete all tables data for event subscription

* Centralize logic for Notifications and Observability

* Change Field Name and Remove redundant code

* Test Fixes

* Add more fields

* Add Optional

* Logical Test Case Addition Chnage Event

* Add Filter By Owner

* Fix Email Entity Url

* added data observability settings page

* localization changes for other languages

* Handler closure and cleanup

* complete create observibility flow

* fix searchIndex method error

* Update Entity Observability Filter

* working observability creation

* complete creation flow

* fix action with arguments

* fix form validations and form data formulation for 'input.actions' field

* localization changes for other languages

* Fixes for Data Observability

* Add mention notification

* Only apply mention to conversation

* fIx test case

* bug fixes for add and edit observability alert form functionality

* fIx test case

* add feature to config internal along with external destinations in alert form

* Fix Formatting

* Fix Url Redirection

* add observability alert details page

* localization changes for other languages

* remove filter and action required rules

* fix styling for observability form and details page

* add delete functionality on observability alert details page

* update view page layout and add a warning in configure destination step

* localization changes for other languages

* feat(ui): support notification with new flow (#14848)

* feat(ui): support notification with new flow

* fix notifications

* add missing filters for notification alert creation flow

* update the filter hiding logic to show at the start always

* add check to show Actions section in observability alert creation form

* code cleanup and improvements

* add breadcrumb to add notification alert page

* Add General Metadata Filters

* remove code smells and code cleanup for add notification page

* add new filter

* show activity feed alert on notifications list page

* add notification alert details page changes

* localization changes for other languages

* show icons for all the entities in the alert trigger dropdown

* redirect to notification alert listing page after deleting the notification alert

* change add alert page title

* add logic to disable all destinations except email for users

* fixed unit tests

* code improvements and cleanups

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2024-01-26 13:56:18 +05:30
Sachin Chaurasiya
12bc515aca
Update EntityLink.g4 with new entity types and fields (#14836) 2024-01-24 15:36:48 +05:30
07Himank
115870b2af
Pipeline and stored procedure lineage fix (#14681)
* pipeline and stored procedure lineage

* pipeline and stored procedure lineage

* fix: pass entity type in url

* fix unit tests

* pipeline and stored procedure lineage

* pipeline lineage

* pipeline and stored procedure lineage

* fix performance issues

* pipeline and stored procedure lineage done

* optimisation

* fix fullscreen mode

* add pipeline animation

* update loading state

* fix styling

* update pipeline status

* fixes

* fix cypress

* fix cypress

* fix cypress

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-01-24 11:04:31 +05:30
Chirag Madlani
45efaefc46
fix(ui): password error message for char limits (#14808)
* fix(ui): password error message for char limits

* fix java side code
2024-01-23 12:38:35 +05:30
Sriharsha Chintalapani
15eb094cff
Improve error response to the client. Added errorType so that client can handle better than depending on the http status codes (#14784) 2024-01-21 08:07:07 -08:00
Mayur Singal
4e02cb1c54
Fix #14764: Profiler make compute metric configurable (#14791) 2024-01-19 18:13:26 +05:30
Ayush Shah
831fce5b7e
Fixes 10709: Add useFqnForFiltering to profiler workflow (#14717) 2024-01-18 18:52:43 +05:30
Mohit Yadav
1e6c1a616b
Add Notification [WIP] (#14674)
* Centralize EventType

* Format better for readability

* Add Handling for Thread as Change Event

* Centralize logic for Entity Message Creation

* Add Thread Parent Message in Conversation

* Add Task and Announcement

* Support for multiple destinations

* Delete existing alerts for schema changes

* Delete all tables data for event subscription

* Centralize logic for Notifications and Observability

* Change Field Name and Remove redundant code

* Test Fixes

* Add more fields

* Add Optional

* Logical Test Case Addition Chnage Event

* Add Filter By Owner

* Fix Email Entity Url

* Handler closure and cleanup
2024-01-18 12:46:01 +05:30
Pere Miquel Brull
eadda0e3f1
#9269 - Add MSSQL Stored Procedure Support (#14739)
* Remove unnecessary field

* Remove unnecessary field

* Support query comments in MSSQL

* Remove unnecessary field

* Format

* Add external type

* Add MSSQL SP support
2024-01-17 06:37:27 +01:00
Sachin Chaurasiya
1a83a112ef
Minor: update ENTITY_TYPE and ENTITY_FIELD in EntityLink.g4 (#14732) 2024-01-16 20:08:03 +05:30
Onkar Ravgan
13b326d11c
Fix 14040: Part 3 Added updated field to status for patched entities (#14708)
* Added updated field to status

* Fixed e2e test
2024-01-16 17:53:05 +05:30
Ayush Shah
98eae7154a
Fix Airflow Source Url (#14712) 2024-01-15 12:31:43 +00:00
Sriharsha Chintalapani
49c1756583
MINOR - Test Case Ownership, keep severity, add test case result incidentId (#14691)
* MINOR - Test Case Ownership, keep severity, add test case result incidentId

* fix immutability

* Fix tests

* MINOR - Test Case Ownership, keep severity, add test case result incidentId

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-01-11 23:36:29 -08:00
Shiyang Xiao
241f3c68d7
Fixes #14413: Add SAS connector (#14415)
* feat: SAS Viya connector

* refactor SASCatalog to SAS

* add SAS logo to UI and connection documentation

* doc changes

* modify ingestion logic

* revert original changes

* added support for dataflow & perfect logic for reports/datatables

* add filter doc

* more updates to perfect ingestion for each asset type

* fix a bug with table lineage not created properly

* Delete ingestion/pipelines/sasCatalog.yaml

* precomit fix

* Conversion to database connector

* minor fixes

* make custom properties type generic

* Add SAS javaEnum

* add dummy variable for sas.yaml

---------

Co-authored-by: lizmc <liz.mcintosh@sas.com>
Co-authored-by: Shiyang Xiao <Shiyang.Xiao@sas.com>
2024-01-11 06:46:57 -08:00
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern (#14467) 2024-01-11 14:23:33 +05:30
Mayur Singal
190212c8ac
Fix #11556: Add support for Db2 for IBM i (#14680) 2024-01-11 12:35:52 +05:30
Mayur Singal
9288620c29
Minor: Fix Feature Tables of Connectors (#14658) 2024-01-10 12:48:42 +05:30
Mayur Singal
2832bbcc8c
Fix #14362: Spark Pipeline Connector & Docs (#14626) 2024-01-09 12:40:43 +05:30
Sriharsha Chintalapani
8f37d7da05
Fix #14610: includeEmptyTestSuite behavior causing not include newly created test suites (#14611)
* Fix #14610: includeEmptyTestSuite behavior causing not include newly created test suites

* Fix tests

* Fix tests

* Fix tests
2024-01-08 18:58:47 -08:00
Mohit Yadav
dad886eaba
Remove LMAX Disruptor , use queue and Quartz (#14422)
* Remove LMAX Disruptor , use queue and Quartz

* Spotless Fix

* Spotless formatting

* Review Comments

* Improvements

* Improvements to add Retry and DLQ

* Spotless fix

* Update Stats

* Fix stats

* Fix delete Failures

* Fix Failing Tests

* Fix Failing Tests

* Fix Failing Tests

* Postgres Fix

* Fix Schema Change for Change Event Table

* Failing test

* Fix Error for non ChangeDescriptions

* Adds Observability On Top of Event Subscription

* Make List Separate

* Add more filters for Test Suite

* Populate Default Schema Updates to Look for

* typo

* Add matchFieldChange and other in paramAdditionalContext

* Review Comments

* Spotless fix

* Merge issue

* Update Filter as per new Syntax

* Fix Failing Test
2024-01-09 00:13:45 +05:30
Sriharsha Chintalapani
b9781746ce
Fix #8496: Segregate ES indexes for use in shared infrastructure (#14606)
* Fix #8496: Segregate ES indexes for use in shared infrastructure

* Fix tests

* use entityType aggregation key for counts

* localisation

* Revert openmetadata.yaml changes

* Update sample_data.yaml

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-08 08:29:11 -08:00
Pere Miquel Brull
e8eb2bda3c
MINOR - Hide AutoTagger, add MetaPilot search, and fix apps status calls (#14201)
* MINOR - Hide AutoTagger & MetaPilot search & Fix apps status calls

* Fix method call

* Fix method call

* Fix method call

* "additionalProperties": true

* Use EntityRef

* Use ES sink

* Update UI schema type
2024-01-08 14:57:07 +01:00
Pere Miquel Brull
b92fd5cde4
MINOR - Move test case incident to the top of the data_quality_data_time_series (#14600)
* Add field and index

* MINOR - Move test case incident to the top of the data_quality_data_time_series table

* Fix test

* Fix compile

* Format

* Update incidentId

* Rename field

* Fix patch
2024-01-08 07:42:15 +01:00
Ayush Shah
d513dc4d4d
#11890: Add GCS Json schema changes (#14559)
* Issue-11890: Add GCS Json schema changes

* Modify Gcs Security config

* Add Test Def for gcs

* Add gcs for storage_metadata_config
2024-01-06 19:27:36 -08:00
Mayur Singal
a789fc86d6
Fix #13053: Remove Connection URI config MongoDB (#14584)
* Fix #13053: Remove Connection URI cofig MongoDB

* pyformat & test fixes
2024-01-05 10:51:12 -08:00
Pere Miquel Brull
0255171218
MINOR - Create Test Case Resolution ts entry & delete resolution when… (#14541)
* MINOR - Create Test Case Resolution ts entry & delete resolution when Test Case is deleted
2024-01-05 09:15:49 +01:00
Sriharsha Chintalapani
34ff582cf2
Allow user login using username and reassign roles based on users' ldap groups when login (#14550)
* Allow users to login with their user name in LDAP mode, and assign roles to user according to their LDAP group while login

* Fix #12503: Ldap feature: Allow user login using username and reassign roles based on users' ldap groups when login

---------

Co-authored-by: wentian <zb-wentian@kingmed.com.cn>
Co-authored-by: EricWent <135603556+EricWent@users.noreply.github.com>
2024-01-05 10:46:07 +05:30
Cristian Calugaru
1631172528
Issue 10587 - Storage Service doesn't handle deleted containers (#14497)
* global manifest option for storage services

* added a no metadata config source option for global manifest s3 services option

* merge fixes

* more merge fixes.

* black stuff

* test fixes

* formatting

* storage service container automatic deletion support

* reverted auto-modified files.

* styling

* linting
2023-12-30 13:27:16 -08:00
Pere Miquel Brull
1c2219517e
#13649 - Add Pipeline Status Skipped (#14524)
* #13649 - Add Pipeline Status Skipped

* Format
2023-12-29 09:41:59 +01:00
Teddy
2f679a9a48
FIXES #14294: return null if no aggregation value is present (#14514)
* fix: return null is no aggregation value is present

* fix: added Optional typing

* style: java linting

* fix: use Optional type hint

* style: java linting
2023-12-29 08:05:36 +01:00
Pere Miquel Brull
b84ce33b80
#11799 - Fix Airfow ownership & add pipeline tasks (#14510)
* Fix airflow owner and add tasks

* Add pipeline tasks ownership

* MINOR - Fix py CI

* Add pipeline tasks ownership

* Add pipeline tasks ownership

* MINOR - Fix py CI

* MINOR - Fix py CI

* Add pipeline tasks ownership

* patch team

* patch team

* Format
2023-12-28 10:25:00 -08:00
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows (#14472)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* style: renamed variables

* feat: added logic to compute failed/passed rows

* feat: add support for row level computation in schema

* chore: add test definition migration

* feat: add logic to explicitly compute row level failure

* chore: clean up code

* style: fix java

* style: fix pyton format

* fix: unhidde API for incident manager

* style: fix java styling
2023-12-27 13:38:51 +01:00
Saurabh Yadav
0691c96eaf
FIxes 13543: Added Duckdb Ingestion Connector (#14468)
* changes

* Added duck db ingestion connector
2023-12-25 18:07:09 -08:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status (#14462)
* Register StackTraceError in spec

* Register StackTraceError in spec

* Register StackTraceError in spec

* Add todos

* Update status

* docs

* format

* Fix tests

* Fix tests

* Fix tests

* Ignore generated

* Fix tests

* Fix tests

* Tests

* Try constants

* Try constants

* Print

* Print

* Print

* order

* Fix service name

* fix ui error

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Teddy
fe4c353bf3
Fixes #14331 - Add Test Case Incident Manager Migration (#14386)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* chore: clean up code

* style: fix java stylecheck
2023-12-22 11:01:51 +01:00
Ayush Shah
ebc0a551e5
Fixes 12947: Add Support For DQ and Profiler in Databricks Unity Catalog (#14424) 2023-12-20 21:18:05 +05:30
Ayush Shah
c3fac5e97e
Fix Adls connection typo (#14449) 2023-12-20 14:25:54 +01:00
NiharDoshi99
492092720d
#12908 support for table owner in postgres (#14289)
* support pick up postgres table owner

* added owner wrapper

* optimize the code

* remove pylint error

* minor fix

* changes as per comments

* fix python checksytyle and comments

* moved process owner to database service

* update docs for version 1.3

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-12-20 18:42:59 +05:30
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Ayush Shah
03f884b376
MINOR: Fix Typos (#14448) 2023-12-20 10:30:23 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Sriharsha Chintalapani
1294f93e15
Minor: Migrate to latest google code style library to support Java 17 and beyond (#14429)
* Minor: add spotless and use simplecontext

* Remove context from rule evaluaiton

* Fix EventSubscription tests

* Minor: Migrate to latest google code style library to support Java 17 and beyond

* Minor: Ignore code style migration from git blame
2023-12-18 12:56:17 -08:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
1af1caf065
Prepare alation connector schemas (#14407) 2023-12-18 13:07:42 +05:30
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Teddy
20bc238413
FIX #14248 - Implement test case resolution task workflow (#14323)
* feat: add test case resolution task workflow

* fix: conflict with main
2023-12-12 10:13:11 +01:00
Onkar Ravgan
d5ece87766
Fixed query millis duration desc (#14332) 2023-12-11 17:40:54 +05:30
Ayush Shah
b8d0b3b6e4
#14168: Mark SP and Data model deletion (#14219) 2023-12-06 08:57:41 +05:30
Onkar Ravgan
b3578cd496
Fix 14040: Part 2 Add Patch Request when updating the entities from create request (#14224) 2023-12-05 17:56:40 +05:30