415 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
2e95fcb98d
Fix #14786: Suggestions API (#14821)
* Fix #14786: Suggestions API

* Handle suggestions in ometa

* Minor: Optimise Databricks Client (#14776)

* MINOR - Fix SP topology context & Looker usage context (#14816)

* MINOR - Fix SP topology context & Looker usage context

* MINOR - Fix SP topology context & Looker usage context

* Fix tests

* Fixes #14598: Fix Tags / Labels ingestion on includeTags as False (#14782)

* fix(ui): password error message for char limits (#14808)

* fix(ui): password error message for char limits

* fix java side code

* Fixes #13556: Support for Salesforce table description ingestion (#14733)

* ISSUE-13556: Add suport for Salesforce table description ingestion

* ISSUE-13556: Remove unnecessary blank line

* ISSUE-13556: Fix to get description for each table

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

* MINOR - Better handling of Ingestion Pipeline Status (#14792)

* MINOR - Better handling of Ingestion Pipeline Status

* format

* format

* MINOR: Added table validation for cost analysis data (#14793)

* Added validation for cost analysis source

* centralized life cycle logic

* CYPRESS: simplify side navigation click in cypress (#14818)

* simplify side navigation click in cypress

* make sidbar item uses common enum

* fix cypress failure of outside import

* fix(#14326): tier dropdown is not working in advance search (#14780)

* improvement in advance search based on custom property

* fix a reading undefined property issue

* wip: advance search based on tier

* some code cleanup and improvement

* some fixes

* fix: ui flicker when advanceSearched is apply and refresh the page

* some cleanup

* no need to call customproperty api call, if entity not suppport customProperties

* minor change

* fix: autocomplete not working in tier search option in advance search modal

* added unit test for advance search provider component

* some cleanup

* added testcase for open modal

* added testcase for resetAllFilters method

* removed unwanted code

* added e2e test for testing tier advance search

* fix: e2e search flow for single field

* fix: string field not working after giving listValues in TierSearch

* fix: group query e2e test fix

* used asyncFetch way to get the tierOptions synchronously

* some cleanup

* remove unwanted lines

* some cleanup

* fix: selected option show option value instead of option title

* fix(minor): update skip icon for executions (#14809)

* Fixes #14803: ignore capitalization when confirming deletes  (#14804)

* ignore case when confirming deletes

* Test confirmation of deletes works when case differs 

Added test case for 'delete' as the confirmation text.

* minor(config): update openmetadata-ui code reviewers (#14823)

* Add Tests

* Add list/accept/reject apis

* initial ui changes

* localisation

* show suggestion for empty description

* ui feedbacks

* Fix permission check for entities without owner

* Fix entityLink and add tests

* Add update suggestion WIP

* Fix test

* Fix PUT and Pagination

* Fix styling

* update test

* Update status

* add OM server connection in apps

* add permissions check

* Fix CI

* Remove TODO

* Fix feedResourceTest

* fix unit tests

* add private configs for apps

* add private configs for apps

* fix update application icons

* minor center align icon

* add private configs for apps

* Format

* Fix pydantic gen

* Remove token

* Update name

* Rework private conf

* Fix apps

* Fix apps

* Format

* Format

* show metapilot only if its installed

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-31 18:51:09 -08:00
Mohit Yadav
270e76bdfb
[Fix] Migrate Ingestion pipeline for Data Insights to be of user provider type (#14942)
* Data Insight Report Link Fix

* Make the Data Insights Pipeline as User
2024-01-30 13:09:44 +05:30
Chirag Madlani
c936ac3a41
feat(ui): Data observability and notifications (#14806)
* Centralize EventType

* Format better for readability

* Add Handling for Thread as Change Event

* Centralize logic for Entity Message Creation

* Add Thread Parent Message in Conversation

* Add Task and Announcement

* Support for multiple destinations

* Delete existing alerts for schema changes

* Delete all tables data for event subscription

* Centralize logic for Notifications and Observability

* Change Field Name and Remove redundant code

* Test Fixes

* Add more fields

* Add Optional

* Logical Test Case Addition Chnage Event

* Add Filter By Owner

* Fix Email Entity Url

* added data observability settings page

* localization changes for other languages

* Handler closure and cleanup

* complete create observibility flow

* fix searchIndex method error

* Update Entity Observability Filter

* working observability creation

* complete creation flow

* fix action with arguments

* fix form validations and form data formulation for 'input.actions' field

* localization changes for other languages

* Fixes for Data Observability

* Add mention notification

* Only apply mention to conversation

* fIx test case

* bug fixes for add and edit observability alert form functionality

* fIx test case

* add feature to config internal along with external destinations in alert form

* Fix Formatting

* Fix Url Redirection

* add observability alert details page

* localization changes for other languages

* remove filter and action required rules

* fix styling for observability form and details page

* add delete functionality on observability alert details page

* update view page layout and add a warning in configure destination step

* localization changes for other languages

* feat(ui): support notification with new flow (#14848)

* feat(ui): support notification with new flow

* fix notifications

* add missing filters for notification alert creation flow

* update the filter hiding logic to show at the start always

* add check to show Actions section in observability alert creation form

* code cleanup and improvements

* add breadcrumb to add notification alert page

* Add General Metadata Filters

* remove code smells and code cleanup for add notification page

* add new filter

* show activity feed alert on notifications list page

* add notification alert details page changes

* localization changes for other languages

* show icons for all the entities in the alert trigger dropdown

* redirect to notification alert listing page after deleting the notification alert

* change add alert page title

* add logic to disable all destinations except email for users

* fixed unit tests

* code improvements and cleanups

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2024-01-26 13:56:18 +05:30
Mohit Yadav
1e6c1a616b
Add Notification [WIP] (#14674)
* Centralize EventType

* Format better for readability

* Add Handling for Thread as Change Event

* Centralize logic for Entity Message Creation

* Add Thread Parent Message in Conversation

* Add Task and Announcement

* Support for multiple destinations

* Delete existing alerts for schema changes

* Delete all tables data for event subscription

* Centralize logic for Notifications and Observability

* Change Field Name and Remove redundant code

* Test Fixes

* Add more fields

* Add Optional

* Logical Test Case Addition Chnage Event

* Add Filter By Owner

* Fix Email Entity Url

* Handler closure and cleanup
2024-01-18 12:46:01 +05:30
Pere Miquel Brull
eadda0e3f1
#9269 - Add MSSQL Stored Procedure Support (#14739)
* Remove unnecessary field

* Remove unnecessary field

* Support query comments in MSSQL

* Remove unnecessary field

* Format

* Add external type

* Add MSSQL SP support
2024-01-17 06:37:27 +01:00
Mohit Yadav
dad886eaba
Remove LMAX Disruptor , use queue and Quartz (#14422)
* Remove LMAX Disruptor , use queue and Quartz

* Spotless Fix

* Spotless formatting

* Review Comments

* Improvements

* Improvements to add Retry and DLQ

* Spotless fix

* Update Stats

* Fix stats

* Fix delete Failures

* Fix Failing Tests

* Fix Failing Tests

* Fix Failing Tests

* Postgres Fix

* Fix Schema Change for Change Event Table

* Failing test

* Fix Error for non ChangeDescriptions

* Adds Observability On Top of Event Subscription

* Make List Separate

* Add more filters for Test Suite

* Populate Default Schema Updates to Look for

* typo

* Add matchFieldChange and other in paramAdditionalContext

* Review Comments

* Spotless fix

* Merge issue

* Update Filter as per new Syntax

* Fix Failing Test
2024-01-09 00:13:45 +05:30
Pere Miquel Brull
b92fd5cde4
MINOR - Move test case incident to the top of the data_quality_data_time_series (#14600)
* Add field and index

* MINOR - Move test case incident to the top of the data_quality_data_time_series table

* Fix test

* Fix compile

* Format

* Update incidentId

* Rename field

* Fix patch
2024-01-08 07:42:15 +01:00
Mayur Singal
e67ffd5c04
Minor: Fix migration location for unity catalog (#14339) 2024-01-03 18:26:11 +05:30
Sriharsha Chintalapani
0303b44b9c
OpenMetadata Operations Command Line Utility (#14504)
* Add new OpenMetadataSetup command line application to migrate/deploy and re-index

* Add new OpenMetadataSetup command line application to migrate/deploy and re-index

* Add deployPipelines option

* Add reIndex option

* add subcommands

* add provision to store upgrade metrics

* rename bootstrap script

* fix styling checks

* Add changelog and store metrics into SERVER_CHANGE_LOG

* Cast jsonb

* Cast jsonb

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-12-29 08:22:58 +01:00
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows (#14472)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* style: renamed variables

* feat: added logic to compute failed/passed rows

* feat: add support for row level computation in schema

* chore: add test definition migration

* feat: add logic to explicitly compute row level failure

* chore: clean up code

* style: fix java

* style: fix pyton format

* fix: unhidde API for incident manager

* style: fix java styling
2023-12-27 13:38:51 +01:00
Sriharsha Chintalapani
4b7f4f43d6
Fix #13982: Fix userFQN encoding while creating mentions (#14496)
* Fix #13982: Fix userFQN encoding while creating mentions

* quote only teams or users
2023-12-25 17:28:13 -08:00
Teddy
fe4c353bf3
Fixes #14331 - Add Test Case Incident Manager Migration (#14386)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* chore: clean up code

* style: fix java stylecheck
2023-12-22 11:01:51 +01:00
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Pere Miquel Brull
ffe4aa319f
MINOR - Truncate automations_workflow (#14446) 2023-12-19 16:53:23 +01:00
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
karanh37
853a83f8b9
fix: allow config update for external app (#14034)
* fix: allow config update for external app

* Prep config

* Add configs

* fix: call configure app on saving of configurations

* fix: add autocomplete widget for json schema forms

* ORganize configuration handling

* Allow config

* Format

* Fix test

* Prep migrations for allowConfiguration

* TODO

* Add pg migrations

* fix: add allowConfiguration support

* Autotagger

* Update install flow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-27 16:55:33 -08:00
Teddy
cabc346a6b
feat: db migration for custom metrics (#14036) 2023-11-23 07:11:10 +01:00
Onkar Ravgan
274a3d1773
fixed viewparsing query (#14076) 2023-11-23 07:08:30 +01:00
Teddy
81242d6cb9
fix: comment in sql migration (#13979) 2023-11-15 10:32:11 +01:00
Mayur Singal
b34111ea00
ElasticSearch: Fix ES connection (#13919) 2023-11-13 14:10:11 +05:30
Onkar Ravgan
b7e7625071
Convert dbt test result timestamps to millis (#13903)
* Fixed dbt test res time to millis

* added postgres migration
2023-11-10 16:15:16 +05:30
Onkar Ravgan
2fdb1af0f4
added 121 migration (#13837) 2023-11-06 18:13:30 +05:30
Pere Miquel Brull
f7daad83da
Fix postgres migrations (#13743) 2023-10-27 17:12:53 +05:30
Onkar Ravgan
954b223927
fixed migration syntax domo (#13687) 2023-10-23 17:52:55 +02:00
Onkar Ravgan
0f0bccdd45
Converted and fixed pipelinestatus timestamps to milliseconds (#13670)
* fixed pipelinestatus timestamps in mills

* Added migrations
2023-10-20 09:39:24 -07:00
Pere Miquel Brull
8cf8720a9d
Clean Airflow Lineage Backend and migrate status to millis (#13666)
* Clean Airflow Lineage Backend and migrate status to millis

* Format

* chore(ui): update executions startTs and endTs to millis

* Remove lineage providers

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-10-20 15:42:38 +02:00
Pere Miquel Brull
255bfb95b1
Remove duplicates from entity_extension_time_series and add the const… (#13626)
* Remove duplicates from entity_extension_time_series and add the constraing if missing

* Add sort buffer and work mem

* Revert "Add sort buffer and work mem"

This reverts commit fcfff5feb60c9212bb7c1cad34b524dc8c03bfc5.
2023-10-19 12:15:02 +02:00
Sriharsha Chintalapani
f3adafd512
Fix extensions migrations path (#13541)
* Fix extensions migrations path

* Fix extensionPath for migrations

* Fix extensionPath for migrations

* Fix bootstrap echo
2023-10-12 17:17:29 +05:30
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Mohit Yadav
e577333b54
- Add Latest App Run Endpoint (#13511)
- Remove not required fields from app_extension table
2023-10-10 22:10:59 +05:30
Mohit Yadav
ab1a156b77
Build Applications Automations (#13318)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* Fix Stats Issue

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-10 11:32:37 +05:30
Mayur Singal
c0ababd8ad
Fix #13336: Clean Mark All Deleted Table Flag (#13344) 2023-10-06 16:04:54 +05:30
Sriharsha Chintalapani
762824665a
Fix #12655: Customizable Landing Page - Add Personas, DocStore to support KnowledgePanels and Pages (#13229)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* Add Multiple Persona support for a user

* add more elaborate tests for docstore

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix build
2023-09-29 19:13:56 -07:00
Sriharsha Chintalapani
3a7f7486d0
Add indexes to name column for efficient ORDER BY name queries (#13383) 2023-09-29 14:19:49 +02:00
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites (#13345)
* Prep v1.1.7 migrations to address test cases

* get or create executable suite

* Format

* Fix tests

* Add postgres
2023-09-27 11:49:21 +02:00
Sriharsha Chintalapani
c5a4c5957e
Add 1.1.6 migrations dir (#13305) 2023-09-22 09:45:00 +02:00
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts (#13247)
* Rename sandboxDomain to instanceDomain

* Test Get Charts in DomoDashboard

* Fix schemas

* Fix test

* Fix test

* Rename to Auto Tag PII

* Fix query test

* Fix query test

* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Pere Miquel Brull
22b0f44e38
Part of #12998 - Query Service & Lineage filter processed queries (#13215)
* Update mapping

* Prep

* prep

* Prep query migration

* Add query index deletion fix

* Docs and Maven CI fix

* Fix tests

* Add service filter

* Add query entity FQN col migration

* Fix lint

* supported serviceFQN in query api

* Prep repo

* Prep ES query search

* Do not recompute lineage

* Format

* Fix test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-09-19 07:37:47 +02:00
Sriharsha Chintalapani
047ab980cc
only add collation to hash columns (#13201) 2023-09-15 12:49:11 +05:30
Teddy
4012f25d64
fix: add test case result extension for ts migration (#13195) 2023-09-14 16:40:11 +02:00
Sriharsha Chintalapani
1af4941797
Minor fix: Do not re-run migration for properly formed native test su… (#13124)
* Minor fix: Do not re-run migration for properly formed native test suites

* rename 1.1.3 to 1.1.5

* rename 1.1.3 to 1.1.5 and add 1.1.4

* Fix table -> native testsuite migration
2023-09-12 11:43:10 -07:00
Pere Miquel Brull
f0995cbddc
Part of #12998 - Prep Stored Procedures Skeleton for Snowflake (#13121)
* Prep Stored Procedures Skeleton for Snowflake

* Update pylint and add migrations

* Fix test

* Reuse source url computation
2023-09-12 14:25:42 +02:00
Mohit Yadav
5f5aeef7f0
Transaction for Delete @UnitOfWork (#13115)
* Fix #12494: Jdbi Transactions opens a new connection per DAO causing recursive delete to not to rollback

* Fix #12494: Jdbi Transactions opens a new connection per DAO causing recursive delete to not to rollback

* rebase with main

* Fix styling

* Add jdbiUnitOfWork

* Improvements for Change Event Repository and missing Annotations

* Add connection commit/rollback

* push schemaChange fix

* Improvements for Change Event Repository and missing Annotations

* Pass wrapped collectionDAO everywhere

* Pass wrapped collectionDAO everywhere

* Separate event handlers transactions

* Fix checkstyle

* Wrap PUT , POST, and DELETE in transactions

* Add Patch for transactions

* Add primary key to tag.id

* Proxy internal @CreateSQLObjects

* Fix exception handling in ManagedHandleInvocationHandler

* Java typo

* Update schema files

* Checkstyle fix and conflicts resolve issue

* Remove @JdbiUnitOfWOrk from Feed Repository

* remove unnecessary @JdbiUnitOfWork annotation

* Test Failures fix

* Test Failures fix

* Increase wait time , changeEventDAO takes time due to handle.commit()

* commit change

* No need to use commit in ChangeEventHandler

* Add a lookup for avoiding transaction that are not started and should not be rolled back

* remove JdbiUnitOfWork.java

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-08 23:19:44 +05:30
Onkar Ravgan
d8e78a32d8
Added Life Cycle fields, API and ingestion integration (#13014)
* Added initial code

* temp commit

* Added snowflake changes

* Added postgres migration

* removed non req changes

* review comments one

* Added ometa tests

* lint fix

* fix lint naming

* fixed merge changes
2023-09-06 12:37:55 +02:00
Teddy
ab3042e8ee
Issue 12297 bis -- Delete Insert logic in the DI workflow (#13058)
* fix: updated ingestion to delete existing data for the ingestion day

* fix: added delete endpoint for the report data

* fix: added migration to delete duplicate data + added tables creation & data deltion logic in a transaction

* fix: made deletion SQL engine aware

* fix: added ES deletion logic back while we work on DI event publisher
2023-09-01 18:01:20 +02:00
Teddy
867e1c4946
Issue-12914 -- Split profiler, data quality and data insight table from entity_time_series_extension (#12991)
* fix: write testCase result testSuite to compute summary

* fix: empty commit to trigger CI

* fix: extracted report data out of entity_extension_time_series and into report_data_time_series

* fix: split profiler data  from entity_extension_time_series

* fix: split data quality data time series table

* fix: updated migration file
2023-08-28 08:14:15 +02:00
Teddy
101cd0ebac
Issue 8930 - Update profiler timestamp from seconds to milliseconds (#12948) 2023-08-25 08:47:16 +02:00