717 Commits

Author SHA1 Message Date
Pere Miquel Brull
060a9d7f68
MINOR - Prepare AI chat feed (#15510)
* MINOR - Prepare AI chat feed

* Rename ai to chatbot

* Rename ai to chatbot
2024-03-18 10:29:52 +01:00
Trs
4db9b775ea
#14169: Support external_account type for GCP Auth (#14166) 2024-03-16 19:59:02 +05:30
IceS2
51e3d7a466
FIXES 15215: First draft implementation on extracting metadata incrementally. Done for Snowflake, BigQuery and Redshift (#15201)
* Initial incremental implementation for snowflake

* Initial unit test refactor for snowflake

* Fix linter complaints

* Propagate change on abstract create method

* Add missing argument to create

* Polish Snowflake incremental extraction

* Fix linters and make enabled required

* Initial proposal for incremental bigquery extraction

* BigQuery incremental tests

* Remove debugging override

* Fix linters

* Remove unused query

* Initial Redshift Incremental Extraction

* Add Incremental Extraction documentation

* Move the default to False

* Improve code based on sonarcloud input

* Apply suggestions

* Fix wrong path

* Change timestamp to be time aware as per sonar

* Move documentation to 1.4

* Move documentation to 1.4

* Fix linters

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-03-15 14:00:49 +01:00
Mayur Singal
88ab7475e7
MINOR: Restructure dbServiceName field in dashboard and pipeline (#15548) 2024-03-15 12:42:47 +05:30
Mayur Singal
b643206bba
Fix #11905: Automated lineage between external table and container snowflake (#15537) 2024-03-15 00:52:41 +05:30
Mayur Singal
8a2ee00fe3
Fix #15432: make sample data external storage path configurable (#15478) 2024-03-12 15:02:28 +05:30
mgorsk1
98850ab5cc
feat: OpenLineage integration (#15317)
* 🎉 Init OpenLineage connector

Co-authored-by: dechoma <dominik.choma@gmail.com>

* MLH - make linter happy

* review fixes

* 🐛 Fix path for ol event in tests

* 🐛 Fix path for ol event in tests

* Update ingestion/setup.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* Update ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

* review fixes 2

* linter

* review

* review

* make linter happy

* fix test_yield_pipeline_lineage_details test

* make linter happy

* fix tests

* fix tests 2

---------

Co-authored-by: dechoma <dominik.choma@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-03-12 08:39:25 +01:00
Imri Paran
4bb5550a46
MINOR: add the systemApp property (#15473)
* feat: add "systemApp" property

* use BAD_REQUEST for System Apps uninstall

* use IllegalArgumentException
2024-03-11 15:39:20 +01:00
Mohit Yadav
9491e04088
- Fix Poll Interval (#15453)
* - Fix Poll Interval

* make poll interval as 1

* - Add random tests
- Disable Tests for Subscription for Query

* - Checkstyle Fix

* - Remove ondemand jobs as well
- On update remove already init instance

* - use alias

* - typo

* - Search Index Fix
2024-03-06 02:21:28 +05:30
Aniket Katkar
110e7a4f4b
Fix #15438: Update filters and destination for 'Conversation' source in notification alert (#15443)
* Add 'Mentions' enum for subscriptionCategory

* fix the user list selection field not showing for 'Mentioned Users' filter

* update the cypress for new changes

* fix teams not showing in the mention suggestions
2024-03-04 21:40:31 +05:30
Sriharsha Chintalapani
cecbf80a2d
Add Custom Propety Config to store format, enum values, entity types (#15302)
* Add Custom Propety Config to store format, enum values, entity types

* Fix import statements and remove unused code

* Add Custom Propety Config to store format, enum values, entity types

* Add support for enum field type in custom properties

* update name in customPropertyConfigTypeValueField

* add custom property config column in custom property table

* Update padding-left in block-editor.less

* Add enum value translation for multiple languages

* update placeholder of config

* fixed python sdk

* add enum type in property value

* add unit tests

* Add Custom Propety Config to store format, enum values, entity types

* update ui to handle the enum config and validation

* Fix enum value handling in EditCustomPropertyModal and PropertyValue

* Update CustomProperty.md with enum values and multi-select option

* add cypress test

* add cypress for multiselect enum value

* Add tests for enum props

* add cypress for editing the enum property

* Add validations to enum

* Fix dependency issue

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2024-02-29 14:36:24 +05:30
Onkar Ravgan
1fc2c7f974
MINOR: Part 1 of #15090: dbt JSON Schema & Parsing Improvements (#15297) 2024-02-29 10:41:21 +05:30
Teddy
056e6368d0
Issue #14765 - Preparatory Work (#15312)
* refactor!: change partition metadata structure for table entities

* refactor!: updated json schema for TypeScript code gen

* chore: migration of partition for table entities

* style: python & java linting

* updated ui side change for table partitioned key

* miner fix

* addressing comments

* fixed ci error

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-02-28 07:11:00 +01:00
Aniket Katkar
0a38334c98
Minor: Add edit alert form item improvements (#15346)
* add "Assignees" option in subscription category enum

* add Assignee option for task alert source destination options

* change alert form section names from "Trigger" to "Source" and "Actions" to "Trigger"

* localization changes for other languages

* rename the trigger item components

* rename observability alert form actions item component to a proper name

* only show "Owners" and "Assignees" options for "Task" source

* change all alert trigger references to source

* change all alert action item references to trigger

* fix failing unit test

* add cypress coverage for task source

* convert alert specs to .ts files

* update the changed key value for other languages
2024-02-28 11:13:13 +05:30
Ashish Gupta
1efbeb1204
MINOR: made ui enhancement in setting page (#15269)
* made ui enhancement in setting page

* ui fixes

* fix ui render in application and other setting page

* fix unit test and user profile page issues

* added loader in team selectable in user profile page

* changes as per comments

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-02-23 14:18:01 +05:30
Imri Paran
18c22c4178
Fixes #10013: Implement first stage of NoSQL profiler (#15189)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format
2024-02-22 11:46:19 +01:00
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Mayur Singal
64e3cedeea
MINOR: Improve Kafka Test Connection (#15207) 2024-02-19 10:11:52 +05:30
Shin-ichi Hashiba
2120b38672
fix-typo: description of default batchsize (#15206) 2024-02-16 08:10:44 +00:00
Teddy
e72c80f264
MINOR - fix test definition deletion (#15203)
* fix: added provider for testDefinition entities

* fix: changed testDefinition <-> testCase relationship + allow recursive deletion of testDefinition

* fix: added migration steps

* style: ran java linting
2024-02-15 18:22:26 +00:00
Supan Shah
4915d6884b
#14881: PinotDB Docs (#14882)
* ISSUE-14881: PinotDB Test Connection Success

* Update ingestion/src/metadata/ingestion/source/database/pinotdb/connection.py

* Update ingestion/src/metadata/ingestion/source/database/pinotdb/connection.py

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-02-13 09:29:23 +01:00
Imri Paran
aeb5fbe303
fixes #12591: add BigTable (#15122)
* feat(connector): add BigTable

* bigtable work

1. docstrings
2. tests
3. created a Row BaseModel
4. implemented a ClassConverter

* docs moved to separate PR

* format files

* minor cosmetic

- removed TODO
- changed headers' year to 2024 for new files
- fixed typos

* format

* formatting and comments

1. added missing docstrings.
2. abstracted the _find_instance method.
3. aliased the IDs used in the BigTable connection

* added comment regarding private key

* added comments regarding column families

* enclose get_schema_name_list in `try/except/else`

* format

* streamlined get_schema_name_list to include all logic in the try block
2024-02-13 08:28:01 +01:00
Teddy
e6cacb19f1
MINOR - DQ incident severity classifier (#15149)
* feat: added severity classifier for DQ incidents

* feat: added severity classifier tests

* style: ran java linting
2024-02-13 07:02:09 +01:00
Teddy
99ede3d14b
fix: convert imageRepository to string type (#15104) 2024-02-12 16:05:05 +01:00
Akash Jain
95bb221025
feat: Prepare main for next release cycle (#15139) 2024-02-12 16:56:50 +05:30
Ashish Gupta
73062f761f
fix sso logo breaking due to antd icons component (#15083)
* fix sso logo breaking due to antd icons component

* fix sonar issue

* fix deploy ingestion icon

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2024-02-08 21:15:04 +05:30
Ashish Gupta
918c6fcc2d
MINOR : remove all count for user's activity feed and fix more option in task feed (#15023)
* remove all count for user's activity feed and fix more option in task feed

* minor changes

* minor fixes
2024-02-05 16:41:39 +05:30
Ashish Gupta
cbfb3bf8d6
MINOR : show only user-task in user profile page in case of admin/normal user (#15005)
* show only user-task in user profile page in case of admin/normal user

* changes as per comments and added some entity field in antlr
2024-02-02 20:11:53 +05:30
Mohit Yadav
7dc0bc5829
[Data Observability] Add Pipeline Observability and status (#14958)
* Add Pipeline Observability and status

* Compare equalsIgnoreCase

* Fix Naming Args

* Fix Thread Type Filtering

* CheckStyle Fix

* Add Pipeline List Options

* References not being used, handle USERS and TEAMS category separately

* Fix Test Case Failures

* Update Code for PipelineRepository

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-02-02 16:21:04 +05:30
Onkar Ravgan
25c6260bd7
MINOR: Optimised dbt topology and better exception handling for athena LF tags (#14978)
* dbt top and athena excp

* Added dbt excp handling

* sourcehash in patch
2024-02-02 09:21:15 +01:00
07Himank
795bd5c867
Added Index Factory Config (#14950)
* Index Factory Config

* CI handling

* checkstyle
2024-02-01 19:50:51 +05:30
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema (#14956)
* linting: fix python linting

* fix: get column types from parquet schema for parquet files

* style: python linting

* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
Sriharsha Chintalapani
2e95fcb98d
Fix #14786: Suggestions API (#14821)
* Fix #14786: Suggestions API

* Handle suggestions in ometa

* Minor: Optimise Databricks Client (#14776)

* MINOR - Fix SP topology context & Looker usage context (#14816)

* MINOR - Fix SP topology context & Looker usage context

* MINOR - Fix SP topology context & Looker usage context

* Fix tests

* Fixes #14598: Fix Tags / Labels ingestion on includeTags as False (#14782)

* fix(ui): password error message for char limits (#14808)

* fix(ui): password error message for char limits

* fix java side code

* Fixes #13556: Support for Salesforce table description ingestion (#14733)

* ISSUE-13556: Add suport for Salesforce table description ingestion

* ISSUE-13556: Remove unnecessary blank line

* ISSUE-13556: Fix to get description for each table

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

* MINOR - Better handling of Ingestion Pipeline Status (#14792)

* MINOR - Better handling of Ingestion Pipeline Status

* format

* format

* MINOR: Added table validation for cost analysis data (#14793)

* Added validation for cost analysis source

* centralized life cycle logic

* CYPRESS: simplify side navigation click in cypress (#14818)

* simplify side navigation click in cypress

* make sidbar item uses common enum

* fix cypress failure of outside import

* fix(#14326): tier dropdown is not working in advance search (#14780)

* improvement in advance search based on custom property

* fix a reading undefined property issue

* wip: advance search based on tier

* some code cleanup and improvement

* some fixes

* fix: ui flicker when advanceSearched is apply and refresh the page

* some cleanup

* no need to call customproperty api call, if entity not suppport customProperties

* minor change

* fix: autocomplete not working in tier search option in advance search modal

* added unit test for advance search provider component

* some cleanup

* added testcase for open modal

* added testcase for resetAllFilters method

* removed unwanted code

* added e2e test for testing tier advance search

* fix: e2e search flow for single field

* fix: string field not working after giving listValues in TierSearch

* fix: group query e2e test fix

* used asyncFetch way to get the tierOptions synchronously

* some cleanup

* remove unwanted lines

* some cleanup

* fix: selected option show option value instead of option title

* fix(minor): update skip icon for executions (#14809)

* Fixes #14803: ignore capitalization when confirming deletes  (#14804)

* ignore case when confirming deletes

* Test confirmation of deletes works when case differs 

Added test case for 'delete' as the confirmation text.

* minor(config): update openmetadata-ui code reviewers (#14823)

* Add Tests

* Add list/accept/reject apis

* initial ui changes

* localisation

* show suggestion for empty description

* ui feedbacks

* Fix permission check for entities without owner

* Fix entityLink and add tests

* Add update suggestion WIP

* Fix test

* Fix PUT and Pagination

* Fix styling

* update test

* Update status

* add OM server connection in apps

* add permissions check

* Fix CI

* Remove TODO

* Fix feedResourceTest

* fix unit tests

* add private configs for apps

* add private configs for apps

* fix update application icons

* minor center align icon

* add private configs for apps

* Format

* Fix pydantic gen

* Remove token

* Update name

* Rework private conf

* Fix apps

* Fix apps

* Format

* Format

* show metapilot only if its installed

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-31 18:51:09 -08:00
Pere Miquel Brull
1492f79da1
MINOR - Revert #14468 - DuckDB Connector (#14934) 2024-01-29 16:10:53 +01:00
Sachin Chaurasiya
8d1780c5b4
Fix(#14262): Format the ES reindexing logs for better readability (#14849)
* Fix(#14262): Format the ES reindexing logs for better readability

* Refactor formatJsonString function in StringsUtils.ts

* Fix Error Schema for Search Indexing Application

* Refactor AppLogsViewer.component.tsx

* Fix import order and add default value for formatJsonString

* Refactor formatJsonString function in StringsUtils.ts

* Fix Schema

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2024-01-29 12:16:20 +05:30
IceS2
373cafcda2
Fixes #5448: Implement initial Iceberg Connector using PyIceberg (#14825)
* Create the iceberg connection schema

* Link the IcebergConnection configuration with the forms on the UI

* Add the pyiceberg dependency on the ingestion package

* Create the get_connection and test_connection functions

* First iteration on the iceberg ingestion logic

* Add A more comprehensive implementation of the Iceberg Source

* Add UnitTests

* Update icebergConnection definition

* Update the iceberg souce code based on new schema

* Updated icebergConnecgtion schema for simplicity and to be able to configure Converters

* Updated setup dependencies to be more flexible

* Updated get_owner_ref logic

* Fix formatting

* Changed the icebergConnection json schema structure to enable the ClassConverters

* Add the IcebergCatalog and IcebergFileSystem ClassConverters

* Refactor the code to take into account the new jsonSchema structure

* Fix formatting

* Add Documentation for the Iceberg Connector

* Fix Menu order for Iceberg

* ui: add Iceberg service icon and constant

* Fix DynamoDb Catalog issue due to how PyIceberg instantes it

* Changed uri title to URI

* Fix ClassConverter for Iceberg

* Fix GetSecretValue for password types

* Fix formatting

* Fix formatting

* Add Iceberg Connector Images for the docs

* Add pylint disable for Hacky super() call

* Add Iceberg.md for the UI docs

* Fix pylint complaint

* Fix pylint complaint

* Fix UnitTests

* fix type error and unit tests

* update pipeline type checks

* Fix Sonar Cloud complaints

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-01-29 06:32:58 +01:00
Pere Miquel Brull
0332fb9fe5
#13934 - Update test connection no response error msg (#14897)
* #13934 - Update test connection no response error msg

* #13934 - Update test connection no response error msg
2024-01-28 10:25:15 -08:00
Sriharsha Chintalapani
f6ecc41d7c
Fix #11944: API support to fetch count for mentions (#14858)
* Fix #11944: API support to fetch count for mentions

* Add closedTask/totalTask

* fetch count as per new count api for entity and user page and remove mentions tab from entity

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2024-01-28 16:19:29 +05:30
Chirag Madlani
c936ac3a41
feat(ui): Data observability and notifications (#14806)
* Centralize EventType

* Format better for readability

* Add Handling for Thread as Change Event

* Centralize logic for Entity Message Creation

* Add Thread Parent Message in Conversation

* Add Task and Announcement

* Support for multiple destinations

* Delete existing alerts for schema changes

* Delete all tables data for event subscription

* Centralize logic for Notifications and Observability

* Change Field Name and Remove redundant code

* Test Fixes

* Add more fields

* Add Optional

* Logical Test Case Addition Chnage Event

* Add Filter By Owner

* Fix Email Entity Url

* added data observability settings page

* localization changes for other languages

* Handler closure and cleanup

* complete create observibility flow

* fix searchIndex method error

* Update Entity Observability Filter

* working observability creation

* complete creation flow

* fix action with arguments

* fix form validations and form data formulation for 'input.actions' field

* localization changes for other languages

* Fixes for Data Observability

* Add mention notification

* Only apply mention to conversation

* fIx test case

* bug fixes for add and edit observability alert form functionality

* fIx test case

* add feature to config internal along with external destinations in alert form

* Fix Formatting

* Fix Url Redirection

* add observability alert details page

* localization changes for other languages

* remove filter and action required rules

* fix styling for observability form and details page

* add delete functionality on observability alert details page

* update view page layout and add a warning in configure destination step

* localization changes for other languages

* feat(ui): support notification with new flow (#14848)

* feat(ui): support notification with new flow

* fix notifications

* add missing filters for notification alert creation flow

* update the filter hiding logic to show at the start always

* add check to show Actions section in observability alert creation form

* code cleanup and improvements

* add breadcrumb to add notification alert page

* Add General Metadata Filters

* remove code smells and code cleanup for add notification page

* add new filter

* show activity feed alert on notifications list page

* add notification alert details page changes

* localization changes for other languages

* show icons for all the entities in the alert trigger dropdown

* redirect to notification alert listing page after deleting the notification alert

* change add alert page title

* add logic to disable all destinations except email for users

* fixed unit tests

* code improvements and cleanups

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2024-01-26 13:56:18 +05:30
Sachin Chaurasiya
12bc515aca
Update EntityLink.g4 with new entity types and fields (#14836) 2024-01-24 15:36:48 +05:30
07Himank
115870b2af
Pipeline and stored procedure lineage fix (#14681)
* pipeline and stored procedure lineage

* pipeline and stored procedure lineage

* fix: pass entity type in url

* fix unit tests

* pipeline and stored procedure lineage

* pipeline lineage

* pipeline and stored procedure lineage

* fix performance issues

* pipeline and stored procedure lineage done

* optimisation

* fix fullscreen mode

* add pipeline animation

* update loading state

* fix styling

* update pipeline status

* fixes

* fix cypress

* fix cypress

* fix cypress

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-01-24 11:04:31 +05:30
Chirag Madlani
45efaefc46
fix(ui): password error message for char limits (#14808)
* fix(ui): password error message for char limits

* fix java side code
2024-01-23 12:38:35 +05:30
Sriharsha Chintalapani
15eb094cff
Improve error response to the client. Added errorType so that client can handle better than depending on the http status codes (#14784) 2024-01-21 08:07:07 -08:00
Mayur Singal
4e02cb1c54
Fix #14764: Profiler make compute metric configurable (#14791) 2024-01-19 18:13:26 +05:30
Ayush Shah
831fce5b7e
Fixes 10709: Add useFqnForFiltering to profiler workflow (#14717) 2024-01-18 18:52:43 +05:30
Mohit Yadav
1e6c1a616b
Add Notification [WIP] (#14674)
* Centralize EventType

* Format better for readability

* Add Handling for Thread as Change Event

* Centralize logic for Entity Message Creation

* Add Thread Parent Message in Conversation

* Add Task and Announcement

* Support for multiple destinations

* Delete existing alerts for schema changes

* Delete all tables data for event subscription

* Centralize logic for Notifications and Observability

* Change Field Name and Remove redundant code

* Test Fixes

* Add more fields

* Add Optional

* Logical Test Case Addition Chnage Event

* Add Filter By Owner

* Fix Email Entity Url

* Handler closure and cleanup
2024-01-18 12:46:01 +05:30
Pere Miquel Brull
eadda0e3f1
#9269 - Add MSSQL Stored Procedure Support (#14739)
* Remove unnecessary field

* Remove unnecessary field

* Support query comments in MSSQL

* Remove unnecessary field

* Format

* Add external type

* Add MSSQL SP support
2024-01-17 06:37:27 +01:00
Sachin Chaurasiya
1a83a112ef
Minor: update ENTITY_TYPE and ENTITY_FIELD in EntityLink.g4 (#14732) 2024-01-16 20:08:03 +05:30
Onkar Ravgan
13b326d11c
Fix 14040: Part 3 Added updated field to status for patched entities (#14708)
* Added updated field to status

* Fixed e2e test
2024-01-16 17:53:05 +05:30
Ayush Shah
98eae7154a
Fix Airflow Source Url (#14712) 2024-01-15 12:31:43 +00:00