743 Commits

Author SHA1 Message Date
Ayush Shah
c3fac5e97e
Fix Adls connection typo (#14449) 2023-12-20 14:25:54 +01:00
NiharDoshi99
492092720d
#12908 support for table owner in postgres (#14289)
* support pick up postgres table owner

* added owner wrapper

* optimize the code

* remove pylint error

* minor fix

* changes as per comments

* fix python checksytyle and comments

* moved process owner to database service

* update docs for version 1.3

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-12-20 18:42:59 +05:30
Onkar Ravgan
7d032977d7
Fix 14084: Removed Airflow Mssql backend connection (#14420)
* Removed Airflow Mssql backend conn

* added breaking changes docs
2023-12-20 12:28:28 +05:30
Ayush Shah
03f884b376
MINOR: Fix Typos (#14448) 2023-12-20 10:30:23 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers (#14358)
* MINOR - Clean ingestion ES code and auth providers

* clean parser

* Clean security config for the client

* Clean security config for the client

* Improve class conversion exceptions

* Fix tests

* Clean up java client

* Clean up java client

* clean parser

* Fix test

* Fix test

* fix NO_AUTH error

* Fix test

* Format

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Sriharsha Chintalapani
1294f93e15
Minor: Migrate to latest google code style library to support Java 17 and beyond (#14429)
* Minor: add spotless and use simplecontext

* Remove context from rule evaluaiton

* Fix EventSubscription tests

* Minor: Migrate to latest google code style library to support Java 17 and beyond

* Minor: Ignore code style migration from git blame
2023-12-18 12:56:17 -08:00
Pere Miquel Brull
a83a5ba3a3
MINOR - Skip delta tests for 3.11 (#14398)
* MINOR - Bump delta for 3.11

* Update flags

* MINOR - Bump delta for 3.11

* Update tests regex

* Update version

* Deprecations

* Format

* Version

* Try delta spark

* Skip delta tests for 3.11

* Update ingestion/tests/unit/topology/pipeline/test_airflow.py
2023-12-18 17:01:57 +01:00
Onkar Ravgan
1af1caf065
Prepare alation connector schemas (#14407) 2023-12-18 13:07:42 +05:30
Onkar Ravgan
ef48b7eae7
MINOR: Added custom properties methods in python sdk (#14402)
* Added custom properties methods in python sdk

* fixed tests
2023-12-18 12:57:14 +05:30
Pere Miquel Brull
d8984d267e
#14340 & #13849 - Clean secret ID and improve encrypt/decrypt exception management (#14356)
* Fix supported characters in SM

* Update SM

* Fixes

* Fixes

* Improve class conversion exceptions

* Comments

* Rename noop to db secrets manager providee

* Update sm

* Fix

* db SM

* db SM

* Fix test

* UI

* Update openmetadata-ui/src/main/resources/ui/src/mocks/IngestionListTable.mock.ts

* update default
2023-12-18 06:43:20 +01:00
Teddy
20bc238413
FIX #14248 - Implement test case resolution task workflow (#14323)
* feat: add test case resolution task workflow

* fix: conflict with main
2023-12-12 10:13:11 +01:00
Onkar Ravgan
d5ece87766
Fixed query millis duration desc (#14332) 2023-12-11 17:40:54 +05:30
Ayush Shah
b8d0b3b6e4
#14168: Mark SP and Data model deletion (#14219) 2023-12-06 08:57:41 +05:30
Onkar Ravgan
b3578cd496
Fix 14040: Part 2 Add Patch Request when updating the entities from create request (#14224) 2023-12-05 17:56:40 +05:30
Mohit Yadav
a68198298b
Adds a validation API (#14244)
* Adds a validation API

* Adds a validation API

* Add info for reindexing
2023-12-05 15:12:30 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Ashish Gupta
42463ff40b
#14134: supported retention period in table entity (#14163)
* supported retention period in table entity

* Add retention period updates

* supported unit test

* added curd operation for retention center

* minor changes

* fix modal issues and added validation

* added unit test for retention period

* fix code smell

* fix sonarcloud

* minor chnages

* Fix java code styling

* added hours in retention period

* changes as per comments

* fix sonar

* remove localization keys

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-12-05 10:42:37 +05:30
Mohit Yadav
0ae2baa9f6
Add Bulk Assets for Domain , Data product and Team (#14222)
* Add Bulk Assets for Domain , Data product and Team

* repetitive code
2023-12-04 19:06:16 +05:30
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
Mohit Yadav
64a56b0183
FEAT Fix Issues in Glosary (#14197)
* Fix User Startup Email

* Add Apis for validation and addtion glossary tags and assets

* Update Status to ApiStatus

* Unique Tags to be listed

* Fix in tag additon

* fix: asset selection modal improvements (#14152)

* fix: assets fqn issue (#14140)

* fix: assets fqn issue

* fix: unit tests

* fix: escape fqn in remaining areas

* fix: domain cypress for adding assets

* fix: add assets to data products cypress

* update imports

* add cypress for glossary term

* fix: loading in asset modal and add cypress

* fix: unit tests

* fix: reintroduce filters in asset selection modal

* fix: update assets

* localization

* Fixes and Refactor redundant code

* Revert

* Add Delete API for removing tags from assets

* ADd ES calls

* Fix Assests added in case of errors

* ui updates

* ui changes

* Add Conditional Validate for SubFields

---------

Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-12-03 00:57:03 +05:30
Pere Miquel Brull
f7a4a67834
MINOR - Add Collate URL to MetaPilot App config (#14186) 2023-11-30 11:11:56 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource (#14087)
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30
karanh37
853a83f8b9
fix: allow config update for external app (#14034)
* fix: allow config update for external app

* Prep config

* Add configs

* fix: call configure app on saving of configurations

* fix: add autocomplete widget for json schema forms

* ORganize configuration handling

* Allow config

* Format

* Fix test

* Prep migrations for allowConfiguration

* TODO

* Add pg migrations

* fix: add allowConfiguration support

* Autotagger

* Update install flow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-27 16:55:33 -08:00
Sachin Chaurasiya
6ff27d90de
fix(#14121): hive connection metastoreConnection field is not updating the value correctly (#14123) 2023-11-27 16:34:06 +05:30
Mohit Yadav
19dbc217b0
Add code flow Configuration (#14026) 2023-11-20 08:44:13 +05:30
Suresh Srinivas
c4d6f681c8
Fixes #14006 Validate tag labels before creating/updating an entity (#14027) 2023-11-19 14:15:17 -08:00
Teddy
c7ac28f2c2
Fixes #11357 - Implement profiler custom metric processing (#14021)
* feat: add backend support for custom metrics

* feat: fix python test

* feat: support custom metrics computation

* feat: updated tests for custom metrics

* feat: added dl support for min max of datetime

* feat: added is safe query check for query sampler

* feat: added support for custom metric computation in dl

* feat: added explicit addProper for pydantic model import fo Extra

* feat: added custom metric to returned obj

* feat: wrapped trino import in __init__

* feat: fix python linting

* feat: fix typing in 3.8
2023-11-17 17:51:39 +01:00
Teddy
f3da919329
Feat: Backend Support for Custom Metrics (#13965)
* feat: add backend support for custom metrics

* feat: fix python test
2023-11-17 19:16:35 +05:30
Pere Miquel Brull
6fb3f12675
App Resource - Add /configure endpoint & WAII app definition (#13983)
* App Configuration

* TODO

* Configure App with OM server info

* Remove init

* Rename configs

* Remove extra source
2023-11-17 09:28:11 +01:00
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Mayur Singal
a8bca817b4
Sample Data Storage & External Profiler Docs (#13960)
* rename title from S3 to Storage config

* Sample Data Storage & External Profiler Docs

* Some changes

* Some changes

* Some changes

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-14 08:59:32 +01:00
Mohit Yadav
10d8ec84fe
Logs added for Search Indexing and Stats issue fixed (#13956)
* Logs added for Search Indexing and Stats issue fixed

* Fix uninstall error

* Add error handling

* fix lint

* Push Job Level Exception on top

* disable flaky tests

* Fix Logs not visible in Search

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-11-13 23:39:56 +05:30
Mayur Singal
b34111ea00
ElasticSearch: Fix ES connection (#13919) 2023-11-13 14:10:11 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework (#13862)
* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Handle app runner

* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Allow deployment

* Fix PII APP

* Fix lint

* Fix PII APP

* Fix PII APP

* Prepare config-based external apps

* Prepare config-based external apps

* Fix lint

* Prepare config-based external apps

* Fix DI errors

* Amend comments
2023-11-13 08:58:38 +01:00
Mayur Singal
367bac9064
Fix #13787: Add support for ES data types (#13916)
* Fix #13787: Add support for ES data types

* fixed tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-11-10 20:14:42 +05:30
Pere Miquel Brull
0eacc829a4
Fix #13794 - Add domain support to the Python SDK (#13931) 2023-11-10 11:00:06 +01:00
Pere Miquel Brull
b250cd8808
Fix #13699 - Add separator for Storage Container manifest (#13924)
* Fix #13699 - Add separator for Storage Container manifest

* Fix #13906 - Fix add_mlmodel_lineage description field

* Add separator

* Add separator
2023-11-10 10:44:47 +01:00
Mayur Singal
4c19bb5a1d
Add support for external profiler workflow (#13887)
* Add support for external profiler workflow

* pylint

* resolve comments

* resolve comments

* pyformat

* fix code smell
2023-11-09 18:50:47 +05:30
Mayur Singal
a8145a82fa
Fix #13603: Configurable Sample Data Rows for Profiler (#13807)
* Fix #13603: Configurable Sample Data Rows

* Fix #13603: Configurable Sample Data Rows for Profiler

* fix table config

* support configurable overwriting of sample data

* add support for schema and database profiler configuration

* chore(ui): put sampleDataStorageConfig under advanced config

* fix tests

* py format

* chore(ui): add sampleDataCount in table profiler config

* fix tests

* pylint & tests

* feat(ui): add profiler settings tab in database and database schema page

* chore(ui): show different inputs for profile sample type

* schema changes to make default storange config null

* add unit test

* schema changes to fix api

* update profiler setting schema

* move profiler settings to manage button

* sync locals

* fix(ui): unit tests

* fix tests

* py format

* fix lint

* minor improvements

* chore(ui): update profiler settings schema

* resolve review comments

* pytest

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-09 18:49:42 +05:30
Mohit Yadav
2577490700
- Removed Logo Configuration from the openmetadata.yaml (#13873)
* - Removed Logi Configuration from the openmetadata.yaml
- Migrated Login Configuration to the UI

* Removed ApplicationConfig from tests

* Added Custom Header Logo Url

* Update field to favicon

* support login conifguration

* chore(ui): update logo and login config md files

* fix validation issue

* add cypress

* restructure folders and set default fallback for favicon

* autofocus to first field of edit form

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-08 20:05:00 +05:30
Suresh Srinivas
a89e317a2b
Fixes #13863 - Show inherited relationships of an entity (#13864)
* Fixes #13863 - Show inherited relationships of an entity

* Test failure fixes

* Commenting out invalid python test
2023-11-07 09:11:06 -08:00
Mohit Yadav
4a73978c77
Fixes #13803 : Reduce the number of call in case of columns (#13819)
* Fixes #13803 : Reduce the number of call in case of columns

* Add Generic Method and Interface for Fields

* Removed unnecessary comments

* Make FieldInterface return List<? extends FieldInterface> for getChildren

* Fix Failing Tests

* - Remove Pipeline or Kill on app Disable
- Add AppMarketPlaceResourceTest
- Reschedule app automatically on restore

* Add Debug Log in validate Change Event

* Fix BotResourceTest

* Api Refactor on UI
2023-11-07 13:06:38 +05:30
07Himank
b82c33574f
added data product to missing things (#13830) 2023-11-03 12:31:29 +05:30
Ayush Shah
0a04ce85bb
Add Multilingual Support in EntityLink (#13826) 2023-11-02 14:35:22 +05:30
Andrey Skibunov
3c8ec74d37
Feature: Dashboard MSTR Integration (#13604)
* Dashboard MSTR Integration

* Format

* Format and linting

---------

Co-authored-by: Andrey.Skibunov <Andrey.Skibunov@life.com.by>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-30 12:10:12 +01:00
Mohit Yadav
3580e4512d
Add DataInsightsApp (#13549)
* Add DataInsightsApp

* Add app Screenshots

* Data insight app UI (#13566)

* fix: add data insights report application

* fix: minor ui feedbacks

* fix: minor css border

* fix: added carousel code for marketplace app details

* fix: add app screenshots

* fix: update app screenshots

* fix: type errors

* Add External App Ingestion Schema

* Add Deploy and Trigger under App resource

* Add pipeline Status and logs as single response

* Fix tests

* Fix tests

---------

Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <mail@harsha.io>
2023-10-17 19:51:33 -07:00
Onkar Ravgan
6e013246a7
dbt fixed null sql updates and source descriptions (#13467) 2023-10-12 11:07:58 +05:30
Suresh Srinivas
c0986a1584
Make domain optional for data product (#13537) 2023-10-11 20:36:27 -07:00
Suresh Srinivas
25ac4fe186
Add support for loading glossaries from JSON data files (#13534) 2023-10-11 15:38:08 -07:00
Ashish Gupta
0ede37f942
feat: supported application automation from ui (#13363)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* feat: supported application automation from ui

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* initial applications ui

* fix path errors

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* add data insights application

* fix code smells

* added confirmation modal and fix active style in setting leftbar for workflow

* add schema on schedule page

* update app config

* fix build errors

* added logs in application history

* add app icons

* fix: install error

* chore version headings

* move apps under integration

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-10-12 00:29:27 +05:30