618 Commits

Author SHA1 Message Date
Mohit Yadav
a68198298b
Adds a validation API (#14244)
* Adds a validation API

* Adds a validation API

* Add info for reindexing
2023-12-05 15:12:30 +05:30
Teddy
3bbf55fcda
FIXES #14049 - Split test case resolution status from test case result (#14204)
* refactor: entityFQN as ListFilter condition

* feat: implement resolution entity timeseries

* fix: rename to testCaseResolutionStatus

* ref: extracted ES query builder into private method

* ref: extract OS query builder in its own method

* ref: remove ingestion logic for test case resolution

* fix: reorganize json schemas to fix circular import in Python

* ref: object names in typescript code

* feat: added indexing of test case resolution

* feat: added test case resolution sample data

* fix: test case resolution api logic

* fix: audit logger for entityTimeSeriesInterface

* fix: DDL generation

* style: python linting

* fix: skip UI test case resolution tests

* fix: remove extension field

* fix: renamed testCaseFailureStatus to testCaseResolutionStatus

* fix: remove reviewer

* fix: rename sequenceId to stateId

* fix: re adjust search weights

* fix: removed InReview status

* style: ran python linting
2023-12-04 23:18:01 -08:00
Ashish Gupta
42463ff40b
#14134: supported retention period in table entity (#14163)
* supported retention period in table entity

* Add retention period updates

* supported unit test

* added curd operation for retention center

* minor changes

* fix modal issues and added validation

* added unit test for retention period

* fix code smell

* fix sonarcloud

* minor chnages

* Fix java code styling

* added hours in retention period

* changes as per comments

* fix sonar

* remove localization keys

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-12-05 10:42:37 +05:30
Mohit Yadav
0ae2baa9f6
Add Bulk Assets for Domain , Data product and Team (#14222)
* Add Bulk Assets for Domain , Data product and Team

* repetitive code
2023-12-04 19:06:16 +05:30
Mayur Singal
389ae79d3c
#14115: Separate Unity Catalog From Databricks (#14138) 2023-12-04 11:22:46 +05:30
Mohit Yadav
64a56b0183
FEAT Fix Issues in Glosary (#14197)
* Fix User Startup Email

* Add Apis for validation and addtion glossary tags and assets

* Update Status to ApiStatus

* Unique Tags to be listed

* Fix in tag additon

* fix: asset selection modal improvements (#14152)

* fix: assets fqn issue (#14140)

* fix: assets fqn issue

* fix: unit tests

* fix: escape fqn in remaining areas

* fix: domain cypress for adding assets

* fix: add assets to data products cypress

* update imports

* add cypress for glossary term

* fix: loading in asset modal and add cypress

* fix: unit tests

* fix: reintroduce filters in asset selection modal

* fix: update assets

* localization

* Fixes and Refactor redundant code

* Revert

* Add Delete API for removing tags from assets

* ADd ES calls

* Fix Assests added in case of errors

* ui updates

* ui changes

* Add Conditional Validate for SubFields

---------

Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-12-03 00:57:03 +05:30
Pere Miquel Brull
f7a4a67834
MINOR - Add Collate URL to MetaPilot App config (#14186) 2023-11-30 11:11:56 +01:00
chyueyi
b6b337e09a
feat: add support for doris datasource (#14087)
* feat: add support for doris datasource

* fix: fix python style check

* fix: add pydoris dependency

* fix: add pydoris dependency

* fix: py_format_check

* fix: parse error when doris view column is VARCHAR(*), check data length if not digit then return 1

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-11-28 13:57:52 +05:30
karanh37
853a83f8b9
fix: allow config update for external app (#14034)
* fix: allow config update for external app

* Prep config

* Add configs

* fix: call configure app on saving of configurations

* fix: add autocomplete widget for json schema forms

* ORganize configuration handling

* Allow config

* Format

* Fix test

* Prep migrations for allowConfiguration

* TODO

* Add pg migrations

* fix: add allowConfiguration support

* Autotagger

* Update install flow

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-27 16:55:33 -08:00
Sachin Chaurasiya
6ff27d90de
fix(#14121): hive connection metastoreConnection field is not updating the value correctly (#14123) 2023-11-27 16:34:06 +05:30
Mohit Yadav
19dbc217b0
Add code flow Configuration (#14026) 2023-11-20 08:44:13 +05:30
Suresh Srinivas
c4d6f681c8
Fixes #14006 Validate tag labels before creating/updating an entity (#14027) 2023-11-19 14:15:17 -08:00
Teddy
c7ac28f2c2
Fixes #11357 - Implement profiler custom metric processing (#14021)
* feat: add backend support for custom metrics

* feat: fix python test

* feat: support custom metrics computation

* feat: updated tests for custom metrics

* feat: added dl support for min max of datetime

* feat: added is safe query check for query sampler

* feat: added support for custom metric computation in dl

* feat: added explicit addProper for pydantic model import fo Extra

* feat: added custom metric to returned obj

* feat: wrapped trino import in __init__

* feat: fix python linting

* feat: fix typing in 3.8
2023-11-17 17:51:39 +01:00
Teddy
f3da919329
Feat: Backend Support for Custom Metrics (#13965)
* feat: add backend support for custom metrics

* feat: fix python test
2023-11-17 19:16:35 +05:30
Pere Miquel Brull
6fb3f12675
App Resource - Add /configure endpoint & WAII app definition (#13983)
* App Configuration

* TODO

* Configure App with OM server info

* Remove init

* Rename configs

* Remove extra source
2023-11-17 09:28:11 +01:00
Mayur Singal
a8bca817b4
Sample Data Storage & External Profiler Docs (#13960)
* rename title from S3 to Storage config

* Sample Data Storage & External Profiler Docs

* Some changes

* Some changes

* Some changes

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-11-14 08:59:32 +01:00
Mohit Yadav
10d8ec84fe
Logs added for Search Indexing and Stats issue fixed (#13956)
* Logs added for Search Indexing and Stats issue fixed

* Fix uninstall error

* Add error handling

* fix lint

* Push Job Level Exception on top

* disable flaky tests

* Fix Logs not visible in Search

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-11-13 23:39:56 +05:30
Mayur Singal
b34111ea00
ElasticSearch: Fix ES connection (#13919) 2023-11-13 14:10:11 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework (#13862)
* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Handle app runner

* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Allow deployment

* Fix PII APP

* Fix lint

* Fix PII APP

* Fix PII APP

* Prepare config-based external apps

* Prepare config-based external apps

* Fix lint

* Prepare config-based external apps

* Fix DI errors

* Amend comments
2023-11-13 08:58:38 +01:00
Mayur Singal
367bac9064
Fix #13787: Add support for ES data types (#13916)
* Fix #13787: Add support for ES data types

* fixed tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-11-10 20:14:42 +05:30
Pere Miquel Brull
0eacc829a4
Fix #13794 - Add domain support to the Python SDK (#13931) 2023-11-10 11:00:06 +01:00
Pere Miquel Brull
b250cd8808
Fix #13699 - Add separator for Storage Container manifest (#13924)
* Fix #13699 - Add separator for Storage Container manifest

* Fix #13906 - Fix add_mlmodel_lineage description field

* Add separator

* Add separator
2023-11-10 10:44:47 +01:00
Mayur Singal
4c19bb5a1d
Add support for external profiler workflow (#13887)
* Add support for external profiler workflow

* pylint

* resolve comments

* resolve comments

* pyformat

* fix code smell
2023-11-09 18:50:47 +05:30
Mayur Singal
a8145a82fa
Fix #13603: Configurable Sample Data Rows for Profiler (#13807)
* Fix #13603: Configurable Sample Data Rows

* Fix #13603: Configurable Sample Data Rows for Profiler

* fix table config

* support configurable overwriting of sample data

* add support for schema and database profiler configuration

* chore(ui): put sampleDataStorageConfig under advanced config

* fix tests

* py format

* chore(ui): add sampleDataCount in table profiler config

* fix tests

* pylint & tests

* feat(ui): add profiler settings tab in database and database schema page

* chore(ui): show different inputs for profile sample type

* schema changes to make default storange config null

* add unit test

* schema changes to fix api

* update profiler setting schema

* move profiler settings to manage button

* sync locals

* fix(ui): unit tests

* fix tests

* py format

* fix lint

* minor improvements

* chore(ui): update profiler settings schema

* resolve review comments

* pytest

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-09 18:49:42 +05:30
Mohit Yadav
2577490700
- Removed Logo Configuration from the openmetadata.yaml (#13873)
* - Removed Logi Configuration from the openmetadata.yaml
- Migrated Login Configuration to the UI

* Removed ApplicationConfig from tests

* Added Custom Header Logo Url

* Update field to favicon

* support login conifguration

* chore(ui): update logo and login config md files

* fix validation issue

* add cypress

* restructure folders and set default fallback for favicon

* autofocus to first field of edit form

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-11-08 20:05:00 +05:30
Suresh Srinivas
a89e317a2b
Fixes #13863 - Show inherited relationships of an entity (#13864)
* Fixes #13863 - Show inherited relationships of an entity

* Test failure fixes

* Commenting out invalid python test
2023-11-07 09:11:06 -08:00
Mohit Yadav
4a73978c77
Fixes #13803 : Reduce the number of call in case of columns (#13819)
* Fixes #13803 : Reduce the number of call in case of columns

* Add Generic Method and Interface for Fields

* Removed unnecessary comments

* Make FieldInterface return List<? extends FieldInterface> for getChildren

* Fix Failing Tests

* - Remove Pipeline or Kill on app Disable
- Add AppMarketPlaceResourceTest
- Reschedule app automatically on restore

* Add Debug Log in validate Change Event

* Fix BotResourceTest

* Api Refactor on UI
2023-11-07 13:06:38 +05:30
07Himank
b82c33574f
added data product to missing things (#13830) 2023-11-03 12:31:29 +05:30
Ayush Shah
0a04ce85bb
Add Multilingual Support in EntityLink (#13826) 2023-11-02 14:35:22 +05:30
Andrey Skibunov
3c8ec74d37
Feature: Dashboard MSTR Integration (#13604)
* Dashboard MSTR Integration

* Format

* Format and linting

---------

Co-authored-by: Andrey.Skibunov <Andrey.Skibunov@life.com.by>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-30 12:10:12 +01:00
Mohit Yadav
3580e4512d
Add DataInsightsApp (#13549)
* Add DataInsightsApp

* Add app Screenshots

* Data insight app UI (#13566)

* fix: add data insights report application

* fix: minor ui feedbacks

* fix: minor css border

* fix: added carousel code for marketplace app details

* fix: add app screenshots

* fix: update app screenshots

* fix: type errors

* Add External App Ingestion Schema

* Add Deploy and Trigger under App resource

* Add pipeline Status and logs as single response

* Fix tests

* Fix tests

---------

Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <mail@harsha.io>
2023-10-17 19:51:33 -07:00
Onkar Ravgan
6e013246a7
dbt fixed null sql updates and source descriptions (#13467) 2023-10-12 11:07:58 +05:30
Suresh Srinivas
c0986a1584
Make domain optional for data product (#13537) 2023-10-11 20:36:27 -07:00
Suresh Srinivas
25ac4fe186
Add support for loading glossaries from JSON data files (#13534) 2023-10-11 15:38:08 -07:00
Ashish Gupta
0ede37f942
feat: supported application automation from ui (#13363)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* feat: supported application automation from ui

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* initial applications ui

* fix path errors

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* add data insights application

* fix code smells

* added confirmation modal and fix active style in setting leftbar for workflow

* add schema on schedule page

* update app config

* fix build errors

* added logs in application history

* add app icons

* fix: install error

* chore version headings

* move apps under integration

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-10-12 00:29:27 +05:30
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Chirag Madlani
4b37861fbc
feat(ui): support persona and customize landing page (#13332)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* feat(ui): support persona and customize landing page

* Add Multiple Persona support for a user

* support curd for persona

* fix code smells

* add more elaborate tests for docstore

* support customise page

* fix types

* revert unwanted file

* fix path

* WIP customise landing page

* complete persona crud operations

* functionality to add widgets separately for right side bar

* fix sonarlint

* cleanup customise settings page with list persona only

* update appbar with new mock designs

* fix useForm error

* fix sonar lint

* minor fixes for userprofile icon

* Updated the customize landing page

* localization changes

* updated settings menu for custom dashboard

* localization change for other languages

* minor fixers

* fix customise page config api call

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-10-10 17:30:30 -07:00
Mohit Yadav
ab1a156b77
Build Applications Automations (#13318)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* Fix Stats Issue

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-10 11:32:37 +05:30
Teddy
eefce68015
fix: updated DI cost analysis aggregated report (#13498) 2023-10-10 07:04:40 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML (#13501)
* Register Ingestion pipelines just from YAML

* Format
2023-10-10 07:04:04 +02:00
Mayur Singal
c0ababd8ad
Fix #13336: Clean Mark All Deleted Table Flag (#13344) 2023-10-06 16:04:54 +05:30
Teddy
518fedd3fd
Cost analysis charts (#13458)
* feat: unused size report

* feat: asset count report

* feat: unused asset size

* feat: used vs unused assets count

* feat: used vs unused assets size

* feat: clients + chart result json schema
2023-10-06 08:42:38 +02:00
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Anatoliy Shulika
b788061157
fixes #12771: Added Greenplum Ingestion Connector (#13128)
* ISSUE-12771: Added Greenplum Ingestion Connector

* fixed python code formating

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-10-04 14:53:53 +02:00
Onkar Ravgan
13f2d3e25e
Added entity_ref instead of fqn in unused data assets aggregrate API (#13421) 2023-10-04 15:07:09 +05:30
Pere Miquel Brull
31b827585b
Allow ometa to create services without storing the connection (#13400)
* Allow ometa to create services without storing the connection

* Allow ometa to create services without storing the connection

* Fix backend tests with null connection
2023-10-04 07:48:49 +02:00
Sriharsha Chintalapani
eb84c524aa
Fix repository registration (#13425)
* Fix failing repository registration in external

* Fix failing repository registration in external

* Scan classPath to find Repository annotation

* Remove additioanl config, register any classes with Repository or Collection annotation

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-10-03 21:03:56 -07:00
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs (#13376)
* Prepare Storage Connector for ADLS & Docs

* Format

* Fix test
2023-10-02 12:15:09 +02:00
Sriharsha Chintalapani
762824665a
Fix #12655: Customizable Landing Page - Add Personas, DocStore to support KnowledgePanels and Pages (#13229)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* Add Multiple Persona support for a user

* add more elaborate tests for docstore

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix build
2023-09-29 19:13:56 -07:00