2200 Commits

Author SHA1 Message Date
Mohit Yadav
3580e4512d
Add DataInsightsApp (#13549)
* Add DataInsightsApp

* Add app Screenshots

* Data insight app UI (#13566)

* fix: add data insights report application

* fix: minor ui feedbacks

* fix: minor css border

* fix: added carousel code for marketplace app details

* fix: add app screenshots

* fix: update app screenshots

* fix: type errors

* Add External App Ingestion Schema

* Add Deploy and Trigger under App resource

* Add pipeline Status and logs as single response

* Fix tests

* Fix tests

---------

Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <mail@harsha.io>
2023-10-17 19:51:33 -07:00
07Himank
e70d95b875
Fixed inconsistency in search entity and remove chart from all alias (#13602) 2023-10-17 18:35:19 +05:30
Pere Miquel Brull
75902ac8d8
Revert "Fixes #13595 Consolidate changes by a user in a single session to a s… (#13596)" (#13598)
This reverts commit 45ad6bd05deccd279b3345270701881f2b07ede5.
2023-10-17 12:58:40 +02:00
Suresh Srinivas
45ad6bd05d
Fixes #13595 Consolidate changes by a user in a single session to a s… (#13596)
* Fixes #13595 Consolidate changes by a user in a single session to a single change

* Fix bugs found during testing
2023-10-16 20:41:14 -07:00
07Himank
cf94097e3c
Added missing search things (#13583)
* missing search things

* revert config local setting

* revert gs term mapping

* fix py test
2023-10-16 19:51:11 +05:30
07Himank
cdb94e476b
Fix glossary highlighter (#13586)
* fix glossary highlighter issue

* indentation
2023-10-16 19:22:44 +05:30
Pere Miquel Brull
6869fc04d7
Prepare migration context (#13518)
* Prepare migration context

* Prepare migration context

* Prepare migration context

* Prepare migration context

* Prepare migration context

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-15 15:17:04 +02:00
Chirag Madlani
7c25b5fddd
fix(ui): ui feedbacks for 1.2 release (#13522)
* fix(ui): ui feedbacks for 1.2 release

* fix notification panel height issue and data model expand icon issue

* Data Quality, test creation, and minimizing the side panel. It shows the name as "Setup Guide.", It should be "Data Profiler Metrics."

* changes locales

* fix: displayName for the test cases

* fix: Remove Domain shows tool tip as "Remove Owner"

* chore: remove box shadow from primary buttons

* fix: icon alignment in activity feed tab

* Long names should be truncated

* Increase the profile picture size to align with both the name and the persona

* Text in Knowledge panels should be of the same size

* Domain should be displayed on the explore card if the domain is available for any entity

* If the Domains not configure the search filter is empty. "No data available." -> "No Domains are Assigned to Tables" etc.

* fix: unit tests

* fix: unit tets

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-10-14 07:10:01 -07:00
Chirag Madlani
753e182e21
fix cypress failures (#13547)
* fix cypress fialures

* fix escape sequence for ES search

* exclude services tests

* revert cypress config

* remove unwated escape call

* skip service cypress

* mapping name mismatch

* escape search query for enter

* fix cypress failures

* fix cypress failures part 2

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
2023-10-14 16:45:28 +05:30
07Himank
7676c3e8ab
added servies to all alias (#13569) 2023-10-13 23:11:39 -07:00
07Himank
2ef49d52ca
Nested columns search fix (#13539)
* nested columns work

* working on nested columns and its nested tags

* worked on nested columns and tags

* removed unneccesary variable

* use only name in arr field

* changed the varable names

* added nested columns logic for container and dashboard datamodel too

* created constants
2023-10-13 11:29:05 -07:00
Sriharsha Chintalapani
001f75ceb8
remove log from initializeRepositories (#13558)
* remove log from initializeRepositories

* remove log from initializeRepositories
2023-10-12 22:37:27 -07:00
07Himank
6ffe79f793
fixed ES Indexing for very large S3 Storage Service buckets fails (#13507) 2023-10-13 10:22:53 +05:30
Sriharsha Chintalapani
fcb06383c1
Disable tags when the classification is disabled (#13532)
* Disable tags when the classification is disabled

* remove unnecessary constants

---------

Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-12 16:11:24 -07:00
07Himank
dfce52589a
Fix credentials issue elastic search (#13552)
* Fix Crentials issue on Elastic Search

* checkstyle
2023-10-12 16:09:03 -07:00
Sriharsha Chintalapani
e79ce5fe07
Remove jdbi unit of work (#13550)
* Remove JDBI Unit of Work transactions

* Remove JDBI Unit of Work transactions

* Remove JDBI Unit of Work transactions
2023-10-12 16:07:50 -07:00
Sriharsha Chintalapani
e66b9bbeea
Add Glossary Migration (#13485)
* Add Glossary Migration

* fix styling
2023-10-12 10:38:30 -07:00
Sriharsha Chintalapani
f3adafd512
Fix extensions migrations path (#13541)
* Fix extensions migrations path

* Fix extensionPath for migrations

* Fix extensionPath for migrations

* Fix bootstrap echo
2023-10-12 17:17:29 +05:30
Onkar Ravgan
6e013246a7
dbt fixed null sql updates and source descriptions (#13467) 2023-10-12 11:07:58 +05:30
Suresh Srinivas
25ac4fe186
Add support for loading glossaries from JSON data files (#13534) 2023-10-11 15:38:08 -07:00
Suresh Srinivas
26e3be5455
Move copy method to create entity from createEntity request to Entity Repository class (#13531) 2023-10-11 12:25:36 -07:00
Ashish Gupta
0ede37f942
feat: supported application automation from ui (#13363)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* feat: supported application automation from ui

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* initial applications ui

* fix path errors

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* add data insights application

* fix code smells

* added confirmation modal and fix active style in setting leftbar for workflow

* add schema on schedule page

* update app config

* fix build errors

* added logs in application history

* add app icons

* fix: install error

* chore version headings

* move apps under integration

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-10-12 00:29:27 +05:30
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Mohit Yadav
dac6e7b478
Remove Shade Plugin causing transitive deps failure (#13516) 2023-10-11 00:21:48 +05:30
Mohit Yadav
e577333b54
- Add Latest App Run Endpoint (#13511)
- Remove not required fields from app_extension table
2023-10-10 22:10:59 +05:30
07Himank
352b025fd7
Custom Property Backend Support and list approved glossary terms (#13496) 2023-10-10 08:50:28 -07:00
Mohit Yadav
4be007a968
Created Uber Jar for openmetada-service (#13512) 2023-10-10 18:39:32 +05:30
Mohit Yadav
ab1a156b77
Build Applications Automations (#13318)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* Fix Stats Issue

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-10 11:32:37 +05:30
Teddy
eefce68015
fix: updated DI cost analysis aggregated report (#13498) 2023-10-10 07:04:40 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML (#13501)
* Register Ingestion pipelines just from YAML

* Format
2023-10-10 07:04:04 +02:00
Sriharsha Chintalapani
35c4e18663
Upgrade search versions (#13479)
* working on maven shading

* update es in docker

* add maven shade for elasticsearch/opensearch dependencies

* add submodule for es/os dependencies

* upgrade docker elasticsearch to latest 8.10.2

* upgrade docker elasticsearch to latest 8.10.2

* Update Postgres dev to use OpenSearch

* Fix container service dependency

* revert conf change

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-08 11:55:34 -07:00
Sriharsha Chintalapani
21bba1c23b
Register system knowledge panels (#13412)
* Register system knowledge panels

* Add Recently Viewed Knowledge Panel

* Fix test
2023-10-08 07:34:01 -07:00
Pere Miquel Brull
c9b3724003
Support migration extensions (#13436)
* Support migration extensions

* Prep test

* Prep test
2023-10-07 16:00:47 -07:00
Suresh Srinivas
d0f3ee6a2a
Removing redunant throws exceptions (#13468) 2023-10-06 15:43:45 -07:00
Teddy
518fedd3fd
Cost analysis charts (#13458)
* feat: unused size report

* feat: asset count report

* feat: unused asset size

* feat: used vs unused assets count

* feat: used vs unused assets size

* feat: clients + chart result json schema
2023-10-06 08:42:38 +02:00
Ashish Gupta
11b034231a
fix(ui): supported pending services in assets (#13372)
* supported pending services in assets

* localization key change

* added storage service

* sonar fixes

* fix broken paths

---------

Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-05 19:20:59 +05:30
Onkar Ravgan
44df02010a
Added delete API for Raw Cost Analysis Report Rows (#13435)
* Added delete API

* review comments

* fixed checkstyle

* fixed naming

* checkstyle

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 14:27:23 +02:00
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Teddy
ddae3d8143
Refactor Data Insight aggregators Classes (#13433)
* fix: removed legacy OS and ES aggregator classes

* fix: centralized aggregator business logic

* fix: implemented client specific aggregator

* fix: updated client instantiation to use client specific aggregator

* fix: clean up json schema

* fix: updated DI index names

* fix: added searchIndex + storedProcedure

* fix: ran linting

* fix: updated python test to include new entity types
2023-10-05 09:31:27 +02:00
Anatoliy Shulika
b788061157
fixes #12771: Added Greenplum Ingestion Connector (#13128)
* ISSUE-12771: Added Greenplum Ingestion Connector

* fixed python code formating

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-10-04 14:53:53 +02:00
Onkar Ravgan
13f2d3e25e
Added entity_ref instead of fqn in unused data assets aggregrate API (#13421) 2023-10-04 15:07:09 +05:30
Sriharsha Chintalapani
1a90c5c1b3
Refactor Repositories: Remove passing around CollectionDAO (#13427)
* Refactor repository registration

* Refactor repository registration

* Refactor repository registration

* Refactor repository registration
2023-10-03 23:41:06 -07:00
Pere Miquel Brull
31b827585b
Allow ometa to create services without storing the connection (#13400)
* Allow ometa to create services without storing the connection

* Allow ometa to create services without storing the connection

* Fix backend tests with null connection
2023-10-04 07:48:49 +02:00
Sriharsha Chintalapani
eb84c524aa
Fix repository registration (#13425)
* Fix failing repository registration in external

* Fix failing repository registration in external

* Scan classPath to find Repository annotation

* Remove additioanl config, register any classes with Repository or Collection annotation

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-10-03 21:03:56 -07:00
Sriharsha Chintalapani
8578aee04f
Search refactor (#13397)
* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* add documentation for sharded tables (#13361)

* Cost analysis agg (#13408)

* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>

* Add Java 17 support (#12895)

* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>

* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* Merge main

* fix style

* deleted field propagation

* Fix style

* close the test client

* Changing to jersey connector

* Fix Authentication Exception headers

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 14:56:53 -07:00
Sriharsha Chintalapani
03ca972887
Add Java 17 support (#12895)
* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 01:35:50 -07:00
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow (#13384)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: ran linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Sriharsha Chintalapani
11f1df437b
Fix SettingsCache initialization (#13395)
* Fix SettingsCache initialization

* Fix SettingsCache initialization

* Fix SettingsCache initialization
2023-10-01 15:47:25 -07:00
Suresh Srinivas
2e2f9a1edb
Separate initialization of Repository and Resources (#13389)
* Separate initialization of Repository and Resources

* Change that speeds up the test. Still WIP

* Rebasing the branch with main

* Revert back previous changes and use Repositories from registry for migration

* Fixing test failures

* Code cleanup

* Fix test failures
2023-09-30 22:13:16 -07:00