1562 Commits

Author SHA1 Message Date
07Himank
2ef49d52ca
Nested columns search fix (#13539)
* nested columns work

* working on nested columns and its nested tags

* worked on nested columns and tags

* removed unneccesary variable

* use only name in arr field

* changed the varable names

* added nested columns logic for container and dashboard datamodel too

* created constants
2023-10-13 11:29:05 -07:00
Sriharsha Chintalapani
001f75ceb8
remove log from initializeRepositories (#13558)
* remove log from initializeRepositories

* remove log from initializeRepositories
2023-10-12 22:37:27 -07:00
07Himank
6ffe79f793
fixed ES Indexing for very large S3 Storage Service buckets fails (#13507) 2023-10-13 10:22:53 +05:30
Sriharsha Chintalapani
fcb06383c1
Disable tags when the classification is disabled (#13532)
* Disable tags when the classification is disabled

* remove unnecessary constants

---------

Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-12 16:11:24 -07:00
07Himank
dfce52589a
Fix credentials issue elastic search (#13552)
* Fix Crentials issue on Elastic Search

* checkstyle
2023-10-12 16:09:03 -07:00
Sriharsha Chintalapani
e79ce5fe07
Remove jdbi unit of work (#13550)
* Remove JDBI Unit of Work transactions

* Remove JDBI Unit of Work transactions

* Remove JDBI Unit of Work transactions
2023-10-12 16:07:50 -07:00
Sriharsha Chintalapani
e66b9bbeea
Add Glossary Migration (#13485)
* Add Glossary Migration

* fix styling
2023-10-12 10:38:30 -07:00
Sriharsha Chintalapani
f3adafd512
Fix extensions migrations path (#13541)
* Fix extensions migrations path

* Fix extensionPath for migrations

* Fix extensionPath for migrations

* Fix bootstrap echo
2023-10-12 17:17:29 +05:30
Onkar Ravgan
6e013246a7
dbt fixed null sql updates and source descriptions (#13467) 2023-10-12 11:07:58 +05:30
Suresh Srinivas
25ac4fe186
Add support for loading glossaries from JSON data files (#13534) 2023-10-11 15:38:08 -07:00
Suresh Srinivas
26e3be5455
Move copy method to create entity from createEntity request to Entity Repository class (#13531) 2023-10-11 12:25:36 -07:00
Ashish Gupta
0ede37f942
feat: supported application automation from ui (#13363)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* feat: supported application automation from ui

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* initial applications ui

* fix path errors

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* add data insights application

* fix code smells

* added confirmation modal and fix active style in setting leftbar for workflow

* add schema on schedule page

* update app config

* fix build errors

* added logs in application history

* add app icons

* fix: install error

* chore version headings

* move apps under integration

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
2023-10-12 00:29:27 +05:30
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Mohit Yadav
e577333b54
- Add Latest App Run Endpoint (#13511)
- Remove not required fields from app_extension table
2023-10-10 22:10:59 +05:30
07Himank
352b025fd7
Custom Property Backend Support and list approved glossary terms (#13496) 2023-10-10 08:50:28 -07:00
Mohit Yadav
ab1a156b77
Build Applications Automations (#13318)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* Fix Stats Issue

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-10 11:32:37 +05:30
Teddy
eefce68015
fix: updated DI cost analysis aggregated report (#13498) 2023-10-10 07:04:40 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML (#13501)
* Register Ingestion pipelines just from YAML

* Format
2023-10-10 07:04:04 +02:00
Sriharsha Chintalapani
35c4e18663
Upgrade search versions (#13479)
* working on maven shading

* update es in docker

* add maven shade for elasticsearch/opensearch dependencies

* add submodule for es/os dependencies

* upgrade docker elasticsearch to latest 8.10.2

* upgrade docker elasticsearch to latest 8.10.2

* Update Postgres dev to use OpenSearch

* Fix container service dependency

* revert conf change

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-08 11:55:34 -07:00
Sriharsha Chintalapani
21bba1c23b
Register system knowledge panels (#13412)
* Register system knowledge panels

* Add Recently Viewed Knowledge Panel

* Fix test
2023-10-08 07:34:01 -07:00
Pere Miquel Brull
c9b3724003
Support migration extensions (#13436)
* Support migration extensions

* Prep test

* Prep test
2023-10-07 16:00:47 -07:00
Suresh Srinivas
d0f3ee6a2a
Removing redunant throws exceptions (#13468) 2023-10-06 15:43:45 -07:00
Teddy
518fedd3fd
Cost analysis charts (#13458)
* feat: unused size report

* feat: asset count report

* feat: unused asset size

* feat: used vs unused assets count

* feat: used vs unused assets size

* feat: clients + chart result json schema
2023-10-06 08:42:38 +02:00
Ashish Gupta
11b034231a
fix(ui): supported pending services in assets (#13372)
* supported pending services in assets

* localization key change

* added storage service

* sonar fixes

* fix broken paths

---------

Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-05 19:20:59 +05:30
Onkar Ravgan
44df02010a
Added delete API for Raw Cost Analysis Report Rows (#13435)
* Added delete API

* review comments

* fixed checkstyle

* fixed naming

* checkstyle

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 14:27:23 +02:00
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Teddy
ddae3d8143
Refactor Data Insight aggregators Classes (#13433)
* fix: removed legacy OS and ES aggregator classes

* fix: centralized aggregator business logic

* fix: implemented client specific aggregator

* fix: updated client instantiation to use client specific aggregator

* fix: clean up json schema

* fix: updated DI index names

* fix: added searchIndex + storedProcedure

* fix: ran linting

* fix: updated python test to include new entity types
2023-10-05 09:31:27 +02:00
Anatoliy Shulika
b788061157
fixes #12771: Added Greenplum Ingestion Connector (#13128)
* ISSUE-12771: Added Greenplum Ingestion Connector

* fixed python code formating

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-10-04 14:53:53 +02:00
Onkar Ravgan
13f2d3e25e
Added entity_ref instead of fqn in unused data assets aggregrate API (#13421) 2023-10-04 15:07:09 +05:30
Sriharsha Chintalapani
1a90c5c1b3
Refactor Repositories: Remove passing around CollectionDAO (#13427)
* Refactor repository registration

* Refactor repository registration

* Refactor repository registration

* Refactor repository registration
2023-10-03 23:41:06 -07:00
Pere Miquel Brull
31b827585b
Allow ometa to create services without storing the connection (#13400)
* Allow ometa to create services without storing the connection

* Allow ometa to create services without storing the connection

* Fix backend tests with null connection
2023-10-04 07:48:49 +02:00
Sriharsha Chintalapani
eb84c524aa
Fix repository registration (#13425)
* Fix failing repository registration in external

* Fix failing repository registration in external

* Scan classPath to find Repository annotation

* Remove additioanl config, register any classes with Repository or Collection annotation

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-10-03 21:03:56 -07:00
Sriharsha Chintalapani
8578aee04f
Search refactor (#13397)
* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* add documentation for sharded tables (#13361)

* Cost analysis agg (#13408)

* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>

* Add Java 17 support (#12895)

* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>

* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* Merge main

* fix style

* deleted field propagation

* Fix style

* close the test client

* Changing to jersey connector

* Fix Authentication Exception headers

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 14:56:53 -07:00
Sriharsha Chintalapani
03ca972887
Add Java 17 support (#12895)
* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 01:35:50 -07:00
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow (#13384)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: ran linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Sriharsha Chintalapani
11f1df437b
Fix SettingsCache initialization (#13395)
* Fix SettingsCache initialization

* Fix SettingsCache initialization

* Fix SettingsCache initialization
2023-10-01 15:47:25 -07:00
Suresh Srinivas
2e2f9a1edb
Separate initialization of Repository and Resources (#13389)
* Separate initialization of Repository and Resources

* Change that speeds up the test. Still WIP

* Rebasing the branch with main

* Revert back previous changes and use Repositories from registry for migration

* Fixing test failures

* Code cleanup

* Fix test failures
2023-09-30 22:13:16 -07:00
Sriharsha Chintalapani
762824665a
Fix #12655: Customizable Landing Page - Add Personas, DocStore to support KnowledgePanels and Pages (#13229)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* Add Multiple Persona support for a user

* add more elaborate tests for docstore

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix build
2023-09-29 19:13:56 -07:00
Onkar Ravgan
855790924e
Cost Analysis - Data Insights Reports and Aggregation (#13379)
* Added reports and indexes

* Clean code

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-29 11:50:43 +02:00
Teddy
96c083dbcd
Fix typo in test case listing query (#13357)
* fix: postgres query typo

* empty commit to run ci
2023-09-28 14:52:20 +02:00
07Himank
6a6e799ee6
missed adding supports search in pipelineService (#13366) 2023-09-28 17:20:22 +05:30
Sriharsha Chintalapani
6e5ad3f44c
Fix #13342: The activity feed does not reflect changes to the objects that the user is subscribed to (#13359) 2023-09-28 13:37:53 +05:30
Suresh Srinivas
314efe918d
Fixes part 1 of 13255 - list filtering for services and data products (#13358) 2023-09-27 14:07:29 -07:00
07Himank
74e29a9f16
Generic search changes. (#13326)
* working on new search changes

* working on new search changes

* working

* working

* owner propogation done

* working on propogation

* done

* change in storageservice index

* Merge conflict fix

* Draft changes

* working on making updates generic

* added code to opensearchClientImp

* renamed suppportsSearchIndex to supportsSearch

* checkstyle

* added generic code for deleted as well

* fix tests

* fix all tests

* addressing comments

* fixed test case failure

* Fix lifecycle validation error name typo

* fix realted domain propogation

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 10:48:33 -07:00
Artiom Darie
7d2f8dc2bb
Fixes issue-11740: Added support for the om service to connect to AWS RDS using IAM roles (#11913)
* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Changed intial implementation accordingly. Added better flexibility for different auth prodvider impl

* ISSUE-11740: Clean up unnecessary classes

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Code formatting

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Moved docs to 1.2 version

---------

Co-authored-by: artiom.darie <artiom.darie@adswizz.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 09:59:24 -07:00
Sriharsha Chintalapani
4d9570c627
Fix #13347: AdminPrincipals username with dot in it is not created wi… (#13348)
* Fix #13347: AdminPrincipals username with dot in it is not created with quoteFQN

* Fix #13347: AdminPrincipals username with dot in it is not created with quoteFQN

* Fix #13347: AdminPrincipals username with dot in it is not created with quoteFQN

* Fix test with new exception msg being thrown

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-27 15:59:24 +02:00
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites (#13345)
* Prep v1.1.7 migrations to address test cases

* get or create executable suite

* Format

* Fix tests

* Add postgres
2023-09-27 11:49:21 +02:00
Ayush Shah
d8ef497b9e
Fixes 13350: Trino Secrets masking (#13351) 2023-09-27 14:28:05 +05:30
Pere Miquel Brull
9900764961
Pipeline Service Client - Handle unexpected status response (#13314)
* airflow install docs

* Pipeline Service Client - Handle unexpected response
2023-09-26 12:58:39 +02:00