1565 Commits

Author SHA1 Message Date
Ashish Gupta
11b034231a
fix(ui): supported pending services in assets (#13372)
* supported pending services in assets

* localization key change

* added storage service

* sonar fixes

* fix broken paths

---------

Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-05 19:20:59 +05:30
Onkar Ravgan
44df02010a
Added delete API for Raw Cost Analysis Report Rows (#13435)
* Added delete API

* review comments

* fixed checkstyle

* fixed naming

* checkstyle

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 14:27:23 +02:00
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Teddy
ddae3d8143
Refactor Data Insight aggregators Classes (#13433)
* fix: removed legacy OS and ES aggregator classes

* fix: centralized aggregator business logic

* fix: implemented client specific aggregator

* fix: updated client instantiation to use client specific aggregator

* fix: clean up json schema

* fix: updated DI index names

* fix: added searchIndex + storedProcedure

* fix: ran linting

* fix: updated python test to include new entity types
2023-10-05 09:31:27 +02:00
Anatoliy Shulika
b788061157
fixes #12771: Added Greenplum Ingestion Connector (#13128)
* ISSUE-12771: Added Greenplum Ingestion Connector

* fixed python code formating

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-10-04 14:53:53 +02:00
Onkar Ravgan
13f2d3e25e
Added entity_ref instead of fqn in unused data assets aggregrate API (#13421) 2023-10-04 15:07:09 +05:30
Sriharsha Chintalapani
1a90c5c1b3
Refactor Repositories: Remove passing around CollectionDAO (#13427)
* Refactor repository registration

* Refactor repository registration

* Refactor repository registration

* Refactor repository registration
2023-10-03 23:41:06 -07:00
Pere Miquel Brull
31b827585b
Allow ometa to create services without storing the connection (#13400)
* Allow ometa to create services without storing the connection

* Allow ometa to create services without storing the connection

* Fix backend tests with null connection
2023-10-04 07:48:49 +02:00
Sriharsha Chintalapani
eb84c524aa
Fix repository registration (#13425)
* Fix failing repository registration in external

* Fix failing repository registration in external

* Scan classPath to find Repository annotation

* Remove additioanl config, register any classes with Repository or Collection annotation

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-10-03 21:03:56 -07:00
Sriharsha Chintalapani
8578aee04f
Search refactor (#13397)
* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* add documentation for sharded tables (#13361)

* Cost analysis agg (#13408)

* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>

* Add Java 17 support (#12895)

* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>

* Refactor Search

* Refactor Search

* Fix propgation bugs

* Fix propgation bugs

* Fix glossary term search

* Fix glossary term search

* Only issue search requests if the client is configured properly

* Only issue search requests if the client is configured properly

* Fix glossary index

* Merge main

* fix style

* deleted field propagation

* Fix style

* close the test client

* Changing to jersey connector

* Fix Authentication Exception headers

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 14:56:53 -07:00
Sriharsha Chintalapani
03ca972887
Add Java 17 support (#12895)
* Add Java 17 support

* Change Test HTTP client provider

* Create Tests HTTP Client once

* Create Tests HTTP Client once

* fix(CI): Update CI to use jdk 17 and dockerfiles as well

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-10-03 01:35:50 -07:00
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow (#13384)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: ran linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Sriharsha Chintalapani
11f1df437b
Fix SettingsCache initialization (#13395)
* Fix SettingsCache initialization

* Fix SettingsCache initialization

* Fix SettingsCache initialization
2023-10-01 15:47:25 -07:00
Suresh Srinivas
2e2f9a1edb
Separate initialization of Repository and Resources (#13389)
* Separate initialization of Repository and Resources

* Change that speeds up the test. Still WIP

* Rebasing the branch with main

* Revert back previous changes and use Repositories from registry for migration

* Fixing test failures

* Code cleanup

* Fix test failures
2023-09-30 22:13:16 -07:00
Sriharsha Chintalapani
762824665a
Fix #12655: Customizable Landing Page - Add Personas, DocStore to support KnowledgePanels and Pages (#13229)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* Add Multiple Persona support for a user

* add more elaborate tests for docstore

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix build
2023-09-29 19:13:56 -07:00
Onkar Ravgan
855790924e
Cost Analysis - Data Insights Reports and Aggregation (#13379)
* Added reports and indexes

* Clean code

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-29 11:50:43 +02:00
Teddy
96c083dbcd
Fix typo in test case listing query (#13357)
* fix: postgres query typo

* empty commit to run ci
2023-09-28 14:52:20 +02:00
07Himank
6a6e799ee6
missed adding supports search in pipelineService (#13366) 2023-09-28 17:20:22 +05:30
Sriharsha Chintalapani
6e5ad3f44c
Fix #13342: The activity feed does not reflect changes to the objects that the user is subscribed to (#13359) 2023-09-28 13:37:53 +05:30
Suresh Srinivas
314efe918d
Fixes part 1 of 13255 - list filtering for services and data products (#13358) 2023-09-27 14:07:29 -07:00
07Himank
74e29a9f16
Generic search changes. (#13326)
* working on new search changes

* working on new search changes

* working

* working

* owner propogation done

* working on propogation

* done

* change in storageservice index

* Merge conflict fix

* Draft changes

* working on making updates generic

* added code to opensearchClientImp

* renamed suppportsSearchIndex to supportsSearch

* checkstyle

* added generic code for deleted as well

* fix tests

* fix all tests

* addressing comments

* fixed test case failure

* Fix lifecycle validation error name typo

* fix realted domain propogation

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 10:48:33 -07:00
Artiom Darie
7d2f8dc2bb
Fixes issue-11740: Added support for the om service to connect to AWS RDS using IAM roles (#11913)
* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Changed intial implementation accordingly. Added better flexibility for different auth prodvider impl

* ISSUE-11740: Clean up unnecessary classes

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Clean up unnecessary properties

* ISSUE-11740: Code formatting

* ISSUE-11740: Added support for the om service to connect to AWS RDS using IAM roles

* ISSUE-11740: Moved docs to 1.2 version

---------

Co-authored-by: artiom.darie <artiom.darie@adswizz.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 09:59:24 -07:00
Sriharsha Chintalapani
4d9570c627
Fix #13347: AdminPrincipals username with dot in it is not created wi… (#13348)
* Fix #13347: AdminPrincipals username with dot in it is not created with quoteFQN

* Fix #13347: AdminPrincipals username with dot in it is not created with quoteFQN

* Fix #13347: AdminPrincipals username with dot in it is not created with quoteFQN

* Fix test with new exception msg being thrown

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-27 15:59:24 +02:00
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites (#13345)
* Prep v1.1.7 migrations to address test cases

* get or create executable suite

* Format

* Fix tests

* Add postgres
2023-09-27 11:49:21 +02:00
Ayush Shah
d8ef497b9e
Fixes 13350: Trino Secrets masking (#13351) 2023-09-27 14:28:05 +05:30
Pere Miquel Brull
9900764961
Pipeline Service Client - Handle unexpected status response (#13314)
* airflow install docs

* Pipeline Service Client - Handle unexpected response
2023-09-26 12:58:39 +02:00
Pere Miquel Brull
a4b9a2d2cb
Fix #13023 - Fix ingestionPipeline list parameters (#13289)
* Fix ingestionPipeline list parameters

* Fix test

* support filters from ui

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-09-26 10:57:14 +02:00
Suresh Srinivas
dc5de12668
13188 (#13331)
* WIP

* Fixes #13188 Non-admin user can't add term on glossary after policy assigned
2023-09-25 12:09:25 -07:00
Sriharsha Chintalapani
106db87eec
Update EnumBackwardCompatibilityTest.java (#13328) 2023-09-25 09:53:40 -07:00
Suresh Srinivas
eb865a997d
Add BindUUID JDBI support to directly pass UUID to queries (#13315) 2023-09-22 12:21:22 -07:00
Pere Miquel Brull
880907f8c8
Check for required fields before update (#13308) 2023-09-22 15:27:17 +02:00
Sriharsha Chintalapani
c368157036
Move 1.1.4 migrations to 1.1.6 and add an empty dir for 1.1.6 migrations to invoke the data migration (#13300) 2023-09-21 16:36:45 -07:00
Teddy
e9ef7b5e81
Issue-12857: Remove ES Dependency from DI Workflow (#13303)
* feat: move elasticsearch indexing to backend + introduced EntityTimeSeries interface for timeseries type object

* feat: make reportData.json inherit from EntityTimeSeriesInterface

* feat: updated type to Object

* feat: deleted elasticsearch dependencies

* feat: removed elasticsearch indexing from workflow

* feat: added data insight sample data

* feat: cleaned up tests
2023-09-21 16:17:47 -07:00
07Himank
8292ad0078
addded aggreagtions for new indexes and fields and missing things (#13278) 2023-09-21 10:41:38 +05:30
Sriharsha Chintalapani
e0ecf49585
Fix #11970: Search by FQN; Refactor Search Indexing, Add API to searc… (#13271)
* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Add wildcard support

* Fix GlossaryTerm Patch
2023-09-20 14:40:10 +05:30
Suresh Srinivas
f45d82484d
Fixes #3090 Glossary Term approval workflow (#13269) 2023-09-19 18:30:20 -07:00
Pere Miquel Brull
ad27b09fac
Clean migration and pass nameHashColumn as param when needed (#13265) 2023-09-19 10:37:42 -07:00
Sriharsha Chintalapani
03dfd9e6b2
Fix transactions , remove unnecessary autoCommit flag check (#13234)
* Fix transactions intermittent test failures

* Fix transactions intermittent test failures

* fix serachIndex list from ui side

* invalid file name

* Revert "Fix transactions intermittent test failures"

This reverts commit 44c4c8c42824beb3e7420e51af0c5f2c6f4de90d.

* Fix commit issue and unnecessary closure in NotificationHandler

* Remove HTTPGetEventlistner, wrap get handle in try

* Fix code style

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-09-19 09:53:55 -07:00
Pere Miquel Brull
351df08957
fix: openmetadata-service/pom.xml to reduce vulnerabilities (#13224)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-ORGECLIPSEJETTY-5902998

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-09-19 08:30:58 -07:00
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts (#13247)
* Rename sandboxDomain to instanceDomain

* Test Get Charts in DomoDashboard

* Fix schemas

* Fix test

* Fix test

* Rename to Auto Tag PII

* Fix query test

* Fix query test

* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Teddy
b07dac8c1d
Issue-12305 - Add logic to sort test cases result by latest execution (#13143)
* fix: added logic to store test case result state with test case entity

* fix: added relative path logic in python unit test

* fix: added logic to sort testCases based on last execution date of test case result

* fix: ran python linting

* fix: added test case listing order param in frontend call

* fix: added tests around test suite result summary state

* fix: remove isDeleted repeated check

* fix: isolated test suite for maven tests + fix some UI failing tests

* fix: run java linting
2023-09-19 08:21:38 +02:00
Pere Miquel Brull
22b0f44e38
Part of #12998 - Query Service & Lineage filter processed queries (#13215)
* Update mapping

* Prep

* prep

* Prep query migration

* Add query index deletion fix

* Docs and Maven CI fix

* Fix tests

* Add service filter

* Add query entity FQN col migration

* Fix lint

* supported serviceFQN in query api

* Prep repo

* Prep ES query search

* Do not recompute lineage

* Format

* Fix test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-09-19 07:37:47 +02:00
Shailesh Parmar
a268f2adf8
cypress: fixed cypress failure in the main (#13221)
* cypress: fixed cypress failure in the main

* fixed flaky cypress for glossary

* fixed failing cypress

* fixed flaky cypress specs

* fixed failing cypress for query and users

* Add immediate refresh policy

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2023-09-18 17:54:43 +05:30
Sriharsha Chintalapani
734629f846
Fix #9972: Add Style with color or icon for Tags,GlossaryTerm, Domain… (#13233)
* Fix #9972: Add Badge with color or icon for Tags,GlossaryTerm, Domain and DataProducts

* Fix #9972: Add Style with color or icon for Tags,GlossaryTerm, Domain and DataProducts

* Add style to TagLabel
2023-09-17 20:01:55 -07:00
Sriharsha Chintalapani
02094179e6
Fix #12899: UI to use Tier TAG displayName if provided (#13232)
* Fix #12899: UI to use Tier TAG displayName if provided

* fix python test
2023-09-17 14:28:10 -07:00
Suresh Srinivas
f7ee1c76a0
Code cleanup necessary for Glossary workflow (#13219) 2023-09-17 09:37:29 -07:00
karanh37
e0e246772d
feat(ui): add data products page (#13178)
* changed properties of dataProducts

* feat: add data products ui

* fix: assets tab widget layout

* fix: assets tab layout in data products

* fix: minor css issues

* fix: review comments

* fix: update domain icon

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
2023-09-16 17:37:00 +05:30
Suresh Srinivas
3abb83052d
Fixes #13049 - Glossary recursive delete fails (#13182) 2023-09-15 10:21:38 -07:00