743 Commits

Author SHA1 Message Date
Teddy
e57849b732
Fixes #12298 - Update report data type to camel case (#13505)
* fix: updated DI to camelCase

* fix: ran linting

* fix: added migration

* fix: remove extra parenthesis in migration file

* fix: psql migration query

* fix: OS compose host

* fix: removed commented code block
2023-10-11 08:14:21 +02:00
Chirag Madlani
4b37861fbc
feat(ui): support persona and customize landing page (#13332)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* feat(ui): support persona and customize landing page

* Add Multiple Persona support for a user

* support curd for persona

* fix code smells

* add more elaborate tests for docstore

* support customise page

* fix types

* revert unwanted file

* fix path

* WIP customise landing page

* complete persona crud operations

* functionality to add widgets separately for right side bar

* fix sonarlint

* cleanup customise settings page with list persona only

* update appbar with new mock designs

* fix useForm error

* fix sonar lint

* minor fixes for userprofile icon

* Updated the customize landing page

* localization changes

* updated settings menu for custom dashboard

* localization change for other languages

* minor fixers

* fix customise page config api call

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-10-10 17:30:30 -07:00
Mohit Yadav
ab1a156b77
Build Applications Automations (#13318)
* [WIP] Applications

* Remove JobRunr Dependency

* Cleanup Files

* Reformatted

* Add Api to delete schedule for app

* Fix Applications issue

* added missing tables for application automation

* fixed buil failure

* Add Tables for Applications

* Migrate Search Reindexing Job to New Apps

* Fixed Merge Conflicts

* Fix Build

* Add DataInsightReportJob as App

* Review Comments Fix

* update err msgs

* Add Features fields

* Add Default Search Index Application

* Fix Failures

* Add Data Insights application

* Fix trigger and add update fields

* Add Stats context

* Fix Scheduler Initialization

* Fix Stats Issue

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: 07Himank <himank07mehta@gmail.com>
Co-authored-by: 07Himank <112613760+07Himank@users.noreply.github.com>
2023-10-10 11:32:37 +05:30
Teddy
eefce68015
fix: updated DI cost analysis aggregated report (#13498) 2023-10-10 07:04:40 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML (#13501)
* Register Ingestion pipelines just from YAML

* Format
2023-10-10 07:04:04 +02:00
Mayur Singal
c0ababd8ad
Fix #13336: Clean Mark All Deleted Table Flag (#13344) 2023-10-06 16:04:54 +05:30
Teddy
518fedd3fd
Cost analysis charts (#13458)
* feat: unused size report

* feat: asset count report

* feat: unused asset size

* feat: used vs unused assets count

* feat: used vs unused assets size

* feat: clients + chart result json schema
2023-10-06 08:42:38 +02:00
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Anatoliy Shulika
b788061157
fixes #12771: Added Greenplum Ingestion Connector (#13128)
* ISSUE-12771: Added Greenplum Ingestion Connector

* fixed python code formating

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-10-04 14:53:53 +02:00
Onkar Ravgan
13f2d3e25e
Added entity_ref instead of fqn in unused data assets aggregrate API (#13421) 2023-10-04 15:07:09 +05:30
Pere Miquel Brull
31b827585b
Allow ometa to create services without storing the connection (#13400)
* Allow ometa to create services without storing the connection

* Allow ometa to create services without storing the connection

* Fix backend tests with null connection
2023-10-04 07:48:49 +02:00
Sriharsha Chintalapani
eb84c524aa
Fix repository registration (#13425)
* Fix failing repository registration in external

* Fix failing repository registration in external

* Scan classPath to find Repository annotation

* Remove additioanl config, register any classes with Repository or Collection annotation

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-10-03 21:03:56 -07:00
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs (#13376)
* Prepare Storage Connector for ADLS & Docs

* Format

* Fix test
2023-10-02 12:15:09 +02:00
Sriharsha Chintalapani
762824665a
Fix #12655: Customizable Landing Page - Add Personas, DocStore to support KnowledgePanels and Pages (#13229)
* Fix #12655: Customizable Landing Page - Add Personas

* Add Knowledge Panels

* Add Doc Store to store KnowledgePanels and Pages

* Add Doc Store to store KnowledgePanels and Pages schemas

* Fix tests

* Add Multiple Persona support for a user

* add more elaborate tests for docstore

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix Page and KnowledgePanel required fields

* Fix build
2023-09-29 19:13:56 -07:00
Onkar Ravgan
855790924e
Cost Analysis - Data Insights Reports and Aggregation (#13379)
* Added reports and indexes

* Clean code

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-29 11:50:43 +02:00
Cristian Calugaru
5d8457b597
Fixes ISSUE-10587: global manifest option for storage services (#12017)
* global manifest option for storage services

* added a no metadata config source option for global manifest s3 services option

* merge fixes

* more merge fixes.

* black stuff

* test fixes

* formatting

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-28 07:55:40 +02:00
Suresh Srinivas
dc5de12668
13188 (#13331)
* WIP

* Fixes #13188 Non-admin user can't add term on glossary after policy assigned
2023-09-25 12:09:25 -07:00
Mohit Yadav
d71647f9fd
Adds Edited By as relationship (#13317) 2023-09-25 02:38:10 +05:30
Teddy
e9ef7b5e81
Issue-12857: Remove ES Dependency from DI Workflow (#13303)
* feat: move elasticsearch indexing to backend + introduced EntityTimeSeries interface for timeseries type object

* feat: make reportData.json inherit from EntityTimeSeriesInterface

* feat: updated type to Object

* feat: deleted elasticsearch dependencies

* feat: removed elasticsearch indexing from workflow

* feat: added data insight sample data

* feat: cleaned up tests
2023-09-21 16:17:47 -07:00
Suresh Srinivas
f45d82484d
Fixes #3090 Glossary Term approval workflow (#13269) 2023-09-19 18:30:20 -07:00
Ashish Gupta
d7c212e893
fix(ui) : supported storedProcedure and dashboardDataModel search index (#13194)
* supported storedProcedure and dashboardDataModel search index

* added multi language support in lineage drawer and added stored procedure and data model in it

* added fix in explore side panel

* made id required in schema and ui side

* remove stored procedure from linages sidebar

* ui: supported storedProcedure as edge in lineage (#13225)

* supported storedProcedure as edge in lineage

* fix unit test

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-09-19 19:42:39 +05:30
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts (#13247)
* Rename sandboxDomain to instanceDomain

* Test Get Charts in DomoDashboard

* Fix schemas

* Fix test

* Fix test

* Rename to Auto Tag PII

* Fix query test

* Fix query test

* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Pere Miquel Brull
22b0f44e38
Part of #12998 - Query Service & Lineage filter processed queries (#13215)
* Update mapping

* Prep

* prep

* Prep query migration

* Add query index deletion fix

* Docs and Maven CI fix

* Fix tests

* Add service filter

* Add query entity FQN col migration

* Fix lint

* supported serviceFQN in query api

* Prep repo

* Prep ES query search

* Do not recompute lineage

* Format

* Fix test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-09-19 07:37:47 +02:00
Onkar Ravgan
cc47f5618f
Added SSL support to Superset (#13130) 2023-09-18 10:45:21 +05:30
Sriharsha Chintalapani
8cb73340ff
Fix glossary test (#13235) 2023-09-17 21:56:30 -07:00
Sriharsha Chintalapani
734629f846
Fix #9972: Add Style with color or icon for Tags,GlossaryTerm, Domain… (#13233)
* Fix #9972: Add Badge with color or icon for Tags,GlossaryTerm, Domain and DataProducts

* Fix #9972: Add Style with color or icon for Tags,GlossaryTerm, Domain and DataProducts

* Add style to TagLabel
2023-09-17 20:01:55 -07:00
Sriharsha Chintalapani
02094179e6
Fix #12899: UI to use Tier TAG displayName if provided (#13232)
* Fix #12899: UI to use Tier TAG displayName if provided

* fix python test
2023-09-17 14:28:10 -07:00
Onkar Ravgan
1e4d48a034
Added Life Cycle sample data and changed datetime to timestamp (#13141) 2023-09-13 10:59:19 +05:30
Sriharsha Chintalapani
90d1e87cd2
Fix #12898: Support custom properties for Glossary and Glossary Term (#13125) 2023-09-13 00:15:55 +05:30
Mayur Singal
e59304ed31
Fix hive metastore testconnection (#13157) 2023-09-12 22:49:54 +05:30
Pere Miquel Brull
f0995cbddc
Part of #12998 - Prep Stored Procedures Skeleton for Snowflake (#13121)
* Prep Stored Procedures Skeleton for Snowflake

* Update pylint and add migrations

* Fix test

* Reuse source url computation
2023-09-12 14:25:42 +02:00
Mayur Singal
81e40b7f59
Fix Ingesion Pipeline Configuration Titles (#13139) 2023-09-12 12:13:24 +05:30
Mayur Singal
2e3e50f0cb
Add used by field in query entity (#13096) 2023-09-11 17:30:34 +05:30
07Himank
e98cb244a4
Refactoring search clients (#13016)
* working on refactoring the search clients

* indention

* working on refactoring

* working on refactoring

* working on the refotring the whole search

* working on the refotring the whole search

* working on update indexes from repository

* Refactoring the clients

* cleanup code

* added index removed comments

* ..

* checkstyle

* working on adding new indexes

* style

* incorrect changes

* missed a catch statement

* error handling at one place and async call to clients

* added new indexes to jp,zh async call , many small handlings

* task type fix

* fixing CI

* checkStyle

* fixed CI testes

* fixed CI glossary test issue

* remove reindexing from the inportcsv

---------

Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-09-09 12:24:46 +05:30
Mohit Yadav
e70f7afe4b
Add Voting to Entities (#13101)
* Add Voting to Entities

* Fix Failing test and maven build failure in main

* revert files
2023-09-08 16:23:32 +05:30
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy (#13061) 2023-09-08 12:40:48 +05:30
karanh37
b6bab6c7dd
feat(ui): Domains functionality (#12839) 2023-09-07 12:26:25 +05:30
Onkar Ravgan
d8e78a32d8
Added Life Cycle fields, API and ingestion integration (#13014)
* Added initial code

* temp commit

* Added snowflake changes

* Added postgres migration

* removed non req changes

* review comments one

* Added ometa tests

* lint fix

* fix lint naming

* fixed merge changes
2023-09-06 12:37:55 +02:00
Onkar Ravgan
a3ca8b6e66
Added project filter pattern to dashboard entity (#12925) 2023-09-05 11:01:28 +05:30
Mayur Singal
029786d773
Fix #9207: Add source in lineage details (#13046) 2023-09-04 11:05:56 +05:30
gauthk6
ba2201f4ea
draft: implementation of lightdash connector. (#12957)
* Fixes a bug while patching the description of a TestCase

* Update docker-compose.yml

* Update docker-compose.yml

* Ran pre-commit checks and linter

* Added some clarifying points and fixed some grammatical errors in the documentation for installation instructions.

* revert changes made to docs

* implementation of lightdash connector.

* ui: add icon for lightdash service

* chore: update lightdash image

* chore: update the icon

* Cleaned up code (took out debug statements, etc.). Still TODO: yield_dashboard_lineage_details() not being called just yet.

* fix: ran linting

* added null checks

* Delete openmetadata-server.md

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-08-31 17:28:07 +02:00
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase (#12836) 2023-08-29 11:16:32 +05:30
Sriharsha Chintalapani
19b5c946a7
Fix #12167: Support for Stored Procedures as another entity under Database Schema (#12999)
* Add Stored Procedure Entity

* Stored Procedure repository

* Stored Procedure repository

* Fix #12998: Support for Stored Procedures as another entity under Database Schema

* Fix #12998: Support for Stored Procedures as another entity under Database Schema
2023-08-25 08:14:30 +02:00
vanshika18
52153dba98
fixed #10691 support multiDatabase, usage & lineage support for Azure SQL (#12982) 2023-08-24 17:47:41 +05:30
Ayush Shah
4464df81ff
Add include views to profiler (#12974) 2023-08-23 17:30:44 +05:30
Pere Miquel Brull
4862b0df28
Fix #10344 - Use ingestionPipeline retries when deploying (#12926) 2023-08-23 14:52:54 +05:30
Mayur Singal
f6d5c7413f
Fix #6700: Add support for table properties: file format for datalake (#12920)
* Fix #6700: Add support for table properties: file format for datalake & storage

* pylint fix

* resolve review comments
2023-08-22 09:46:22 +02:00
Suresh Srinivas
28b5e00c0c
Clean up documentation typos and grammar issues (#12930) 2023-08-20 20:08:30 -07:00