3385 Commits

Author SHA1 Message Date
Onkar Ravgan
1e48d2ecff
Added sd changes (#13446)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 12:24:32 +02:00
Mayur Singal
0090286924
Fix Bigquery Test connection for multiproject (#13380)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-05 14:50:42 +05:30
Mayur Singal
f879656f0a
Fix #12047: Clean commonregex package from setup (#13439) 2023-10-05 13:41:31 +05:30
Teddy
c4a3de6a85
fix: handle tableConfig for profiler CLI (#13437)
* fix: handle tableConfig for profiler CLI

* fix: empty commit for CI
2023-10-05 10:02:57 +02:00
Teddy
ddae3d8143
Refactor Data Insight aggregators Classes (#13433)
* fix: removed legacy OS and ES aggregator classes

* fix: centralized aggregator business logic

* fix: implemented client specific aggregator

* fix: updated client instantiation to use client specific aggregator

* fix: clean up json schema

* fix: updated DI index names

* fix: added searchIndex + storedProcedure

* fix: ran linting

* fix: updated python test to include new entity types
2023-10-05 09:31:27 +02:00
Nguyen Huu Loc
ef1974edd6
Support LookML multi repos (#13140)
* Draft: Support LookML multi repos

* [Looker] manually create Dashboard datamodel

* [Looker] Support remote import & lineage for looker view

* Rollback parser.py

* refactor code

* Update code

* Remove logs & add comments

* Remove Middle & Nothing

* - Fix yield datamodel error
- Remove logs

* Support clone repo from Bitbucket

* Fix typo

* Optimize imports

* Fix pylint

---------

Co-authored-by: Loc Nguyen <loc.nguyenhuu@xendit.co>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-04 15:16:21 +02:00
Ayush Shah
97f4f8fbf3
Fixes 12922: Trino NaN issue + TrinoUserError (#13244)
* Fix Trino NaN issue + TrinoUserError
2023-10-04 18:39:39 +05:30
RyoAriyama
b2ee1a54ef
fix return type of docstring powerbi (#13422) 2023-10-04 15:00:06 +02:00
Anatoliy Shulika
b788061157
fixes #12771: Added Greenplum Ingestion Connector (#13128)
* ISSUE-12771: Added Greenplum Ingestion Connector

* fixed python code formating

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-10-04 14:53:53 +02:00
mitchellmann
0ba02a5977
Fixes 13249: Added ingestion support - Presto tbl/col comments (#13250)
* Added ingestion support - Presto tbl/col comments

* now supports scenario of NONE schema

* PY style fixes

---------

Co-authored-by: Mitchell Mann <mitchell.mann@tideworks.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-10-04 14:41:12 +02:00
Mayur Singal
3b640b43b7
Fix column lineage nonetype error (#13432) 2023-10-04 17:48:56 +05:30
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config (#13310)
* Create ometa client once and pass it around & improve pycln config

* Fix

* Fix

* Fix tests

* Fix maven ci

* Fix tests

* Fix tests

* Fix tests

* Format

* Fix DI
2023-10-04 09:14:03 +02:00
Pere Miquel Brull
31b827585b
Allow ometa to create services without storing the connection (#13400)
* Allow ometa to create services without storing the connection

* Allow ometa to create services without storing the connection

* Fix backend tests with null connection
2023-10-04 07:48:49 +02:00
Mayur Singal
4f4d1c725c
Fix failing E2Es (#13419) 2023-10-04 10:56:34 +05:30
Teddy
9ef3ff7a58
Cost analysis agg (#13408)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: added unusedReport to DI

* feat: added aggregated unused report

* feat: ran linting

* feat: reverted compose file changes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-03 09:27:18 +02:00
Onkar Ravgan
bc491be5ad
Fixed dbt optional files for local config (#13242) 2023-10-03 12:44:06 +05:30
Ayush Shah
462b2f9445
Fix Latest Pylint 3.0.0 issues (#13413)
* Fix Latest Pylint issues

* add compatible bound to pylint version
2023-10-03 07:43:09 +02:00
Pere Miquel Brull
b5596a4640
Batch PII tagging (#13385)
* Batch PII tagging

* Batch PII tagging

* Fix tests

* Fix tests
2023-10-02 14:44:41 +02:00
Pere Miquel Brull
d915254fac
Prepare Storage Connector for ADLS & Docs (#13376)
* Prepare Storage Connector for ADLS & Docs

* Format

* Fix test
2023-10-02 12:15:09 +02:00
Teddy
6ca71ae323
Issue 12679 - Handle Entity Object Instantiation Error + Refactor Workflow (#13384)
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes

* feat: __init__.py files creation

* feat: updated workflow import classes in code and doc

* feat: moved kpi runner from runner to processor folder

* fix: skip failure on list entities

* feat: deleted unused files

* feat: updated status reporter

* feat: ran linting

* feat: fix test error with typing and fqn

* feat: updated test dependencies

* feat: ran linting

* feat: move execution order up

* feat: updated cost analysis report to align with new workflow

* feat: fix entity already exists for pipeline entity status

* feat: ran python linting

* feat: move skip_on_failure to method

* feat: ran linting

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-10-02 12:05:30 +02:00
Onkar Ravgan
855790924e
Cost Analysis - Data Insights Reports and Aggregation (#13379)
* Added reports and indexes

* Clean code

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-29 11:50:43 +02:00
Teddy
162b181b4f
fix: pass metadata object to interface (#13339) 2023-09-29 08:02:02 +02:00
Cristian Calugaru
5d8457b597
Fixes ISSUE-10587: global manifest option for storage services (#12017)
* global manifest option for storage services

* added a no metadata config source option for global manifest s3 services option

* merge fixes

* more merge fixes.

* black stuff

* test fixes

* formatting

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-28 07:55:40 +02:00
Mayur Singal
1dee004dee
Fix snowflake test connection (#13354)
* Fix: Flaky Test Connection for Snowflake

* optimize code

* pyformat
2023-09-27 10:56:20 -07:00
07Himank
74e29a9f16
Generic search changes. (#13326)
* working on new search changes

* working on new search changes

* working

* working

* owner propogation done

* working on propogation

* done

* change in storageservice index

* Merge conflict fix

* Draft changes

* working on making updates generic

* added code to opensearchClientImp

* renamed suppportsSearchIndex to supportsSearch

* checkstyle

* added generic code for deleted as well

* fix tests

* fix all tests

* addressing comments

* fixed test case failure

* Fix lifecycle validation error name typo

* fix realted domain propogation

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-27 10:48:33 -07:00
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites (#13345)
* Prep v1.1.7 migrations to address test cases

* get or create executable suite

* Format

* Fix tests

* Add postgres
2023-09-27 11:49:21 +02:00
vanshika18
6f92c3be1f
fix #11882 upgrade MLflow to 2.x (#13299) 2023-09-26 11:04:15 +05:30
Mayur Singal
3769a9d105
Fix: Bigquery Table Tag not assigned (#13325) 2023-09-26 08:56:05 +05:30
Ayush Shah
04760177f6
Fixes 13321: Fix Test Connection timeout (#13323) 2023-09-25 15:17:38 +05:30
Mayur Singal
65f65137e6
Fix: Bigquery query log not picked up for multiproject (#13313) 2023-09-25 08:07:48 +05:30
Teddy
a7dd7012ea
fix: python test to remove database race condition (#13307) 2023-09-22 15:05:57 +02:00
Teddy
e9ef7b5e81
Issue-12857: Remove ES Dependency from DI Workflow (#13303)
* feat: move elasticsearch indexing to backend + introduced EntityTimeSeries interface for timeseries type object

* feat: make reportData.json inherit from EntityTimeSeriesInterface

* feat: updated type to Object

* feat: deleted elasticsearch dependencies

* feat: removed elasticsearch indexing from workflow

* feat: added data insight sample data

* feat: cleaned up tests
2023-09-21 16:17:47 -07:00
William Geuns
270a923eb9
Fixes #10096: adding RuntimeDataBatchSpec as a GX backend (#13264)
* test

* support different gx backend

* remove unused import

* fix formatting

* Update ingestion/src/metadata/great_expectations/action.py

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

---------

Co-authored-by: William Geuns <william.geuns@vrt.be>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-09-21 21:55:29 +02:00
Onkar Ravgan
8acebbb892
Update ingestion logic to use PATCH API for lifeCycle info (#13283) 2023-09-21 16:40:09 +05:30
Ayush Shah
386b7e9405
Fix Storage Key in response (#13273) 2023-09-20 15:57:08 +05:30
Sriharsha Chintalapani
e0ecf49585
Fix #11970: Search by FQN; Refactor Search Indexing, Add API to searc… (#13271)
* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Add wildcard support

* Fix GlossaryTerm Patch
2023-09-20 14:40:10 +05:30
VolkovGeoPhy
68fcd551fe
GX up to 0.17+ (#12038)
* GX up to 0.17+

* ~=

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 18:33:31 +02:00
Teddy
a424c37a34
[Proposal] Playwright CI (#13248)
* feat: playwright CI tests

* feat: ran python linting

* feat: updated workflow to run in PR

* feat: updated server start step

* feat: fixed playwright run command

* feat: fix matrix value

* feat: removed browser option in test run

* feat: updated typo in make file

* feat: move playwritgh installation

* feat: fail CI on error

* feat: save tracing on failure

* feat: enforce expect on .fill

* feat: change tracing to video

* feat: clean up code and update CI workflow for main branch
2023-09-19 14:44:36 +02:00
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts (#13247)
* Rename sandboxDomain to instanceDomain

* Test Get Charts in DomoDashboard

* Fix schemas

* Fix test

* Fix test

* Rename to Auto Tag PII

* Fix query test

* Fix query test

* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Pere Miquel Brull
7d711dc680
Fix Stored Procedures - Don't get queries without context (#13257) 2023-09-19 12:02:29 +02:00
Teddy
b07dac8c1d
Issue-12305 - Add logic to sort test cases result by latest execution (#13143)
* fix: added logic to store test case result state with test case entity

* fix: added relative path logic in python unit test

* fix: added logic to sort testCases based on last execution date of test case result

* fix: ran python linting

* fix: added test case listing order param in frontend call

* fix: added tests around test suite result summary state

* fix: remove isDeleted repeated check

* fix: isolated test suite for maven tests + fix some UI failing tests

* fix: run java linting
2023-09-19 08:21:38 +02:00
Pere Miquel Brull
22b0f44e38
Part of #12998 - Query Service & Lineage filter processed queries (#13215)
* Update mapping

* Prep

* prep

* Prep query migration

* Add query index deletion fix

* Docs and Maven CI fix

* Fix tests

* Add service filter

* Add query entity FQN col migration

* Fix lint

* supported serviceFQN in query api

* Prep repo

* Prep ES query search

* Do not recompute lineage

* Format

* Fix test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-09-19 07:37:47 +02:00
Onkar Ravgan
e2a6d0cd71
Added caching for life cycle query (#13203) 2023-09-18 12:38:07 +05:30
Onkar Ravgan
cc47f5618f
Added SSL support to Superset (#13130) 2023-09-18 10:45:21 +05:30
Sriharsha Chintalapani
02094179e6
Fix #12899: UI to use Tier TAG displayName if provided (#13232)
* Fix #12899: UI to use Tier TAG displayName if provided

* fix python test
2023-09-17 14:28:10 -07:00
Pere Miquel Brull
442528267c
Simplify topology & update context management (#13196) 2023-09-15 09:44:42 +02:00
Ayush Shah
947565d8a3
check importlib setup (#13200) 2023-09-15 12:03:47 +05:30
Pere Miquel Brull
5edad7b418
Part of #12998 - Add Stored Procedures support for Redshift (#13174)
* Part of #12998 - Add Stored Procedures support for Redshift

* Lint

* Fix bq queries

* Update ingestion/src/metadata/ingestion/source/database/bigquery/queries.py
2023-09-14 15:32:05 +02:00
Ayush Shah
387128f1d2
Bigquery: Add Table Level Tags, fix dataset issue (#13098) 2023-09-13 23:04:16 +05:30