2362 Commits

Author SHA1 Message Date
Ayush Shah
04760177f6
Fixes 13321: Fix Test Connection timeout (#13323) 2023-09-25 15:17:38 +05:30
Mayur Singal
65f65137e6
Fix: Bigquery query log not picked up for multiproject (#13313) 2023-09-25 08:07:48 +05:30
Teddy
e9ef7b5e81
Issue-12857: Remove ES Dependency from DI Workflow (#13303)
* feat: move elasticsearch indexing to backend + introduced EntityTimeSeries interface for timeseries type object

* feat: make reportData.json inherit from EntityTimeSeriesInterface

* feat: updated type to Object

* feat: deleted elasticsearch dependencies

* feat: removed elasticsearch indexing from workflow

* feat: added data insight sample data

* feat: cleaned up tests
2023-09-21 16:17:47 -07:00
William Geuns
270a923eb9
Fixes #10096: adding RuntimeDataBatchSpec as a GX backend (#13264)
* test

* support different gx backend

* remove unused import

* fix formatting

* Update ingestion/src/metadata/great_expectations/action.py

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

---------

Co-authored-by: William Geuns <william.geuns@vrt.be>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-09-21 21:55:29 +02:00
Onkar Ravgan
8acebbb892
Update ingestion logic to use PATCH API for lifeCycle info (#13283) 2023-09-21 16:40:09 +05:30
Ayush Shah
386b7e9405
Fix Storage Key in response (#13273) 2023-09-20 15:57:08 +05:30
Sriharsha Chintalapani
e0ecf49585
Fix #11970: Search by FQN; Refactor Search Indexing, Add API to searc… (#13271)
* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Fix #11970: Search by FQN; Refactor Search Indexing, Add API to search for specific field

* Add wildcard support

* Fix GlossaryTerm Patch
2023-09-20 14:40:10 +05:30
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts (#13247)
* Rename sandboxDomain to instanceDomain

* Test Get Charts in DomoDashboard

* Fix schemas

* Fix test

* Fix test

* Rename to Auto Tag PII

* Fix query test

* Fix query test

* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Pere Miquel Brull
7d711dc680
Fix Stored Procedures - Don't get queries without context (#13257) 2023-09-19 12:02:29 +02:00
Pere Miquel Brull
22b0f44e38
Part of #12998 - Query Service & Lineage filter processed queries (#13215)
* Update mapping

* Prep

* prep

* Prep query migration

* Add query index deletion fix

* Docs and Maven CI fix

* Fix tests

* Add service filter

* Add query entity FQN col migration

* Fix lint

* supported serviceFQN in query api

* Prep repo

* Prep ES query search

* Do not recompute lineage

* Format

* Fix test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-09-19 07:37:47 +02:00
Onkar Ravgan
e2a6d0cd71
Added caching for life cycle query (#13203) 2023-09-18 12:38:07 +05:30
Onkar Ravgan
cc47f5618f
Added SSL support to Superset (#13130) 2023-09-18 10:45:21 +05:30
Pere Miquel Brull
442528267c
Simplify topology & update context management (#13196) 2023-09-15 09:44:42 +02:00
Pere Miquel Brull
5edad7b418
Part of #12998 - Add Stored Procedures support for Redshift (#13174)
* Part of #12998 - Add Stored Procedures support for Redshift

* Lint

* Fix bq queries

* Update ingestion/src/metadata/ingestion/source/database/bigquery/queries.py
2023-09-14 15:32:05 +02:00
Ayush Shah
387128f1d2
Bigquery: Add Table Level Tags, fix dataset issue (#13098) 2023-09-13 23:04:16 +05:30
Teddy
d4593e9caa
fix: implement percentile computation logic for SingleStore (#13170) 2023-09-13 16:32:55 +02:00
Teddy
a84ea5e548
fix: set engine for each project ID (#13176) 2023-09-13 14:17:39 +02:00
Ayush Shah
5fea08cd33
Datalake: Add manifest file support, fix profiler metrics, add array and json column type support (#13017) 2023-09-13 15:15:49 +05:30
Pere Miquel Brull
42393f5de3
Part of #12998 - Add Stored Procedures support in BigQuery (#13160)
* Fix snowflake SP source url

* Update sourceUrl on PUT

* Add BigQuery Stored Procedures support

* Linting

* Linting

* lint

* Linting
2023-09-13 08:47:45 +02:00
Onkar Ravgan
1e4d48a034
Added Life Cycle sample data and changed datetime to timestamp (#13141) 2023-09-13 10:59:19 +05:30
Pere Miquel Brull
f0995cbddc
Part of #12998 - Prep Stored Procedures Skeleton for Snowflake (#13121)
* Prep Stored Procedures Skeleton for Snowflake

* Update pylint and add migrations

* Fix test

* Reuse source url computation
2023-09-12 14:25:42 +02:00
Mayur Singal
2e3e50f0cb
Add used by field in query entity (#13096) 2023-09-11 17:30:34 +05:30
Pere Miquel Brull
66a89ffe61
Fix e2e profiler status (#13127) 2023-09-10 18:46:28 +02:00
Mayur Singal
4e633877b3
Fix ElasticSearch Test Connection & Deploy (#13061) 2023-09-08 12:40:48 +05:30
Mayur Singal
ba09f874df
Fix usage count issue (#13109) 2023-09-08 10:36:29 +05:30
Onkar Ravgan
d8e78a32d8
Added Life Cycle fields, API and ingestion integration (#13014)
* Added initial code

* temp commit

* Added snowflake changes

* Added postgres migration

* removed non req changes

* review comments one

* Added ometa tests

* lint fix

* fix lint naming

* fixed merge changes
2023-09-06 12:37:55 +02:00
Pere Miquel Brull
de7e06d024
Update structure for PII processing (#13079)
* Update structure for PII processing

* Fix tests

* Fix tests

* Lint

* Remove typo
2023-09-06 11:30:46 +02:00
Pere Miquel Brull
a9b13de893
Move workflows back into src (#13100)
* Move workflows back into src

* Fix test
2023-09-06 11:30:33 +02:00
Teddy
c52af7eba0
fix: added upsert logic back for system metrics (#13092) 2023-09-06 08:12:38 +02:00
Onkar Ravgan
9af14a5a1c
fixed either in lineage req (#13088) 2023-09-05 18:18:16 +05:30
Mayur Singal
011eaf8ad5
Fix #13001: Fix query not populating to all tables (#13004) 2023-09-05 14:16:37 +05:30
Onkar Ravgan
a3ca8b6e66
Added project filter pattern to dashboard entity (#12925) 2023-09-05 11:01:28 +05:30
vanshika18
db592a157e
fix #12518 support get column lineage in spline (#13030) 2023-09-04 17:16:04 +05:30
Pere Miquel Brull
a3bfd4e696
Part of #11968 - Restructure Profiler Workflow and PII Processor (#13059)
* Structure PII

* Restructure Profiler Workflow

* Update signature for abc

* remove profiler sink

* Fix tests

* Fix lint

* Fix test

* Fix test
2023-09-04 11:02:57 +02:00
Mayur Singal
029786d773
Fix #9207: Add source in lineage details (#13046) 2023-09-04 11:05:56 +05:30
Onkar Ravgan
d24155e173
rmv fqn field tableau (#13064) 2023-09-02 12:00:33 +02:00
Teddy
0ef085cfc2
fix: summary for logical test suite (#13005)
* fix: summary for logical test suite + DI aggregation on OS

* fix: added logic to attached test case result summary to logical test suite

* fix: test case timestamp in ingestion to milliseconds

* fix: updated entityExtensionTimeSeries DAO to dataQualityDataTimeSeries DAO

* fix: revert aggregation changes
2023-09-01 18:01:59 +02:00
Teddy
ab3042e8ee
Issue 12297 bis -- Delete Insert logic in the DI workflow (#13058)
* fix: updated ingestion to delete existing data for the ingestion day

* fix: added delete endpoint for the report data

* fix: added migration to delete duplicate data + added tables creation & data deltion logic in a transaction

* fix: made deletion SQL engine aware

* fix: added ES deletion logic back while we work on DI event publisher
2023-09-01 18:01:20 +02:00
gauthk6
ba2201f4ea
draft: implementation of lightdash connector. (#12957)
* Fixes a bug while patching the description of a TestCase

* Update docker-compose.yml

* Update docker-compose.yml

* Ran pre-commit checks and linter

* Added some clarifying points and fixed some grammatical errors in the documentation for installation instructions.

* revert changes made to docs

* implementation of lightdash connector.

* ui: add icon for lightdash service

* chore: update lightdash image

* chore: update the icon

* Cleaned up code (took out debug statements, etc.). Still TODO: yield_dashboard_lineage_details() not being called just yet.

* fix: ran linting

* added null checks

* Delete openmetadata-server.md

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-08-31 17:28:07 +02:00
Pere Miquel Brull
a5667bb585
Fix import (#13044) 2023-08-31 08:30:18 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management (#13029)
* Prep changes

* Prep changes

* prep changes

* Update imports

* Format

* Prep delete

* Prep delete

* Fix sink

* Prep test

* Commit

* passing either

* passing either

* Prep Either

* Metadata source with Either

* Update status

* Merge remote-tracking branch 'upstream/main' into issue-7272

* Format

* Linting

* Linting

* Linting

* Linting

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Comments
2023-08-30 15:49:42 +02:00
vanshika18
3cc15e6d0b
Fixed #8418 added new connector couchbase (#12836) 2023-08-29 11:16:32 +05:30
Teddy
101cd0ebac
Issue 8930 - Update profiler timestamp from seconds to milliseconds (#12948) 2023-08-25 08:47:16 +02:00
Sriharsha Chintalapani
19b5c946a7
Fix #12167: Support for Stored Procedures as another entity under Database Schema (#12999)
* Add Stored Procedure Entity

* Stored Procedure repository

* Stored Procedure repository

* Fix #12998: Support for Stored Procedures as another entity under Database Schema

* Fix #12998: Support for Stored Procedures as another entity under Database Schema
2023-08-25 08:14:30 +02:00
vanshika18
496abf0392
Fixed #12445 support lineage for clickhouse materialized views (#12938) 2023-08-24 17:48:44 +05:30
vanshika18
52153dba98
fixed #10691 support multiDatabase, usage & lineage support for Azure SQL (#12982) 2023-08-24 17:47:41 +05:30
Ayush Shah
4464df81ff
Add include views to profiler (#12974) 2023-08-23 17:30:44 +05:30
Ayush Shah
1e8ea26bbd
Refactor converter, unique count, add registry (#12939) 2023-08-23 11:43:12 +05:30
Mayur Singal
817b9853df
Oracle add filter for lineage query (#12945) 2023-08-22 09:46:37 +02:00