1489 Commits

Author SHA1 Message Date
Mayur Singal
c758d83276
Fix #9058: Add time elapsed in seconds instead of miliseconds (#9310) 2022-12-16 08:11:30 +00:00
Sriharsha Chintalapani
e42e7e41d8
remove norms for ES index mappings (#9334) 2022-12-16 07:43:13 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322)
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Ayush Shah
a633befe2a
Fix: Snyk Severity for Delta Spark (#9323) 2022-12-16 12:12:44 +05:30
Pere Miquel Brull
d2deb38766
Fix edge nodes (#9274) 2022-12-15 14:04:59 +01:00
Onkar Ravgan
b539b299ee
Integrated schema parsers (#9305)
* Integrated schema parsers

* Addressed review comments

* fixed pytests
2022-12-15 16:54:55 +05:30
NiharDoshi99
c2221fa596
Fix: Datalake s3 get list of files (#9294)
* Fix: changed func to get list of files

* Fix: minor change

* Fix: minor change

* Fix: adding exception
2022-12-15 14:33:25 +05:30
Mayur Singal
8fe3b09d21
Add Support for Endpoint URL in Datalake (#9293) 2022-12-15 07:30:32 +00:00
Mayur Singal
099853ab10
Fix Clickhouse Comments (#9295) 2022-12-14 21:42:06 +05:30
Ayush Shah
a6ae9fd11a
Add Test Suite Implementation for Datalake (#9235) 2022-12-14 21:14:51 +05:30
Sriharsha Chintalapani
38074f763b
Fix #8509: Add schema fields (#9209)
* Fix #8509: Add schema fields

* Fix #4675: Ingestion deployment from UI is broken

* Added sample data for topics

* Fixed pytests

* Address comments

* Refactored sampledata according to new schema

* Added return type

* Feat(ui)  : Add support for Avro editor (#9224)

* Feat(ui)  : Add support for Avro editor

* chore : add mock data for nested fields

* style: add group and opacity css

* add schema fields component to topic details and unit test

* Add locale keys

* Add support for edit description.

* refactor: expandableConfig

* test: add unit test for util method

* chore : make changes according to the schema change

* test : fix unit test and add new mock data

* chore : rename files

* Add row key

* chore : add default value for tags

* chore : update util method

* chore : add support for editing field tags

* chore : rename util files

* test : add unit test

* add comments

* addressing comments

* Address comments

* Added avro requirments

* Added requirement versions

* fixed versions

* protobuf version fix

* chore : rename util test file

* Fixed Dataype

* test: add unit test for schema component

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-14 19:56:37 +05:30
NiharDoshi99
b959709275
Fix: added datalake gen2 (#9245)
* Fix: added datalake gen2

* Fix: pytest

* Fix: pytest

* Fix: made changes to incoporate azure in datalake

* Fix: adlfs version

* Fix: adlfs version

* Fix: add tsv file type

* Fix: refractor for reading tsv file

* Fix: refractor for reading tsv file
2022-12-14 17:05:59 +05:30
Nahuel
0eb49c6100
Use ES call to get table entity (#9273)
* Use ES call to get table entity

* Address PR comments
2022-12-13 18:32:48 +00:00
Milan Bariya
c6b98751af
Logic improved for BYTES column (#9261)
* Logic improved for BYTES column

* Try Except block added

* cahange based on comments
2022-12-13 20:10:45 +05:30
Teddy
c7d0fede2f
Fixes #8184 Athena Profiler (#9270)
* fix(profiler): Athena Median

* fix: linting
2022-12-13 12:03:22 +00:00
Teddy
d1a739ec55
Fixes #9025 -- Added deletion of WebAnalytics events in dataInsight Workflow (#9208) 2022-12-13 11:43:29 +01:00
Teddy
387cd570f7
Fixes 9192 - Add entity link in response (#9239)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch

* feat: Added entity URL for most viewed entity

* feat(dataInsight): Added href to entity

* feat(DataInsight): Added backend logic for entity href

* Fix conflict resolving typos

* Update openmetadata-service/src/main/resources/json/data/dataInsight/mostViewedEntities.json

Co-authored-by: Nahuel <nahuel@getcollate.io>

* fix: addressed comments from PR review

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-13 10:39:09 +00:00
Onkar Ravgan
8ca555d959
Fix #8944: Added pagination logic for PowerBi (#9263)
* Added pagination logic

* Added optional keyword
2022-12-13 10:29:09 +01:00
Nahuel
9a4e3a7a46
Fix#8424: Remove brackets from tables and schemas on lineage (#9257)
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test

* Remove brackets from tables and schemas on lineage
2022-12-13 06:40:37 +01:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
2022-12-13 06:36:55 +01:00
Pere Miquel Brull
9a955036cf
Get sample data for e2e tests (#9252) 2022-12-12 15:08:01 +01:00
Sriharsha Chintalapani
145e9d48f6
Fix #9227: Fix metabase connector to work with non-admin user (#9228)
* Fix #9227: Fix metabase connector to work with non-admin user

* Fix #9227: Fix metabase connector to work with non-admin user

* fix pycheck

* fix pycheck

* fix pycheck
2022-12-12 07:56:58 +01:00
Ayush Shah
2e1fb96751
Add LRU to ES fqn (#9233) 2022-12-12 06:46:15 +00:00
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state (#9206) 2022-12-10 19:54:41 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection (#9196)
* Add Test Connection to Profiler

* remove Code Lint

* Fix AttributeError

* Fix Pytest

* Fix Bigquery Partition tests

* Fix Lint
2022-12-10 17:16:28 +01:00
Teddy
4a5cb870ec
Fixes #9192 - Update ES response for DI (#9221)
* staging commit

* updated terms to french

* fix(dataInsight): updated ES response to get entityType and total non tiered entities

* fix(dataInsight): removed fields=*

* fix(dataInsight): Added entity migration

* cleaned up branch
2022-12-09 18:53:17 -08:00
Pere Miquel Brull
b1565b8337
Fix #9181 - Handle failed callback status without workflow (#9193)
* Handle failed status without workflow

* Add log
2022-12-09 10:42:24 +01:00
Milan Bariya
6f12c971e1
Add: function to get different catalogs (#9179)
* Add: function to get different catalogs

* Change Based on comments

* Change Based on comments

* Change Based on comments
2022-12-08 15:42:58 +05:30
NiharDoshi99
96e77af8a4
Fix: #1308 Docs and Atlas Connector Improvement (#9072)
* Fix: docs changes

* Fix: changes as per comment

* Fix: atlas connector

* Fix: as per comments

* Fix: as per comments
2022-12-08 12:14:15 +05:30
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159)
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Sriharsha Chintalapani
25449001ca
Fix #9040: Remove fields such as tableQueries, tableProfile, tests, sample data as part of table fields (#9041) 2022-12-06 21:07:04 -08:00
Suresh Srinivas
16d97d6f96
Fix typos in JSON schemas and follow standard naming conventions (#9183) 2022-12-06 20:50:34 -08:00
Teddy
f86186d5b0
Fixed time field type for profiler (#9177) 2022-12-06 18:30:48 +01:00
07Himank
b5b29c93f5
Track new primary key (#9145) 2022-12-06 09:13:07 +05:30
Mayur Singal
81cbf93df5
Fix #9139: Fix shcema filter being added twice to status (#9148) 2022-12-05 15:05:55 +00:00
Ayush Shah
2efa720f59
Fix E2E: new envs (#9143) 2022-12-05 19:15:12 +05:30
Pere Miquel Brull
58f3c01470
Fix #7056 - Automatic lineage from ML Model to ML Feature Data Source (#9113) 2022-12-05 16:43:03 +05:30
NiharDoshi99
e0a3f3481b
Fix: aggregate type (#9138) 2022-12-05 14:50:37 +05:30
Sriharsha Chintalapani
14d82ee7b2
Fix #6463: Add support for default suggestions for suggestion API (#9090) 2022-12-02 10:31:37 -08:00
Pere Miquel Brull
1b3ff505c2
Fix #8858 - Add chart description and add lineage flexibility (#9124)
Fix #8858 - Add chart description and add lineage flexibility (#9124)
2022-12-02 16:22:09 +01:00
Onkar Ravgan
23468fb868
Fixed Broken docs links (#9125) 2022-12-02 15:42:06 +01:00
NiharDoshi99
ead5ce7227
Fix: dagster keyerror solved (#9099) 2022-12-01 18:43:11 +05:30
Pere Miquel Brull
7d2eab332d
Lineage Backend example and fix (#9095) 2022-12-01 13:09:36 +01:00
Milan Bariya
2f9f169de9
Fix: Safer default Mark Deleted Tables (#9065)
* Fix: Safer default Mark Deleted Tables

* Fix: Change Based On Comments

* Fix: Change Based On Comments

* Add: Migration Files

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-01 16:26:43 +05:30
Nahuel
00375341f7
Fix#7690: Use ES to build Tables FQN when processing Lineage (#9079)
* Use ES to build Tables FQN when processing Lineage

* Fix pylent check

* Fix test failing

* Remove warning when calling ES with empty query

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-01 09:56:42 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use (#9073)
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test
2022-11-30 16:02:21 +01:00
Ayush Shah
2b2eadf611
Fix Credentials sent via env (#9071)
* Fix Credentials sent via env

* add new env var

* fix lint

* Add Comment
2022-11-30 15:37:24 +01:00
Abhishek Pandey
e8edf70374
removed-whitespace-from-postgres-and-redshift (#8927) 2022-11-30 13:16:24 +05:30
Pere Miquel Brull
6a3ce624b7
Fix #9008 - Add Airflow callback, introduce Workflow Status Mixin and remove retries (#9049)
Fix #9008 - Add Airflow callback, introduce Workflow Status Mixin and remove retries (#9049)
2022-11-30 08:30:45 +01:00
Sriharsha Chintalapani
125fefcf6d
Fix logging metadata webhook event (#9067) 2022-11-29 20:57:43 -08:00