1157 Commits

Author SHA1 Message Date
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mayur Singal
8c7c08ae5d
Fix py-cli-e2e-tests (#8662) 2022-11-11 08:55:34 +01:00
Teddy
9d594efcb5
Fixes #8473 - Implements KPI Workflow (#8657)
* Added KPI workflow logic

* Update ingestion/src/metadata/data_insight/runner/run_result_registry.py

* ran py format

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-11 06:35:33 +01:00
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler (#8623)
* Added logic to handle partitioning config in profiler

* extracted get_partition_details out of workflow classes
2022-11-10 10:54:31 +01:00
NiharDoshi99
4ed088f55e
WIP: Fixed amundsen ingestion (#8544) 2022-11-09 14:21:44 +05:30
Milan Bariya
4d16be2608
Fix: Hive connection issue (#8583)
* Fix: Hive connection issue

* Change based on comments
2022-11-08 16:22:37 +01:00
michizhou
da06d183db
feat: add QuickSight connector (#8442) 2022-11-08 19:54:49 +05:30
Abhishek Pandey
73b370b5e2
schema-filter-added-in-datalake-for-bucket (#8516)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-11-08 10:57:16 +05:30
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events (#8528)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Fixed typos brought up during reviews
2022-11-07 17:08:20 +01:00
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387)
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
2022-11-03 18:19:50 +01:00
Tushar Mittal
6f2c93089c
feat: add SageMaker connector (#8435)
* feat: add sagemaker connector

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* fix: fix linting errors and update imports

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* test: add unit tests for sagemake source

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-03 18:19:20 +01:00
Pere Miquel Brull
c326cc1370
Move logging to DEBUG (#8519) 2022-11-03 13:04:33 +01:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector (#8452)
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-02 16:12:45 +05:30
Teddy
989f2911c2
Fixes #7810 - Allow to only pass min or max (#8474)
* ISSUE-7810 Added default values for min and max
For all data validations on columns:-
min_bound is set to float("-inf"), if there is no next value
max_bound is set to float("inf"), if there is no next value

* Fixed PR errors by removing tuple + added tests

Co-authored-by: demi <deepak1212365@gmail.com>
2022-11-01 13:26:51 +01:00
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse (#8182) 2022-10-31 18:12:26 +05:30
Pere Miquel Brull
6d404ccdc1
Domo Pipeline - Use pipeline ID with display name (#8374)
* Use pipeline ID with display name

* handle id

* lint

* update test for ID
2022-10-26 21:22:26 +02:00
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Nahuel
1583adee29
Fix lineage to support 'merge into' queries starting with comments (#8355) 2022-10-25 10:29:59 -07:00
Nahuel
36b12bd6f1
Fix lineage issues with merge_into and copy grants queries (#8335) 2022-10-24 17:22:22 +02:00
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files (#8304) 2022-10-21 08:38:52 -07:00
Milan Bariya
1565aa7733
FIX: Trino&Presto catalogs as databases (#8189)
* FIX: Trino&Presto catalogs as databases

* FIX: Trino&Presto catalogs as databases

* Change based on comments
2022-10-18 20:00:17 +05:30
NiharDoshi99
84a89d7181
Added DOMO Pipeline (#8133) 2022-10-18 17:52:28 +05:30
Milan Bariya
5b3c877dd1
FIX:connection failed because of having '@' character in uname and pass (#8172) 2022-10-17 16:57:55 +05:30
Pere Miquel Brull
d48fd468d4
Tag PATCH + Cleanup of helpers methods (#8150)
* cleanup

* lint

* Add tag patch

* Fix rename

* Dont kill tests
2022-10-15 14:56:30 +02:00
Milan Bariya
e7a3770240
Add: databricks usage and lineage added (#7386) 2022-10-14 07:06:28 +05:30
NiharDoshi99
0e44f7d35d
Fixed: Intergrated DOMO Connector APIs (#7989) 2022-10-13 10:55:48 +05:30
Teddy
f883863b8a
Fixes #7490 - Split Profiler and TestSuite Interface (#8032)
* Clean up test suite workflow and interface

* Fixed tests

* Split profiler and testSuite interfaces

* Cleaned up workflows and runners

* Fixed code formatting

* - remove old code
- remove `table` attribute used for testing and used mock instead

* Fixed execution bugs from refactor

* Fixed static type checking for profiler/api/workflow.py

* Fixed linting

* Added __init__ files
2022-10-11 15:57:25 +02:00
Ashish Gupta
8ddab79dc1
UI :- Remove rc-tree and react-table dependencies (#8049)
* Remove rc-tree and react-table dependencies

* fix unit test issue

* File Formatted

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-10-11 15:42:58 +05:30
Teddy
3b7f576d04
Fixed linting for Profiler (#7922)
* - Fixed linting
- Added logic to skip partition check when engine is not BQ
- Added ingestion partition logic to testSuite

* Fixed python formating

* Fixed test for BQ partition
2022-10-11 09:36:36 +02:00
Pere Miquel Brull
f031293af0
Fix Pylint ingestion/bulksink and ingestion/stage (#8030)
* Pylint ingestion bulksink

* Pylint stage

* Remove test

* Remove stage
2022-10-10 14:06:43 +02:00
Nahuel
5c499d2a7a
Add CLI E2E tests for MySQL (#8041)
* Add CLI E2E tests for MySQL

* Fix setup.py and pylint

* Add missing doc and update code after pylint refactor
2022-10-10 11:36:20 +02:00
Teddy
e244a241c3
added logic to clean up quotes from table constraints (#7959) 2022-10-05 16:09:33 +02:00
Milan Bariya
cd43f1fdcf
Fix: connector for hive has double dash (#7953) 2022-10-05 12:01:13 +02:00
Teddy
15f7c4aa41
Fix param name for median test (#7942)
* Fixed param name for median test

* Fixed unite test for median DQ
2022-10-05 06:32:28 +02:00
Teddy
63b5233fd3
Added util command to overwrite dag_generated_config path in DAG file (#7831)
* Added util command to overwrite dag_generated_config path in DAG file

* Added logic to only write file on change
2022-10-03 09:24:47 +02:00
Nahuel
593ca3a4a0
Fix bot creation and edition logic (#7796)
* Fix bot creation and edition logic

* Fix minor error creating user

* Fix failing test

* Minor fixes

* Add missing tests for new flows

* Fix put_failIfBotHasARelationshipToAnotherUser test

* Changes after manual testing

* Move where auth_provider is retrieved in the secret manager
2022-10-01 11:25:56 +02:00
Teddy
962866a30e
Fixes #7280 - Implements support for BYTEA DType (#7749)
* Added support for postgres bytea data type

* Added BYTEA support for profiler and converts bytes to hex on results read
2022-09-28 13:28:35 +02:00
Nahuel
df909703eb
Fix#6615: Fix failing test still in CI (#7725)
* Fix failing test still some tests are failing

* Fix bug in service entity repository

* Fix all tests except Ingestion Pipeline integration tests

* Fix Ingestion Pipeline integration tests

* Fix failing Python tests
2022-09-26 20:39:42 +02:00
Ayush Shah
56c71c4101
Fix Ingestion Time Partitioned Tables on Profiler (#7506) 2022-09-26 19:41:40 +05:30
Onkar Ravgan
35efd49256
Added control for DBT descriptions (#7653)
* Added control for DBT descriptions

* Fixed tests

* Added UI changes

* fixed maven ci tests

* Java formatting changes

* ui review fixes

* Fixed pytests

* Fixed python integration tests

* fixed airflow tests

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-09-26 16:19:47 +05:30
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API (#7579) 2022-09-23 17:46:40 +05:30
Teddy
f2bf5194bb
Fixes #7623 -- Added logic to encode and decode entityLink (#7670)
* Encode entityLink string when processing request

* Added logic to decode column type from entityLink

* mvn code formating

* Extracted unquote step into its own function
2022-09-23 09:42:33 +02:00
Teddy
c34281251f
Fixes #7509 - implement import path migration script (#7515)
* Added script to migarte import

* Added tests

* Updated doc to include info about import migration

* renamed test resource files
2022-09-22 14:43:25 +02:00
Ayush Shah
517e9a4aac
Remove unused imports (#7657) 2022-09-22 16:42:10 +05:30
Sriharsha Chintalapani
adaa8e7493
Fix #6917: startTs & endTs filters providing paginated data (#7617)
* Fix #6917: startTs & endTs filters providing paginated data

* Updated ui side changes

* Fixed OM tests end to end test

* Fix #6917: startTs & endTs filters providing paginated data

* Fix #6917: startTs & endTs filters providing paginated data

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-09-21 13:33:47 -07:00
Pere Miquel Brull
1578fa7f1b
Fix #7121 - Support Spark metastore DB connection (#7520)
* Fix #7121 - Support Spark metastore DB connection

* appname

* Update docs

* test validation

* Address PR comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-09-20 16:47:57 +02:00
Nahuel
2a6c6134f4
Fix#7272: Improve logging when initializing workflow from CLI (#7522)
* Improve logging when initializing workflow from CLI

* Fix broken tests
2022-09-19 08:00:00 -07:00
Teddy
1ba6e284fe
Fixes #7118 by cleaning up test names (#7494)
* Cleaned up tests names and add registry name tests

* Updated documentation for test types supported by OM
2022-09-16 07:04:56 +02:00
Teddy
3fdbe6ecfb
Added logic to report column level metric computation failure for profiler (#7465) 2022-09-15 17:39:29 +02:00