yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-27 10:26:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
9,576 Commits 761 Branches 156 Tags
Commit Graph

843 Commits

Author SHA1 Message Date
Mayur Singal
5164a55341
Fix Mysql e2e test (#9364) 2022-12-16 19:39:02 -08:00
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler (#9340) 
* feat(Profiler): Added timeout logic in the profiler workflow

* feat(Profiler): Added additional input in workflow UI

* fix(Profiler): timeout text blob in UI

* fix(Profiler): Return profile_results outside the context manager

* Fix(Profiler): Added localizer for timeout label and message

* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py

Co-authored-by: Nahuel <nahuel@getcollate.io>

Co-authored-by: Nahuel <nahuel@getcollate.io>
 2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322) 
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-12-16 07:43:18 +01:00
Onkar Ravgan
b539b299ee
Integrated schema parsers (#9305) 
* Integrated schema parsers

* Addressed review comments

* fixed pytests
 2022-12-15 16:54:55 +05:30
Ayush Shah
a6ae9fd11a
Add Test Suite Implementation for Datalake (#9235) 2022-12-14 21:14:51 +05:30
Teddy
d1a739ec55
Fixes #9025 -- Added deletion of WebAnalytics events in dataInsight Workflow (#9208) 2022-12-13 11:43:29 +01:00
Nahuel
9a4e3a7a46
Fix#8424: Remove brackets from tables and schemas on lineage (#9257) 
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test

* Remove brackets from tables and schemas on lineage
 2022-12-13 06:40:37 +01:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241) 
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
 2022-12-13 06:36:55 +01:00
Pere Miquel Brull
9a955036cf
Get sample data for e2e tests (#9252) 2022-12-12 15:08:01 +01:00
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state (#9206) 2022-12-10 19:54:41 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection (#9196) 
* Add Test Connection to Profiler

* remove Code Lint

* Fix AttributeError

* Fix Pytest

* Fix Bigquery Partition tests

* Fix Lint
 2022-12-10 17:16:28 +01:00
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159) 
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
 2022-12-07 14:33:30 +01:00
Sriharsha Chintalapani
25449001ca
Fix #9040: Remove fields such as tableQueries, tableProfile, tests, sample data as part of table fields (#9041) 2022-12-06 21:07:04 -08:00
Ayush Shah
2efa720f59
Fix E2E: new envs (#9143) 2022-12-05 19:15:12 +05:30
Pere Miquel Brull
58f3c01470
Fix #7056 - Automatic lineage from ML Model to ML Feature Data Source (#9113) 2022-12-05 16:43:03 +05:30
Pere Miquel Brull
1b3ff505c2
Fix #8858 - Add chart description and add lineage flexibility (#9124) 
Fix #8858 - Add chart description and add lineage flexibility (#9124)
 2022-12-02 16:22:09 +01:00
Nahuel
00375341f7
Fix#7690: Use ES to build Tables FQN when processing Lineage (#9079) 
* Use ES to build Tables FQN when processing Lineage

* Fix pylent check

* Fix test failing

* Remove warning when calling ES with empty query

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-12-01 09:56:42 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use (#9073) 
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test
 2022-11-30 16:02:21 +01:00
Ayush Shah
2b2eadf611
Fix Credentials sent via env (#9071) 
* Fix Credentials sent via env

* add new env var

* fix lint

* Add Comment
 2022-11-30 15:37:24 +01:00
Teddy
3cad959e44
Fixes #6760 -- Implements REGEX for regex test (#9033) 
feat(testCase): impelemented regex logic for test suite
 2022-11-29 13:00:28 +01:00
Ayush Shah
293de4a894
E2E for Bigquery (#9007) 2022-11-29 16:22:08 +05:30
Pere Miquel Brull
eccbb087d3
Add table query test (#9000) 
* Add table query test

* Add table query test
 2022-11-25 06:56:12 +01:00
Teddy
8789bdb847
Ometa mixin for ingestion pipeline (#8931) 
* Added sleep between data processing ingestion and KPI processing

* Added ometa mixin for ingestion pipeline and test suite
 2022-11-23 09:46:56 +00:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change (#8909) 2022-11-21 12:35:23 +01:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Pere Miquel Brull
b203a92565
Add config file env var loading tests (#8799) 
* Add env var load test

* Add env var load test
 2022-11-17 16:52:42 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849) 
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
 2022-11-17 16:44:26 +00:00
Teddy
4ab6dbc201
Data insight tests addition (#8814) 
* Added additional tests around web analytics and data insight

* Added playwright dependency installation

* removed -m no-ui option

* Removed playwright tests

* Added event ingestion in setUp class
 2022-11-17 10:44:52 +01:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Pere Miquel Brull
d1cc3e880c
Fix e2e test and add slack alert (#8817) 
* Fix mysql test

* Run alarm on failed test

* Force failure for testing
 2022-11-16 17:43:48 +01:00
Teddy
4ccdce50e0
Fix data insight bugs (#8761) 
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day

* Added time utils

* fixed timezone
 2022-11-15 16:10:30 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler (#8694) 
* DQ commit

* Add DL Profiler

* Fix Ingestion and Profliing pylint checks

* Fix Tests

* PyFormat files

* Fix Tests

* Resolve Comments

* Fix Tests and Format Files

* Resolve Comments

* Fix Pylint and Code smells

* Resolve Comments

* Fix S3 parquet

* Fix Metrics Code Smell
 2022-11-15 16:01:10 +01:00
Mohit Yadav
db0215f964
Mapping from kpi to dichart(One to one) (#8707) 
* Mapping from kpi to dichart(One to one)

* Mapping from kpi to dichart(One to one)

* Removed second tests creating KPI

* Fix some ui improve as well

* review comment changed function name

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
 2022-11-14 19:44:01 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682) 
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
 2022-11-13 11:59:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas (#8654) 
* Added topic parsers

* Fixed pylint

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-11-11 16:35:09 +05:30
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617) 
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
 2022-11-11 09:59:15 +01:00
Mayur Singal
8c7c08ae5d
Fix py-cli-e2e-tests (#8662) 2022-11-11 08:55:34 +01:00
Teddy
9d594efcb5
Fixes #8473 - Implements KPI Workflow (#8657) 
* Added KPI workflow logic

* Update ingestion/src/metadata/data_insight/runner/run_result_registry.py

* ran py format

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-11-11 06:35:33 +01:00
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler (#8623) 
* Added logic to handle partitioning config in profiler

* extracted get_partition_details out of workflow classes
 2022-11-10 10:54:31 +01:00
NiharDoshi99
4ed088f55e
WIP: Fixed amundsen ingestion (#8544) 2022-11-09 14:21:44 +05:30
Milan Bariya
4d16be2608
Fix: Hive connection issue (#8583) 
* Fix: Hive connection issue

* Change based on comments
 2022-11-08 16:22:37 +01:00
michizhou
da06d183db
feat: add QuickSight connector (#8442) 2022-11-08 19:54:49 +05:30
Abhishek Pandey
73b370b5e2
schema-filter-added-in-datalake-for-bucket (#8516) 
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
 2022-11-08 10:57:16 +05:30
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events (#8528) 
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Fixed typos brought up during reviews
 2022-11-07 17:08:20 +01:00
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387) 
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
 2022-11-03 18:19:50 +01:00
Tushar Mittal
6f2c93089c
feat: add SageMaker connector (#8435) 
* feat: add sagemaker connector

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* fix: fix linting errors and update imports

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* test: add unit tests for sagemake source

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
 2022-11-03 18:19:20 +01:00
Pere Miquel Brull
c326cc1370
Move logging to DEBUG (#8519) 2022-11-03 13:04:33 +01:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector (#8452) 
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
 2022-11-02 16:12:45 +05:30