yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-11 11:09:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,962 Commits 661 Branches 144 Tags
Commit Graph

1157 Commits

Author SHA1 Message Date
Mayur Singal
9ef32eef99
Fix #6829: Improve usage filtering (#9613) 2023-01-05 15:07:03 +00:00
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector (#9554) 
* feat: update databricks_client

* feat: create pipelineconnection & connection

* test: add unit test

* feat: add examples file

* fix: remove unused imports

* style: fix python style

* fix: code review adjustments

* fix: remove unneeded schema ref

* fix: format requirements

* fix: change context calls

* test: add pipeline_status test

* format: pylint format change

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2023-01-05 14:14:12 +01:00
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588) 
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588)
 2023-01-04 08:50:53 +01:00
8vicat
7e7528e13b
Improve SQL lineage parsing (#9537) 
* CTEs support
* Supports inline comments
* Expands `*` in sources
 2023-01-03 09:27:45 +01:00
NiharDoshi99
8f69386076
Fix: refactor get_column_type (#9559) 
* Fix: refactor get_column_type

* Fix: added changes as per comments

* Fix: pylint

* Fix: minor changes

* Fix: minor changes
 2023-01-03 10:28:38 +05:30
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients (#9545) 2023-01-02 13:52:27 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package (#9521) 
Fix #6562 - Sources have their own package (#9521)
 2022-12-27 15:00:22 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs (#9260) 2022-12-26 12:32:17 -08:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers (#9513) 
Organise dynamic importers (#9513)
 2022-12-23 12:17:06 +01:00
Teddy
9e01fe0636
feat(CLI): added migration logic cmd (#9437) 2022-12-21 07:55:18 +00:00
Mayur Singal
07aed10a7e
Snowflake E2E Tests (#8781) 2022-12-20 18:00:26 +05:30
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number (#9104) 2022-12-20 14:55:11 +05:30
Milan Bariya
a3495c67f0
Databricks improve code and doc update (#9394) 
* Databricks improve code and doc update

* Change base on comments

* Fix pytest fail
 2022-12-19 18:53:10 +05:30
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test (#9363) 
* Prep airflow operator integration tests

* Add integration test to Makefile
 2022-12-16 19:52:12 -08:00
Mayur Singal
5164a55341
Fix Mysql e2e test (#9364) 2022-12-16 19:39:02 -08:00
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler (#9340) 
* feat(Profiler): Added timeout logic in the profiler workflow

* feat(Profiler): Added additional input in workflow UI

* fix(Profiler): timeout text blob in UI

* fix(Profiler): Return profile_results outside the context manager

* Fix(Profiler): Added localizer for timeout label and message

* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py

Co-authored-by: Nahuel <nahuel@getcollate.io>

Co-authored-by: Nahuel <nahuel@getcollate.io>
 2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322) 
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-12-16 07:43:18 +01:00
Onkar Ravgan
b539b299ee
Integrated schema parsers (#9305) 
* Integrated schema parsers

* Addressed review comments

* fixed pytests
 2022-12-15 16:54:55 +05:30
Ayush Shah
a6ae9fd11a
Add Test Suite Implementation for Datalake (#9235) 2022-12-14 21:14:51 +05:30
Teddy
d1a739ec55
Fixes #9025 -- Added deletion of WebAnalytics events in dataInsight Workflow (#9208) 2022-12-13 11:43:29 +01:00
Nahuel
9a4e3a7a46
Fix#8424: Remove brackets from tables and schemas on lineage (#9257) 
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test

* Remove brackets from tables and schemas on lineage
 2022-12-13 06:40:37 +01:00
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241) 
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241)
 2022-12-13 06:36:55 +01:00
Pere Miquel Brull
9a955036cf
Get sample data for e2e tests (#9252) 2022-12-12 15:08:01 +01:00
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state (#9206) 2022-12-10 19:54:41 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection (#9196) 
* Add Test Connection to Profiler

* remove Code Lint

* Fix AttributeError

* Fix Pytest

* Fix Bigquery Partition tests

* Fix Lint
 2022-12-10 17:16:28 +01:00
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159) 
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
 2022-12-07 14:33:30 +01:00
Sriharsha Chintalapani
25449001ca
Fix #9040: Remove fields such as tableQueries, tableProfile, tests, sample data as part of table fields (#9041) 2022-12-06 21:07:04 -08:00
Ayush Shah
2efa720f59
Fix E2E: new envs (#9143) 2022-12-05 19:15:12 +05:30
Pere Miquel Brull
58f3c01470
Fix #7056 - Automatic lineage from ML Model to ML Feature Data Source (#9113) 2022-12-05 16:43:03 +05:30
Pere Miquel Brull
1b3ff505c2
Fix #8858 - Add chart description and add lineage flexibility (#9124) 
Fix #8858 - Add chart description and add lineage flexibility (#9124)
 2022-12-02 16:22:09 +01:00
Nahuel
00375341f7
Fix#7690: Use ES to build Tables FQN when processing Lineage (#9079) 
* Use ES to build Tables FQN when processing Lineage

* Fix pylent check

* Fix test failing

* Remove warning when calling ES with empty query

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-12-01 09:56:42 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use (#9073) 
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test
 2022-11-30 16:02:21 +01:00
Ayush Shah
2b2eadf611
Fix Credentials sent via env (#9071) 
* Fix Credentials sent via env

* add new env var

* fix lint

* Add Comment
 2022-11-30 15:37:24 +01:00
Teddy
3cad959e44
Fixes #6760 -- Implements REGEX for regex test (#9033) 
feat(testCase): impelemented regex logic for test suite
 2022-11-29 13:00:28 +01:00
Ayush Shah
293de4a894
E2E for Bigquery (#9007) 2022-11-29 16:22:08 +05:30
Pere Miquel Brull
eccbb087d3
Add table query test (#9000) 
* Add table query test

* Add table query test
 2022-11-25 06:56:12 +01:00
Teddy
8789bdb847
Ometa mixin for ingestion pipeline (#8931) 
* Added sleep between data processing ingestion and KPI processing

* Added ometa mixin for ingestion pipeline and test suite
 2022-11-23 09:46:56 +00:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change (#8909) 2022-11-21 12:35:23 +01:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Pere Miquel Brull
b203a92565
Add config file env var loading tests (#8799) 
* Add env var load test

* Add env var load test
 2022-11-17 16:52:42 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849) 
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
 2022-11-17 16:44:26 +00:00
Teddy
4ab6dbc201
Data insight tests addition (#8814) 
* Added additional tests around web analytics and data insight

* Added playwright dependency installation

* removed -m no-ui option

* Removed playwright tests

* Added event ingestion in setUp class
 2022-11-17 10:44:52 +01:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Pere Miquel Brull
d1cc3e880c
Fix e2e test and add slack alert (#8817) 
* Fix mysql test

* Run alarm on failed test

* Force failure for testing
 2022-11-16 17:43:48 +01:00
Teddy
4ccdce50e0
Fix data insight bugs (#8761) 
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day

* Added time utils

* fixed timezone
 2022-11-15 16:10:30 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler (#8694) 
* DQ commit

* Add DL Profiler

* Fix Ingestion and Profliing pylint checks

* Fix Tests

* PyFormat files

* Fix Tests

* Resolve Comments

* Fix Tests and Format Files

* Resolve Comments

* Fix Pylint and Code smells

* Resolve Comments

* Fix S3 parquet

* Fix Metrics Code Smell
 2022-11-15 16:01:10 +01:00
Mohit Yadav
db0215f964
Mapping from kpi to dichart(One to one) (#8707) 
* Mapping from kpi to dichart(One to one)

* Mapping from kpi to dichart(One to one)

* Removed second tests creating KPI

* Fix some ui improve as well

* review comment changed function name

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
 2022-11-14 19:44:01 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682) 
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
 2022-11-13 11:59:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas (#8654) 
* Added topic parsers

* Fixed pylint

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-11-11 16:35:09 +05:30