yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-14 04:29:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
10,673 Commits 676 Branches 144 Tags
Commit Graph

984 Commits

Author SHA1 Message Date
Pere Miquel Brull
ed0a01edea
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804) 
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
 2022-08-19 11:19:20 +02:00
Teddy
78b5f8c8e2
Part 1 of #5831 -- Profiler workflow implementation (#6809) 
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells
 2022-08-19 10:52:08 +02:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755) 
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
 2022-08-18 00:29:59 -07:00
Teddy
abaf8a84e9
Fixes #5661 by removing association between profiler and data quality (#6715) 
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs
 2022-08-17 12:53:16 +02:00
Nahuel
00ce67c01e
Fix#6643: Secure in secrets manager the dbtConfigSource object (#6743) 
* Secure in secrets manager the dbtConfigSource object

* Add missing tests

* Fix minor bug

* Minor code refactor
 2022-08-17 10:32:43 +02:00
Pere Miquel Brull
ad95bdb9c6
Fix #6735 - Improve source validation for inner connections (#6738) 
* Add inner connection validations and throw better error

* Use custom classes when retrieving metadata from Airflow to allow more versions
 2022-08-16 18:47:50 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added (#6513) 
* dagster-connector-added

* code-formatted

* dagster-comments-removed

* dagster-comments-removed

* dagster-unittest-added

* code-formatted-pr-comments-resolved

* improved-code-quality

* increased-py-test-time-out

* improved-dagster-tests

* improved-dagster-tests

* reverted-pytests-timeout

* dagster-pytest-improved

* added-schema-in-omdcore

* updated-schema-in-omd-core

* ui-schema-updated-omd-core

* updated-dagster-schema

* dagter-schema-test-updated
 2022-08-13 15:18:14 +05:30
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610) 
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
 2022-08-09 09:00:43 +02:00
Ayush Shah
a6db2e8a84
Fix for profiler: modified filter patterns and added error handling (#6608) 2022-08-08 10:43:17 +05:30
Milan Bariya
f3ec8c55a6
Clean col-level lineage internal logic (#6500) 
* Clean col-level lineage internal logic

* Fix: The Python checkstyle failed

* Fix: SonarCloud Bugfix

* Add test

* Fix: Add multiple queries

* Fix: Based on review comments

* Fix: make py_format issue

* Fix: pytest issue

* Fix: BUG FIX
 2022-08-05 20:48:28 +05:30
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579) 
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
 2022-08-05 10:28:04 +05:30
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416) 
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
 2022-08-04 07:22:47 -07:00
Mayur Singal
12f81c43f8
ES pipeline minor changes (#6565) 2022-08-04 15:22:09 +02:00
Pere Miquel Brull
6701ae6416
Fix #6489 - Update backup CLI (#6488) 
Fix #6489 - Update backup CLI (#6488)
 2022-08-03 12:01:57 +02:00
Pere Miquel Brull
e2916b2521
Organise ingestion module (#6515) 
Organise ingestion module (#6515)
 2022-08-03 09:55:50 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441) 
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
 2022-08-02 09:13:46 +02:00
Teddy
818736e2ca
Fix SQLite same thread error (#6486) 2022-08-01 17:33:53 +02:00
Teddy
7c70adc33e
Changed ometa_service_name to database_service_name (#6454) 2022-08-01 07:41:46 +02:00
Francisco J. Jurado Moreno
512b241069
[ISSUE-5522] Fix profiler timeout (#6448) 
* ChangeIsFunction by IsMethod

* Update test

* Remove whitespace
 2022-07-30 10:03:10 -07:00
Teddy
a920a4c17d
Fixed SQA Warning (#6442) 2022-07-30 09:39:36 -07:00
Teddy
6397b6a0b1
Fixes #6325 -- Implement multithreading for metrics computation (#6406) 
* Added tests for multithreading SQA interface

* Added multithread support for metric computation

* Added thread ID to log debuger

* Cleaned up tests

* Fixed python formatting issues

* Added non blocking result processing + threadCount in config file to set numbers of threads

* Added frontend input field to set number of threads

* Fixed code smell, bug and comments from reviewer
 2022-07-29 10:41:53 +02:00
Onkar Ravgan
7fdab21a72
Glue Topology Refactor (#6231) 
* rebase with main

* refactored glue for topology

* added glue unit tests

* resolved review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-07-28 15:04:43 +02:00
Mayur Singal
9c7d8f7d8f
Fix #5847: Handled Exception For Source Config (#6386) 
* Fix #5847: Handled Exception For Source Config

* Comment Fix

* Added Test Cases
 2022-07-27 20:21:16 +02:00
Mayur Singal
c6efe8c142
Fix #6158: Fivetran Connector Added (#6329) 
* Fivetran Connector Added

* Fivetran unit test added

* Fixed Unit Tests

* Added Fivetran Docs & Added Fivetran in Pipeline Listing
 2022-07-27 20:50:56 +05:30
Mayur Singal
9adaec1b3a
Fix #6248: Use PUT API instead of POST for tag & tagCategory (#6344) 
* Fix #6248: Use PUT API instead of POST for tag & tagCategory

* Restructure yield_tag topology

* Renamed method to create_or_update
 2022-07-27 08:41:55 +02:00
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager (#6330) 
* Implentation for retrieving auth provider config from Secret Manager

* Address PR comments

* Address code smells from SonarCloud

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-07-26 15:42:40 +02:00
Abhishek Pandey
bef4d412ce
uniontype-coverage-added (#6348) 2022-07-26 15:21:34 +02:00
Teddy
aae4410c93
Fies #6183 - Ability to set profile sample at the profilier workflow level (#6292) 
Fies  #6183 - Ability to set profile sample at the profilier workflow level (#6292)
 2022-07-25 12:08:20 +02:00
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs (#6307) 
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
 2022-07-24 18:49:15 +02:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props (#5700) 
Fix #5822 - Remove looker env and rename props (#5700)
 2022-07-24 16:02:26 +02:00
Francisco J. Jurado Moreno
78dd44ed32
[TASK-6140] Add OMeta header to SQLAlchemy-based queries (#6293) 
[TASK-6140] Add OMeta header to SQLAlchemy-based queries (#6293)
 2022-07-23 12:31:26 +02:00
Francisco J. Jurado Moreno
756dae5605
[TASK-6241] Cache ES ingestion when ingesting usage (#6276) 
* Cache ingestion

* Uncomment secrets manager

* Fix experiment stuff

* Fix style

* Add LRU cache

* Add tests

* Fix code smell
 2022-07-22 23:33:22 +02:00
Nahuel
4d4a2fc2cf
FIX#5916: Only admin user retrieves connection params when external Secrets Manager is configured (#6228) 
* Implementation for stop sending connection credentials when user is BOT

* Change way we add the connection to the service in the Secret Manager

* Services connection is not required as we want to stop returning it when SM is configured
 2022-07-22 14:02:29 -07:00
Teddy
28623c7455
Fixes #6078 - DB column named metadata conflict with SQA reserved attribute (#6282) 2022-07-22 19:05:32 +02:00
Mayur Singal
75ccc803ad
Fix #5410: Topology for Pipeline Services (#6085) 
Fix #5410: Topology for Pipeline Services (#6085)
 2022-07-22 13:59:14 +02:00
Teddy
893665f33d
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233) 
Fixes #6031 & #6236 -- Handle binary field & UUID when ingesting sample data (#6233)
 2022-07-21 20:14:44 +02:00
Teddy
e1fac99353
Fixes #5723 and implement interface processor logic (#6219) 
* Added datetime for min/max

* Added profiler interface

* Update core.py to work with profiler_interface

* Implement interface logic for orm_profiler object

* Fix unique_ratio logic

* removed changes to table.json

* Added Protocol for type hint

* Changed protocol to abc + fixed sonar code smell

* Fixed py_format
 2022-07-20 17:54:10 +02:00
Pere Miquel Brull
2936c0e4e3
Remove retry in ES query (#6191) 
Remove retry in ES query (#6191)
 2022-07-20 12:44:50 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128) 
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
 2022-07-19 14:51:44 +02:00
Pere Miquel Brull
9fa5d5cab2
Fix #6154 - Python Client to PATCH descriptions (#6165) 
* Prepare patch description utilities for python client

* Improvements
 2022-07-19 12:58:58 +02:00
Pere Miquel Brull
7b6332edef
Fix #6045 - Parse gracefully the connection on Test Connection (#6149) 
* Parse gracefully test connection

* Test file renames
 2022-07-18 18:50:27 +02:00
Pere Miquel Brull
479a8de486
Fix #5885 - Provider loading fails for airflow <2.3 (#5927) 
Fix #5885 - Provider loading fails for airflow <2.3 (#5927)
 2022-07-14 15:07:39 +02:00
Milan Bariya
954635b6dc
FIX 5907: Ingest Trino complex type (#5931) 
* FIX 5907: Ingest Trino complex type

* Included Row & Array Datatype

* Resolved review comments

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
 2022-07-14 11:28:50 +05:30
nna077
baa5295cc2
Add Date/Time Metrics In Profiler Tab (#5821) 
Add Date/Time Metrics In Profiler Tab (#5821)
 2022-07-13 21:23:03 +02:00
Teddy
d097199d2f
Added validation in profiler workflow to ensure service name exists and raise more explicit error (#6036) 
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-07-13 14:43:48 +02:00
Teddy
f0926d9b75
Fixes #3463 -- Implement Custom SQL Data Quality Tests (#5973) 
Fixes #3463 -- Implement Custom SQL Data Quality Tests (#5973)
 2022-07-12 06:21:11 +02:00
Mayur Singal
ebc9a24ee0
Connection options fix (#5971) 
* Connection Options Fix

* Fixed Condition
 2022-07-08 17:44:18 +02:00
Teddy
11487571b0
Fixes #5964 -- Trino Median function and columnValuesToMatchRegex wording (#5965) 
* added logic for trino median function

* Fixed columnValuesToMatchRegex result wording

* Replaced table with column for toMatchRegex test
 2022-07-08 15:55:50 +02:00
Teddy
48f6553fb3
Fixes #5866 -- Implement Support for Median Metric (#5887) 
* Added additional table + test coverage

* Added logic for front end input fields

* Added comment for median metric

* skipping `Update owner and check description` cypress test

* Added support to run window metrics for the profiler

* Fix except code smell

* moved median metric to windown folder

* Fix pyformat

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
 2022-07-07 06:58:42 +02:00