yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-09-14 03:21:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
10,284 Commits 788 Branches 159 Tags
Commit Graph

2362 Commits

Author SHA1 Message Date
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805) 
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
 2022-08-19 16:15:40 +02:00
Mayur Singal
7c004a6785
Fix #6823: Fix hive attribute error (#6824) 
* Fix #6823: Fix hive attribute error

* Empty commit to start test
 2022-08-19 14:22:54 +02:00
Pere Miquel Brull
ed0a01edea
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804) 
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
 2022-08-19 11:19:20 +02:00
Pere Miquel Brull
6f3cd4cfed
Fix #6818 - Superset code cleanup and docs (#6821) 
* code cleaning

* superset docs

* Fix metabase docs
 2022-08-19 11:19:06 +02:00
Teddy
78b5f8c8e2
Part 1 of #5831 -- Profiler workflow implementation (#6809) 
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells
 2022-08-19 10:52:08 +02:00
Onkar Ravgan
1ec0fa3e37
Fixed PowerBi Exception (#6803) 
* Fixed exception for chart name

* added id in exception

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-18 19:47:56 +02:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Ayush Shah
399a2a103d
Fix profiler item_type sqlalchemy datatype issue (#6794) 2022-08-18 18:30:00 +05:30
NiharDoshi99
f552960ca7
Clean Atlas, Amundsen and metadata_rest (#6649) 
* Clean Atlas, Amundsen and metadata_rest

* FIX: Based on comment

* Added test case for coverage

* Fixed: Py test

* FIX: Added atlas yaml mapping file

* FIX: Py Test

* FIX: Revert Test

* FIX: Removed as per dataset
 2022-08-18 16:18:07 +05:30
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755) 
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
 2022-08-18 00:29:59 -07:00
Milan Bariya
1417541a81
Dashboard Service Lineage Issue For Metabase (#6679) 2022-08-18 06:50:48 +02:00
Teddy
abaf8a84e9
Fixes #5661 by removing association between profiler and data quality (#6715) 
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs
 2022-08-17 12:53:16 +02:00
Mayur Singal
66a934c837
Fix #4694: Refactor Source Status (#6480) 
* Fix #4694: Refactor Source Status

* Using Calculate Time Decorator

* Fixed Decorator

* Fixed Decorator

* perf_counter

* Move Calculate Sucesss to Soruce

* make py_format

* Fixed Sample Data

* Fixed Pipeline, Messaging, Dashboard & MlModel Service

* Fix ZeroDivisionError
 2022-08-17 14:39:50 +05:30
Nahuel
00ce67c01e
Fix#6643: Secure in secrets manager the dbtConfigSource object (#6743) 
* Secure in secrets manager the dbtConfigSource object

* Add missing tests

* Fix minor bug

* Minor code refactor
 2022-08-17 10:32:43 +02:00
Onkar Ravgan
424239840f
Added result limits to usage queries (#6748) 
* Added result limits to queries

* Fixed formatting

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-17 08:07:39 +02:00
Pere Miquel Brull
ad95bdb9c6
Fix #6735 - Improve source validation for inner connections (#6738) 
* Add inner connection validations and throw better error

* Use custom classes when retrieving metadata from Airflow to allow more versions
 2022-08-16 18:47:50 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added (#6513) 
* dagster-connector-added

* code-formatted

* dagster-comments-removed

* dagster-comments-removed

* dagster-unittest-added

* code-formatted-pr-comments-resolved

* improved-code-quality

* increased-py-test-time-out

* improved-dagster-tests

* improved-dagster-tests

* reverted-pytests-timeout

* dagster-pytest-improved

* added-schema-in-omdcore

* updated-schema-in-omd-core

* ui-schema-updated-omd-core

* updated-dagster-schema

* dagter-schema-test-updated
 2022-08-13 15:18:14 +05:30
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568) 
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-12 13:24:58 +05:30
Onkar Ravgan
af33cc0a31
Fixed Tableau ingestion issue (#6671) 
* Fixed Tableau ingestion issue

* revmoved doc string

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-09 23:55:32 +05:30
Sriharsha Chintalapani
60fe344aaf
Fix #6657: Change PathParam ID type in Resources from String to UUID (#6658) 2022-08-09 05:57:39 -07:00
Milan Bariya
c38ab8c8aa
Support for AWS-delegated credentials for Elasticsearch (#6644) 
* Support for AWS-delegated credentials for Elasticsearch

* FIX: Based on comments

* FIX: Based on comments

* FIX: Based on comments
 2022-08-09 12:36:16 +05:30
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610) 
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
 2022-08-09 09:00:43 +02:00
Abhishek Pandey
77ad1b1c87
close-connection-after-each-batch-using-with (#6641) 2022-08-09 11:15:56 +05:30
Abhishek Pandey
a08201e383
multiple-database-service-names-added-in-dashboard (#6591) 
* multiple-database-service-names-added-in-dashboard

* code-formatted

* code-smell-removed

* code-formatted

* improved-dbserviceName-handling-in-dashboard

* fix checks failing

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
 2022-08-08 22:21:28 +05:30
Mayur Singal
997486f1bb
Fix #6540: Added Exception handling in metadata_rest (#6650) 
* Added Exception handling in metadata_rest

* Added HTTPException
 2022-08-08 20:25:31 +05:30
Ayush Shah
a6db2e8a84
Fix for profiler: modified filter patterns and added error handling (#6608) 2022-08-08 10:43:17 +05:30
Milan Bariya
f3ec8c55a6
Clean col-level lineage internal logic (#6500) 
* Clean col-level lineage internal logic

* Fix: The Python checkstyle failed

* Fix: SonarCloud Bugfix

* Add test

* Fix: Add multiple queries

* Fix: Based on review comments

* Fix: make py_format issue

* Fix: pytest issue

* Fix: BUG FIX
 2022-08-05 20:48:28 +05:30
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579) 
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
 2022-08-05 10:28:04 +05:30
Onkar Ravgan
3d93aaad53
Moved view lineage to post process (#6585) 
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-04 20:03:17 +02:00
Teddy
b37bca5471
Fixes #2954 -- Ingest sample data for profile, test suite and co. (#6586) 
Fixes #2954 -- Ingest sample data for profile, test suite and co. (#6586)
 2022-08-04 19:44:21 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416) 
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
 2022-08-04 07:22:47 -07:00
Mayur Singal
12f81c43f8
ES pipeline minor changes (#6565) 2022-08-04 15:22:09 +02:00
Milan Bariya
45e6c2c7d7
Sample Data Ingestion use Create request (#6546) 
* Sample Data Ingestion use Create request

* Fix: Code smell

* Fix: make_pyformat

* Fix: Changed based on comments
 2022-08-04 11:13:48 +02:00
Mayur Singal
5754b1424a
Bigquery Docs Min Requirements (#6561) 2022-08-04 10:29:45 +02:00
Mayur Singal
91d9e810ea
Revert Rename on MetadataEs (#6537) 2022-08-03 19:36:09 +05:30
Pere Miquel Brull
6701ae6416
Fix #6489 - Update backup CLI (#6488) 
Fix #6489 - Update backup CLI (#6488)
 2022-08-03 12:01:57 +02:00
Pere Miquel Brull
e2916b2521
Organise ingestion module (#6515) 
Organise ingestion module (#6515)
 2022-08-03 09:55:50 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441) 
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
 2022-08-02 09:13:46 +02:00
Teddy
a7fffc778c
Fixes #6476 -- Multiple db connections opened (#6478) 
* Fixed scoped_session factory creation

* Fixed py format

* Added comment for GC + added explicit QueuePool definition
 2022-08-01 13:42:35 +02:00
Teddy
7c70adc33e
Changed ometa_service_name to database_service_name (#6454) 2022-08-01 07:41:46 +02:00
Francisco J. Jurado Moreno
512b241069
[ISSUE-5522] Fix profiler timeout (#6448) 
* ChangeIsFunction by IsMethod

* Update test

* Remove whitespace
 2022-07-30 10:03:10 -07:00
Teddy
a920a4c17d
Fixed SQA Warning (#6442) 2022-07-30 09:39:36 -07:00
Mayur Singal
82de3a4cc6
GCS Datalake Object Filtering Fix (#6447) 2022-07-30 08:48:06 +05:30
Sriharsha Chintalapani
27558457a7
Add ES indexing option to sort alphabetically (#6444) 2022-07-29 14:11:22 -07:00
Teddy
6397b6a0b1
Fixes #6325 -- Implement multithreading for metrics computation (#6406) 
* Added tests for multithreading SQA interface

* Added multithread support for metric computation

* Added thread ID to log debuger

* Cleaned up tests

* Fixed python formatting issues

* Added non blocking result processing + threadCount in config file to set numbers of threads

* Added frontend input field to set number of threads

* Fixed code smell, bug and comments from reviewer
 2022-07-29 10:41:53 +02:00
Mayur Singal
bafca3b7b6
Fix #6387: Clean Ingestion (#6405) 
* Fix #6387: Clean Ingestion

* postgres fix

* Fixed Location in Sample Data
 2022-07-29 08:51:58 +02:00
Pere Miquel Brull
ea918495ff
Fix #6382 - Snyk workflows (#6413) 
Fix #6382 - Snyk workflows (#6413)
 2022-07-28 23:06:33 +02:00
Onkar Ravgan
7fdab21a72
Glue Topology Refactor (#6231) 
* rebase with main

* refactored glue for topology

* added glue unit tests

* resolved review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-07-28 15:04:43 +02:00
Mayur Singal
9c7d8f7d8f
Fix #5847: Handled Exception For Source Config (#6386) 
* Fix #5847: Handled Exception For Source Config

* Comment Fix

* Added Test Cases
 2022-07-27 20:21:16 +02:00
Mayur Singal
c6efe8c142
Fix #6158: Fivetran Connector Added (#6329) 
* Fivetran Connector Added

* Fivetran unit test added

* Fixed Unit Tests

* Added Fivetran Docs & Added Fivetran in Pipeline Listing
 2022-07-27 20:50:56 +05:30