yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-13 09:48:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,851 Commits 850 Branches 169 Tags
Commit Graph

211 Commits

Author SHA1 Message Date
Teddy
78b5f8c8e2
Part 1 of #5831 -- Profiler workflow implementation (#6809) 
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells
 2022-08-19 10:52:08 +02:00
Ayush Shah
a6db2e8a84
Fix for profiler: modified filter patterns and added error handling (#6608) 2022-08-08 10:43:17 +05:30
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416) 
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
 2022-08-04 07:22:47 -07:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441) 
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
 2022-08-02 09:13:46 +02:00
Teddy
818736e2ca
Fix SQLite same thread error (#6486) 2022-08-01 17:33:53 +02:00
Teddy
6397b6a0b1
Fixes #6325 -- Implement multithreading for metrics computation (#6406) 
* Added tests for multithreading SQA interface

* Added multithread support for metric computation

* Added thread ID to log debuger

* Cleaned up tests

* Fixed python formatting issues

* Added non blocking result processing + threadCount in config file to set numbers of threads

* Added frontend input field to set number of threads

* Fixed code smell, bug and comments from reviewer
 2022-07-29 10:41:53 +02:00
Mayur Singal
9adaec1b3a
Fix #6248: Use PUT API instead of POST for tag & tagCategory (#6344) 
* Fix #6248: Use PUT API instead of POST for tag & tagCategory

* Restructure yield_tag topology

* Renamed method to create_or_update
 2022-07-27 08:41:55 +02:00
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager (#6330) 
* Implentation for retrieving auth provider config from Secret Manager

* Address PR comments

* Address code smells from SonarCloud

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-07-26 15:42:40 +02:00
Teddy
aae4410c93
Fies #6183 - Ability to set profile sample at the profilier workflow level (#6292) 
Fies  #6183 - Ability to set profile sample at the profilier workflow level (#6292)
 2022-07-25 12:08:20 +02:00
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs (#6307) 
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
 2022-07-24 18:49:15 +02:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props (#5700) 
Fix #5822 - Remove looker env and rename props (#5700)
 2022-07-24 16:02:26 +02:00
Pere Miquel Brull
2936c0e4e3
Remove retry in ES query (#6191) 
Remove retry in ES query (#6191)
 2022-07-20 12:44:50 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128) 
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
 2022-07-19 14:51:44 +02:00
Pere Miquel Brull
9fa5d5cab2
Fix #6154 - Python Client to PATCH descriptions (#6165) 
* Prepare patch description utilities for python client

* Improvements
 2022-07-19 12:58:58 +02:00
Sriharsha Chintalapani
ad0c6dad88
Fix #5613: Add fernet encryption and remove connection details for non admin users (#5614) 
* Fix #5613: Add fernet encryption and remove connection details for non-admin users
 2022-06-24 16:44:37 -07:00
Pere Miquel Brull
0ecc9f0da6
Fix #5459 - Remove sql-metadata in favor of sqllineage (#5494) 
Fix #5459 - Remove sql-metadata in favor of sqllineage (#5494)
 2022-06-21 18:02:50 +02:00
Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices (#5446) 
Fix #5367 - MlFlow connector & MLServices (#5446)
 2022-06-21 14:54:36 +02:00
Teddy
5067e24374
[ISSUE-4723] Fix Snowflake Case Sensitive Error with Profiler (#5533) 
* Fixed snowflake profiler + enabled profiler integration tests

* Fixed py formating
 2022-06-20 22:23:17 +02:00
Teddy
c151a34c70
[ISSUE-4659] Add Additional GE Tests to the Profiler (#5506) 
* Added table tests 'column count to be between' and 'column name to be in set'

* Added 'column values to be in set' test

* Fixed wrong filed name when invoking TestCaseResult

* Added col min and col max to be between

* Fix formatting

* Fix test after adding column name list to table profile

* Added 'column values sum between' test

* Added 'table column name to exist' test

* Fixed JSON description white space + spelling mistake
 2022-06-20 08:54:42 +02:00
Snyk bot
c717cb1545
fix: ingestion/tests/integration/lineage/airflow/requirements.txt to reduce vulnerabilities (#5178) 
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-FLASKAPPBUILDER-2433100
 2022-06-08 13:36:31 -07:00
Pere Miquel Brull
8e9d0a73f6
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265) 
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
 2022-06-08 16:10:40 +02:00
Pere Miquel Brull
9138c70dee
Fix #4696 - Only store URL suffixes when extracting metadata (#5281) 
* Only store URL suffix

* Fix test service name
 2022-06-03 11:43:40 +02:00
Mayur Singal
b3428771a3
Fix #4767: Refactor Usage & Lineage (#5190) 
Fix #4767: Refactor Usage & Lineage (#5190)
 2022-06-03 10:12:28 +02:00
Sriharsha Chintalapani
cf3d4314dd
Fix #1607 Operational metrics for OpenMetadata system observability (#4386) 
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
 2022-05-27 15:39:13 +05:30
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code (#5165) 2022-05-26 21:00:18 +02:00
mosiac1
a47a2ce5b7
Add backend and UI support for direct joins between tables (#4943) 2022-05-25 23:41:26 -07:00
Pere Miquel Brull
35e67890b8
Fix #5141 - Iterate over all Entities in the profiler workflow (#5146) 
Fix #5141 - Iterate over all Entities in the profiler workflow (#5146)
 2022-05-26 07:35:23 +02:00
Mayur Singal
41ee3a5aaf
Fix #3940 : Refractor Sql Source (#5046) 
Fix #3940 : Refractor Sql Source (#5046)
 2022-05-25 15:41:38 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125) 
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125)
 2022-05-25 08:35:16 +02:00
Abhishek Pandey
0a69a82b98
issue-3828: updated-workflow-and-stage-test (#5117) 2022-05-25 07:40:22 +02:00
Pere Miquel Brull
753d901fb2
Fix Python tests & flakiness (#5095) 
Fix Python tests & flakiness (#5095)
 2022-05-23 14:54:47 +02:00
Pere Miquel Brull
cd35447792
Fix #4688 - Python FQN with ANTRL & ES (#5073) 
Fix #4688 - Python FQN with ANTRL & ES (#5073)
 2022-05-23 09:51:34 +02:00
Pere Miquel Brull
04421901c0
[WIP] Fix #4610 - Pipeline Tasks removal (#4615) 
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
 2022-05-07 21:10:35 -07:00
Pere Miquel Brull
a22ff4627f
Fix #4728 - Fix Kafka test connection (#4730) 
* Fix Kafka connection

* Update defaults and required values

* Update test connection

* Add filter tests

* Fix kafka tests
 2022-05-06 08:59:05 +02:00
Ayush Shah
4d2bde4fe7
Fixes Tableau - added env with default value (#4408) 2022-04-23 21:53:13 +05:30
Ayush Shah
a94bb5fc8e
Fixes Tableau + fix Charts not getting ingested as part of dashboard (#4327) 
* Fixes Tableau + fix Charts not getting ingested as part of dashboard

* pytest fix

* Fix Tableau json config
 2022-04-22 12:49:55 +02:00
Mayur Singal
db0e34c709
Fixing Test Connection for Dynamo & Glue (#4316) 
* Fixing Test Connection for Dynamo

* Fixed Glue Connector

* renamed engine to connection

* Fixed the return signature

* Added dataclass
 2022-04-22 11:30:59 +05:30
Pere Miquel Brull
256b16d877
Fix #4032 - Bigquery properties & GCS Credentials (#4202) 
Fix #4032 - Bigquery properties & GCS Credentials (#4202)
 2022-04-19 12:31:34 +02:00
Milan Bariya
924e211ba1
ISSUE-4076: Use the central get_engine method for usage (#4180) 
* ISSUE-4076: Use the central get_engine method for usage

* ISSUE-4076: Use the central get_engine method for usage

* ISSUE-4076: Use the central get_engine method for usage

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
 2022-04-19 12:35:01 +05:30
Pere Miquel Brull
63415952e3
Fix sample data (#4200) 
Fix sample data (#4200)
 2022-04-18 19:00:36 +02:00
Pere Miquel Brull
2aecebd595
Fix #4189 - Fix airflow lineage callback (#4191) 
Fix #4189 - Fix airflow lineage callback (#4191)
 2022-04-18 15:13:26 +02:00
Pere Miquel Brull
3f4129d239
Prepare custom encoder for SecretStr (#4186) 2022-04-18 14:43:22 +05:30
codingwithabhi
8115c2ece4
issue-4135: updated-mysql-connection-json (#4136) 2022-04-14 14:16:09 -07:00
Pere Miquel Brull
6768bf708d
Fix Py tests (#4122) 
Fix Py tests (#4122)
 2022-04-14 11:22:39 +02:00
Pere Miquel Brull
43be01b92e
Fix #4091 & #3948 - Create services correctly (#4101) 2022-04-13 18:04:52 +02:00
Ayush Shah
1f3667b46f
Fix Python test (#4095) 2022-04-13 21:03:33 +05:30
Sriharsha Chintalapani
be836e5404
Fix #4071: PUT IngestionPipeline missing property & error message (#4085) 
Fix #4071: PUT IngestionPipeline missing property & error message (#4085)
 2022-04-13 08:40:21 +02:00
Pere Miquel Brull
6a6507e754
Fix #3962 - Profiler uses DatabaseSchema & Sample Data fix (#4056) 2022-04-12 13:40:59 +05:30
Pere Miquel Brull
06a3e4c989
Fix #3825 - Schema Name, SQL Source FQDN & ORM Profiler (#3942) 
* Fix db schema name

* Fix sqlite connection

* Correctly register scanned tables

* improve sqlite connection

* Adapt schemas on ORM profiler

* Format
 2022-04-08 19:28:10 +05:30
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field (#3909) 
* Fix sample data DAG

* Fix callback imports

* Use --set-default-enum-member in generate

* Format

* Add faulty merge hard_delete

* Fix airflow lineage, improve naming and fix lineage tests

* Add mysql url test

* Add mysql url test

* Update CI name

* Fix test ometa endpoint

* Format

* Fix metadata config
 2022-04-07 18:22:50 +05:30