187 Commits

Author SHA1 Message Date
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159)
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Sriharsha Chintalapani
25449001ca
Fix #9040: Remove fields such as tableQueries, tableProfile, tests, sample data as part of table fields (#9041) 2022-12-06 21:07:04 -08:00
Pere Miquel Brull
58f3c01470
Fix #7056 - Automatic lineage from ML Model to ML Feature Data Source (#9113) 2022-12-05 16:43:03 +05:30
Pere Miquel Brull
eccbb087d3
Add table query test (#9000)
* Add table query test

* Add table query test
2022-11-25 06:56:12 +01:00
Teddy
8789bdb847
Ometa mixin for ingestion pipeline (#8931)
* Added sleep between data processing ingestion and KPI processing

* Added ometa mixin for ingestion pipeline and test suite
2022-11-23 09:46:56 +00:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Pere Miquel Brull
d48fd468d4
Tag PATCH + Cleanup of helpers methods (#8150)
* cleanup

* lint

* Add tag patch

* Fix rename

* Dont kill tests
2022-10-15 14:56:30 +02:00
Onkar Ravgan
35efd49256
Added control for DBT descriptions (#7653)
* Added control for DBT descriptions

* Fixed tests

* Added UI changes

* fixed maven ci tests

* Java formatting changes

* ui review fixes

* Fixed pytests

* Fixed python integration tests

* fixed airflow tests

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-09-26 16:19:47 +05:30
Ayush Shah
517e9a4aac
Remove unused imports (#7657) 2022-09-22 16:42:10 +05:30
Teddy
0677b5258f
Fixes #7299 by updating GE submodule to new test API (#7332)
* Update GE logic to the new test API

* Cleaned up part of the code from
update to TestSuiteMixin

* Added + fixed tests

* Deleted unused files for GE tests

* fixed python tests
2022-09-11 20:07:19 -07:00
Nahuel
b829a2cbf3
Fix: Improvements on secret manager implementation (#7282)
* Change local secret manager by noop

* Update openmetadata-secure-test.yaml
2022-09-07 09:18:59 +02:00
Sriharsha Chintalapani
d29a230726
Fix #7104: Add support for get resource permission by FQN (#7270) 2022-09-06 12:30:52 -07:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832)
* Implementation of temp secret for testing connection

* Fix tests
2022-08-22 08:43:23 +02:00
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805)
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
2022-08-19 16:15:40 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441)
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
2022-08-02 09:13:46 +02:00
Mayur Singal
9adaec1b3a
Fix #6248: Use PUT API instead of POST for tag & tagCategory (#6344)
* Fix #6248: Use PUT API instead of POST for tag & tagCategory

* Restructure yield_tag topology

* Renamed method to create_or_update
2022-07-27 08:41:55 +02:00
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager (#6330)
* Implentation for retrieving auth provider config from Secret Manager

* Address PR comments

* Address code smells from SonarCloud

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 15:42:40 +02:00
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
Fix #6280 - Bump DeltaLake version, tests and docs (#6307)
2022-07-24 18:49:15 +02:00
Pere Miquel Brull
810e6c7dfc
Fix #5822 - Remove looker env and rename props (#5700)
Fix #5822 - Remove looker env and rename props (#5700)
2022-07-24 16:02:26 +02:00
Pere Miquel Brull
2936c0e4e3
Remove retry in ES query (#6191)
Remove retry in ES query (#6191)
2022-07-20 12:44:50 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
2022-07-19 14:51:44 +02:00
Pere Miquel Brull
9fa5d5cab2
Fix #6154 - Python Client to PATCH descriptions (#6165)
* Prepare patch description utilities for python client

* Improvements
2022-07-19 12:58:58 +02:00
Sriharsha Chintalapani
ad0c6dad88
Fix #5613: Add fernet encryption and remove connection details for non admin users (#5614)
* Fix #5613: Add fernet encryption and remove connection details for non-admin users
2022-06-24 16:44:37 -07:00
Pere Miquel Brull
0ecc9f0da6
Fix #5459 - Remove sql-metadata in favor of sqllineage (#5494)
Fix #5459 - Remove sql-metadata in favor of sqllineage (#5494)
2022-06-21 18:02:50 +02:00
Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices (#5446)
Fix #5367 - MlFlow connector & MLServices (#5446)
2022-06-21 14:54:36 +02:00
Teddy
c151a34c70
[ISSUE-4659] Add Additional GE Tests to the Profiler (#5506)
* Added table tests 'column count to be between' and 'column name to be in set'

* Added 'column values to be in set' test

* Fixed wrong filed name when invoking TestCaseResult

* Added col min and col max to be between

* Fix formatting

* Fix test after adding column name list to table profile

* Added 'column values sum between' test

* Added 'table column name to exist' test

* Fixed JSON description white space + spelling mistake
2022-06-20 08:54:42 +02:00
Pere Miquel Brull
8e9d0a73f6
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
2022-06-08 16:10:40 +02:00
Mayur Singal
b3428771a3
Fix #4767: Refactor Usage & Lineage (#5190)
Fix #4767: Refactor Usage & Lineage (#5190)
2022-06-03 10:12:28 +02:00
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code (#5165) 2022-05-26 21:00:18 +02:00
mosiac1
a47a2ce5b7
Add backend and UI support for direct joins between tables (#4943) 2022-05-25 23:41:26 -07:00
Pere Miquel Brull
35e67890b8
Fix #5141 - Iterate over all Entities in the profiler workflow (#5146)
Fix #5141 - Iterate over all Entities in the profiler workflow (#5146)
2022-05-26 07:35:23 +02:00
Mayur Singal
41ee3a5aaf
Fix #3940 : Refractor Sql Source (#5046)
Fix #3940 : Refractor Sql Source (#5046)
2022-05-25 15:41:38 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125)
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125)
2022-05-25 08:35:16 +02:00
Pere Miquel Brull
753d901fb2
Fix Python tests & flakiness (#5095)
Fix Python tests & flakiness (#5095)
2022-05-23 14:54:47 +02:00
Pere Miquel Brull
cd35447792
Fix #4688 - Python FQN with ANTRL & ES (#5073)
Fix #4688 - Python FQN with ANTRL & ES (#5073)
2022-05-23 09:51:34 +02:00
Pere Miquel Brull
04421901c0
[WIP] Fix #4610 - Pipeline Tasks removal (#4615)
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
2022-05-07 21:10:35 -07:00
Pere Miquel Brull
a22ff4627f
Fix #4728 - Fix Kafka test connection (#4730)
* Fix Kafka connection

* Update defaults and required values

* Update test connection

* Add filter tests

* Fix kafka tests
2022-05-06 08:59:05 +02:00
Ayush Shah
4d2bde4fe7
Fixes Tableau - added env with default value (#4408) 2022-04-23 21:53:13 +05:30
Ayush Shah
a94bb5fc8e
Fixes Tableau + fix Charts not getting ingested as part of dashboard (#4327)
* Fixes Tableau + fix Charts not getting ingested as part of dashboard

* pytest fix

* Fix Tableau json config
2022-04-22 12:49:55 +02:00
Pere Miquel Brull
256b16d877
Fix #4032 - Bigquery properties & GCS Credentials (#4202)
Fix #4032 - Bigquery properties & GCS Credentials (#4202)
2022-04-19 12:31:34 +02:00
Pere Miquel Brull
63415952e3
Fix sample data (#4200)
Fix sample data (#4200)
2022-04-18 19:00:36 +02:00
Pere Miquel Brull
3f4129d239
Prepare custom encoder for SecretStr (#4186) 2022-04-18 14:43:22 +05:30
Pere Miquel Brull
6768bf708d
Fix Py tests (#4122)
Fix Py tests (#4122)
2022-04-14 11:22:39 +02:00
Pere Miquel Brull
43be01b92e
Fix #4091 & #3948 - Create services correctly (#4101) 2022-04-13 18:04:52 +02:00
Ayush Shah
1f3667b46f
Fix Python test (#4095) 2022-04-13 21:03:33 +05:30
Sriharsha Chintalapani
be836e5404
Fix #4071: PUT IngestionPipeline missing property & error message (#4085)
Fix #4071: PUT IngestionPipeline missing property & error message (#4085)
2022-04-13 08:40:21 +02:00