15 Commits

Author SHA1 Message Date
Shailesh Parmar
3b67cc824d
Data qaulity add test's workflow (#6957)
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-29 17:26:59 +05:30
Nahuel
bdbbca0efe
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
2022-08-26 07:29:38 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Pere Miquel Brull
667c837ae0
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
2022-08-24 16:27:31 +02:00
Pere Miquel Brull
a975d5177d
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
2022-08-23 08:59:44 +02:00
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832)
* Implementation of temp secret for testing connection

* Fix tests
2022-08-22 08:43:23 +02:00
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805)
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
2022-08-19 16:15:40 +02:00
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568)
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-12 13:24:58 +05:30
Nahuel
66e0527a90
Fix: Profiler DAG takes loggerLevel from ingestion pipeline (#6685) 2022-08-10 17:02:16 +02:00
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610)
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
2022-08-09 09:00:43 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Pere Miquel Brull
dc712e3179
Add executionDate to ingestionPipeline (#6481) 2022-08-03 08:13:55 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441)
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
2022-08-02 09:13:46 +02:00
Pere Miquel Brull
8ec2dbe6e4
Update vuln (#6432) 2022-07-29 13:08:12 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00