6 Commits

Author SHA1 Message Date
Teddy
595e5c1b89
Added checks to see if DAG can reach OM (#7794) 2022-09-30 17:57:03 +02:00
Shailesh Parmar
3b67cc824d
Data qaulity add test's workflow (#6957)
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-29 17:26:59 +05:30
Nahuel
bdbbca0efe
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
2022-08-26 07:29:38 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441)
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
2022-08-02 09:13:46 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00